Abstract: An embodiment may involve a network interface configured to capture data packets into a binary format and a non-volatile memory configured to temporarily store the data packets received by way of the network interface. The embodiment may also involve a first array of processing elements each configured to independently and asynchronously: (i) read a chunk of data packets from the non-volatile memory, (ii) identify flows of data packets within the chunk, and (iii) generate flow representations for the flows. The embodiment may also involve a second array of processing elements configured to: (i) receive the flow representations from the first array of processing elements, (ii) identify and aggregate common flows across the flow representations into an aggregated flow representation, (iii) based on a filter specification, remove one or more of the flows from the aggregated flow representation, and (iv) write information from the aggregated flow representation to the database.

Abstract: An embodiment may involve a network interface module; volatile memory configured to temporarily store data packets received from the network interface module; high-speed non-volatile memory; an interface connecting to low-speed non-volatile memory; a first set of processors configured to perform a first set of operations that involve: (i) reading the data packets from the volatile memory, (ii) arranging the data packets into chunks, each chunk containing a respective plurality of the data packets, and (iii) writing the chunks to the high-speed non-volatile memory; and a second set of processors configured to perform a second set of operations in parallel to the first set of operations, where the second set of operations involve: (i) reading the chunks from the high-speed non-volatile memory, (ii) compressing the chunks, (iii) arranging the chunks into blocks, each block containing a respective plurality of the chunks, and (iv) writing the blocks to the low-speed non-volatile memory.

Abstract: An embodiment may involve non-volatile memory configured to store chunks of data packets, wherein the chunks are associated with sequence numbers; a shared producer queue; one or more processors configured to transfer the chunks to the shared producer queue in order of the sequence numbers; an array of n sets of processors configured to: (i) read the chunks from the shared producer queue, (ii) re-write network addresses within the data packets to create modified chunks, and (iii) write the modified chunks to queues; and a field programmable gate array based network interface containing the queues and m physical ports, and configured to: (i) read the modified chunks in order of their sequence numbers, (ii) unpack the modified chunks into data packets, (iii) write updated checksums to the data packets, (iv) respectively select output ports for the data packets, and (v) transmit the data packets from the selected output ports.

Abstract: An embodiment may involve a network interface configured to capture data packets into a binary format and a non-volatile memory configured to temporarily store the data packets received by way of the network interface. The embodiment may also involve a first array of processing elements each configured to independently and asynchronously: (i) read a chunk of data packets from the non-volatile memory, (ii) identify flows of data packets within the chunk, and (iii) generate flow representations for the flows. The embodiment may also involve a second array of processing elements configured to: (i) receive the flow representations from the first array of processing elements, (ii) identify and aggregate common flows across the flow representations into an aggregated flow representation, (iii) based on a filter specification, remove one or more of the flows from the aggregated flow representation, and (iv) write information from the aggregated flow representation to the database.

Abstract: An embodiment may involve receiving a chunk and a chunk index, where the chunk contains packets captured by a network interface unit and the chunk index contains timestamps of first and last packets within the chunk. The chunk may be stored in a first ring buffer of a first memory and the chunk index may be stored in an index buffer of the first memory. A processor may allocate an entry in an I/O queue of a second memory and an entry in a chunk processing queue of the first memory. The processor may read the chunk processing queue to identify and copy the chunk from the first ring buffer to a location in a second ring buffer of the second memory, the location associated with the entry in the I/O queue. The same or a different processor may instruct a controller to write the chunk to a non-volatile memory unit.

Abstract: An embodiment may involve executing a set of instructions, where the set of instructions define how to generate outputs that represent one or more data packets, and where segments of the outputs are copied from first parts of respective instructions in the set of instructions. The embodiment may further involve: retrieving, from a plurality of registers, a data packet header; retrieving, from the plurality of registers, a first part of a data packet payload and an increment value; applying the increment value to the first part of the data packet payload to generate a second part of the data packet payload; storing, in the plurality of registers, the first part of the data packet payload with the increment value applied; and providing, as additional segments of the outputs, the data packet header, the first part of the data packet payload, and the second part of the data packet payload.

Abstract: An embodiment may involve a network interface module; volatile memory configured to temporarily store data packets received from the network interface module; high-speed non-volatile memory; an interface connecting to low-speed non-volatile memory; a first set of processors configured to perform a first set of operations that involve: (i) reading the data packets from the volatile memory, (ii) arranging the data packets into chunks, each chunk containing a respective plurality of the data packets, and (iii) writing the chunks to the high-speed non-volatile memory; and a second set of processors configured to perform a second set of operations in parallel to the first set of operations, where the second set of operations involve: (i) reading the chunks from the high-speed non-volatile memory, (ii) compressing the chunks, (iii) arranging the chunks into blocks, each block containing a respective plurality of the chunks, and (iv) writing the blocks to the low-speed non-volatile memory.

Abstract: An embodiment may involve non-volatile memory configured to store chunks of data packets, wherein the chunks are associated with sequence numbers; a shared producer queue; one or more processors configured to transfer the chunks to the shared producer queue in order of the sequence numbers; an array of n sets of processors configured to: (i) read the chunks from the shared producer queue, (ii) re-write network addresses within the data packets to create modified chunks, and (iii) write the modified chunks to queues; and a field programmable gate array based network interface containing the queues and m physical ports, and configured to: (i) read the modified chunks in order of their sequence numbers, (ii) unpack the modified chunks into data packets, (iii) write updated checksums to the data packets, (iv) respectively select output ports for the data packets, and (v) transmit the data packets from the selected output ports.

Abstract: An embodiment may involve receiving a chunk and a chunk index, where the chunk contains packets captured by a network interface unit and the chunk index contains timestamps of first and last packets within the chunk. The chunk may be stored in a first ring buffer of a first memory and the chunk index may be stored in an index buffer of the first memory. A processor may allocate an entry in an I/O queue of a second memory and an entry in a chunk processing queue of the first memory. The processor may read the chunk processing queue to identify and copy the chunk from the first ring buffer to a location in a second ring buffer of the second memory, the location associated with the entry in the I/O queue. The same or a different processor may instruct a controller to write the chunk to a non-volatile memory unit.