Abstract: A system and method are disclosed for processing a data stream. A data segment is received. It is determined whether the data segment has been previously stored. In the event that the data segment is determined not to have been previously stored, a unique identifier for specifying the data segment in a representation of the data stream is generated.

Abstract: A system and method are disclosed for providing efficient data storage. A data stream comprising a plurality of data segments is received. The system determines whether one of the plurality of data segments has been stored previously using a summary in a low latency memory; in the event that the data segment is determined not to have been stored previously, assigning an identifier to the data segment.

Abstract: A method for storing data comprising is disclosed. The method comprises receiving a data stream comprising a plurality of data segments wherein each data segment is associated with an identifier. The method further determining using a subset of identifiers that are stored in a low latency memory whether a data segments has been previously stored and returning the identifier for the data segment in the event the data segment is determined to have been stored previously.