Abstract: Efficient deduplication for storage systems is disclosed, including: using the current data structure to track a set of deduplication data entries associated with a collection group; determining a flush criterion has been met; and storing the set of deduplication data entries of the current data structure as one or more persistent objects associated with the collection group in a persistent storage, wherein the one or more persistent objects are usable to update a set of persistently stored metadata associated with the collection group that tracks deduplication data entries generated before a previous merge operation.

Abstract: Efficient recovery of deduplication data for high capacity systems is disclosed, including: reading from the data storage device a data structure that tracks a plurality of segments to which a plurality of persistent objects have been recently written, wherein segments are written to in a monotonically increasing numerical order; selecting a checkpoint segment from among the plurality of segments based at least in part on a plurality of segment numbers corresponding to respective ones of the plurality of segments; using the checkpoint segment and a segment associated with a latest available segment number to determine a set of candidate segments; reading at least a portion of the set of candidate segments to identify a data storage block for which a corresponding deduplication data entry is not already stored in persistently stored deduplication data entries; and storing a new deduplication data entry to insert a fingerprint associated with the data storage block in a current data structure stored in a memory

Abstract: Efficient deduplication for storage systems is disclosed, including: using the current data structure to track a set of deduplication data entries associated with a collection group; determining a flush criterion has been met; and storing the set of deduplication data entries of the current data structure as one or more persistent objects associated with the collection group in a persistent storage, wherein the one or more persistent objects are usable to update a set of persistently stored metadata associated with the collection group that tracks deduplication data entries generated before a previous merge operation.

Abstract: Efficient recovery of deduplication data for high capacity systems is disclosed, including: reading from the data storage device a data structure that tracks a plurality of segments to which a plurality of persistent objects have been recently written, wherein segments are written to in a monotonically increasing numerical order; selecting a checkpoint segment from among the plurality of segments based at least in part on a plurality of segment numbers corresponding to respective ones of the plurality of segments; using the checkpoint segment and a segment associated with a latest available segment number to determine a set of candidate segments; reading at least a portion of the set of candidate segments to identify a data storage block for which a corresponding deduplication data entry is not already stored in persistently stored deduplication data entries; and storing a new deduplication data entry to insert a fingerprint associated with the data storage block in a current data structure stored in a memory