Abstract: In some implementations, a storage device may detect partial similarity between a candidate storage page and a target storage page using an XOR operation on the candidate storage page and the target storage page. The storage device may generate a data structure indicative of the partial similarity. The storage device may store the data structure with a pointer to the target storage page. The storage device may receive a read command associated with the candidate storage page. The storage device may reconstruct, in response to the read command, the candidate storage page based on the data structure and the pointer.

Abstract: Techniques for providing efficient memory management for Bloom filters based on index fullness. The techniques include, in each of multiple destaging cycles at an in-memory index level L1, destaging index entries from a single bucket at L1 across N buckets at an intermediate on-drive index level L2, and allocating a Bloom filter for each bucket at L2 and constructing it based on the index entries in the bucket at L2, in which the Bloom filter has a size dynamically proportional to a current fullness of the bucket at L2. The techniques include, in response to an expected fullness of the bucket at L2 resulting from a next destaging cycle at L1 being 100%, destaging index entries from the single bucket at L1 across M buckets at an on-drive index level L3. The techniques include, destaging the index entries from the N buckets at L2 across the M buckets at L3.

Abstract: Techniques for improving amortization when hardening index entries across a multilevel hash table. The techniques include, in a first hardening cycle, in response to a first group of index entries filling a bucket at an in-memory hash table level L1, hardening the bucket at L1 to an initial bucket at an intermediate on-drive hash table level L2. The techniques include, in subsequent successive hardening cycles, incrementally increasing the number of buckets at L2 to a final number of buckets according to an arithmetic series, and, in response to a next group of index entries up to a last group of index entries filling the bucket at L1, hardening the bucket at L1 across the incrementally increased number of buckets at L2. The techniques include, in response to the final number of buckets at L2 being filled, hardening the buckets at L2 across buckets at an on-drive hash table level L3.

Abstract: Techniques can include: recording, in a log, log entries for write operations writing data pages to logical addresses; and flushing the log entries from the log, wherein flushing includes performing deduplication of the data pages. Flushing can include: partitioning the data pages into i) a first portion of the data pages that are determined to be duplicates of existing data pages, and ii) a second portion of the data pages determined to be unique data pages; atomically incrementing reference counts each corresponding to one of the existing data pages determined as identical to a data page of the first portion; and atomically binding the logical addresses to the data pages, including: binding the first portion of data pages to a corresponding third portion of the logical addresses; and binding the second portion of data pages to a corresponding fourth portion of the logical addresses.

Abstract: Techniques for keeping on-disk, physical deduplication (“dedupe”) index levels up to date to increase the efficiency of data deduplication in storage systems. The techniques include providing a plurality of dedupe indexes across a memory level and a disk level of a storage system, in which the plurality of dedupe indexes include an in-memory dedupe index at the memory level, and an on-disk dedupe index at the disk level. The techniques include applying a plurality of eviction rules regarding evicting entries of the on-disk dedupe index, in which the plurality of eviction rules avoid random evictions of entries from the on-disk dedupe index. The techniques include, in response to applying the plurality of eviction rules, identifying a plurality of entries of the on-disk dedupe index as eviction candidates. The techniques include evicting the identified eviction candidates from the on-disk dedupe index.

Abstract: Techniques for keeping on-disk, physical deduplication (“dedupe”) index levels up to date to increase the efficiency of data deduplication in storage systems. The techniques include providing a plurality of dedupe indexes across a memory level and a disk level of a storage system, in which the plurality of dedupe indexes include an in-memory dedupe index at the memory level, and an on-disk dedupe index at the disk level. The techniques include applying a plurality of eviction rules regarding evicting entries of the on-disk dedupe index, in which the plurality of eviction rules avoid random evictions of entries from the on-disk dedupe index. The techniques include, in response to applying the plurality of eviction rules, identifying a plurality of entries of the on-disk dedupe index as eviction candidates. The techniques include evicting the identified eviction candidates from the on-disk dedupe index.

Abstract: A method of data deduplication during log flushing includes use of an in-memory deduplication index as a cached sub-set of entries from a full on-drive deduplication index. In a pre-flush process, the data is scanned and the on-drive index is accessed to identify matching deduplication entries, and identified entries are copied into the in-memory index. The subsequent flush includes, for each data unit, (1) accessing the in-memory index to identify a matching deduplication entry for the data unit, (2) upon identifying a matching entry, storing the data unit in a deduplicated manner as a reference to previously stored data, and (3) upon not identifying a matching entry, storing the data unit in a non-deduplicated manner including a new data instance, without accessing the on-drive index. Performance and efficiency are improved by accessing only the in-memory index and avoiding the penalties of accessing the on-drive index.

Abstract: Techniques for achieving efficient removal of stale index entries from on-drive deduplication indexes using hash prefix indexing. The techniques include providing an on-drive dedupe index that includes a plurality of index entries. Each index entry includes a hash value of a data page, and an address associated with a storage location of the data page. Each index entry is assigned to a bucket defined by a respective hash prefix. The techniques include, for each data page associated with a reference count decremented to zero, logging a hash prefix and a storage address of the data page. The techniques include, for each bucket, constructing an address bag, storing, in the address bag, logged addresses whose corresponding hash prefix is the same as the respective hash prefix defining the bucket, and removing, from the bucket, each stale index entry that includes an address matching one of the addresses in the address bag.

Abstract: A flush set of data pages to flush from a cache to non-volatile data storage is identified, and a compression group is identified that is made up of similar data pages within the flush set. A previously compressed data page is identified in non-volatile data storage that is similar to the data pages in the compression group. Data compression is performed on the compression group using a compression dictionary for the previously compressed data page. A flush set specific hash index and a global hash index may be used to identify the similar data pages in the compression group and the similar previously compressed data page. An indication of the similar previously compressed data page may be stored in the compressed block generated for the compression group, and the compression dictionary for the previously compressed data page need not be duplicated in the non-volatile data storage.

Abstract: A method of data deduplication during log flushing includes use of an in-memory deduplication index as a cached sub-set of entries from a full on-drive deduplication index. In a pre-flush process, the data is scanned and the on-drive index is accessed to identify matching deduplication entries, and identified entries are copied into the in-memory index. The subsequent flush includes, for each data unit, (1) accessing the in-memory index to identify a matching deduplication entry for the data unit, (2) upon identifying a matching entry, storing the data unit in a deduplicated manner as a reference to previously stored data, and (3) upon not identifying a matching entry, storing the data unit in a non-deduplicated manner including a new data instance, without accessing the on-drive index. Performance and efficiency are improved by accessing only the in-memory index and avoiding the penalties of accessing the on-drive index.