Abstract: An aspect includes implementing capacity reduction in a storage system includes for each of a candidate page and a target page in the storage system, identifying a subset of sectors having identical data or a minimum amount of non-identical data, performing a bit-wise exclusive OR (XOR) operation on sectors of the candidate page and the target page, determining entropy from results of the XOR operation. Upon determining the entropy is less than or equal to a threshold value, an aspect includes building a reference page from an XOR sector containing results of the bit-wise XOR operation, and performing a compression operation on the reference page.

Abstract: A mapper tree for a logical volume is provided by storing, in each leaf node of the mapper tree, pointers to pages of non-volatile storage that store host data written to corresponding pages within a segment of the logical address space of the logical volume that corresponds to the leaf node. In response to receiving an initial write operation directed to a segment of the logical address space of the logical volume for which no leaf node currently exists in the mapper tree, a representation of a new leaf node is added to a super leaf node in the mapper tree that efficiently stores representations of multiple leaf nodes.

Abstract: An aspect includes providing a metadata structure having a logical level that points to a virtual level and a physical level to which the virtual level points. The method also includes storing, at the virtual level, a reference counter for each of a plurality of virtual-level type storage address entries in the metadata structure, and providing a pointer in the metadata structure between each pair of a number of pairs of virtual level address entries in which corresponding pages share a set of common sectors. The reference counter tracks a number of instances in which a corresponding pointer points to a corresponding virtual level address entry. An aspect further includes storing a single instance of the common sectors at the physical level.

Abstract: A technique for performing deduplication identifies representative sub-blocks within candidate blocks and performs sub-block matching to entries in a digest database. When a representative sub-block is matched to a differently-aligned target sub-block that belongs to a target block, the technique effectuates storage of the candidate block using the target block and a block adjacent to the target block.

Abstract: An aspect of performance optimization in a storage system environment includes providing a metadata structure that describes a storage space of a storage system. For each storage object of a plurality of storage objects, an aspect includes monitoring input/output (IO) patterns for IO operations conducted, monitoring deduplication characteristics of the storage object, and configuring local mapping layer parameters. An aspect further includes modifying local object metadata in the metadata structure as a function of the monitoring and the configuring.

Abstract: A technique is directed to managing a redundant array of independent disks (RAID) group. The technique involves forming the RAID group from storage devices of different types that provide different data storage characteristics. The technique further involves receiving input/output (I/O) requests, and performing I/O operations on the RAID group in response to the I/O requests.

Abstract: Virtual block redirection clean-up is accomplished for a virtual block in a mapper tree for a logical volume that is redirected to a destination virtual block in the mapper tree. In response to redirection of the virtual block, a redirection notification is sent to each component in a set of dependent components. Each dependent component may store one or more pointers to pointers located within virtual blocks of the mapper tree. The redirected virtual block is cleaned up in response to determining that each dependent component has expressly indicated, in response to the redirection notification, that it does not store any pointer to any pointer that is located within the redirected virtual block.

Abstract: Techniques for providing metadata (and/or data) protection in a data storage system. The techniques can include storing, for a specific data page, two (2) instances of a main metadata page and one (1) instance of a corresponding alternative metadata page in a protected storage object referred to herein as the “2+1-way mirror”. For the specific data page, two (2) instances of a main metadata page can be stored on a first drive and a second drive, respectively, of the 2+1-way mirror such that the first and second drives each have a copy of the same main metadata page. Further, an instance of a corresponding alternative metadata page can be stored on a third drive of the 2+1-way mirror. The 2+1-way mirror provides protection against metadata loss due to the concurrent failure of two (2) physical drives, and further provides protection against catastrophic metadata errors and/or inconsistencies due to software failures.

Abstract: An aspect includes providing a metadata structure having a logical level that points to a virtual level and a physical level to which the virtual level points. The method also includes storing, at the virtual level, a reference counter for each of a plurality of virtual-level type storage address entries in the metadata structure, and providing a pointer in the metadata structure between each pair of a number of pairs of virtual level address entries in which corresponding pages share a set of common sectors. The reference counter tracks a number of instances in which a corresponding pointer points to a corresponding virtual level address entry. An aspect further includes storing a single instance of the common sectors at the physical level.

Abstract: A technique for performing deduplication traverses a deduplication database and assigns digest values in the database to buckets, where each bucket covers a respective range of digest values. To deduplicate a particular candidate block, the technique generates a digest from the candidate block and searches for the computed digest in a subset of the buckets, where the subset is selected based on the computed digest. If a target block providing an exact match or a suitably close partial match is found in the subset of buckets, the technique effects storage of the candidate block at least in part by providing a reference to the target block.

Abstract: An aspect of implementing amortized execution of updates for a hash table includes aggregating, within a data structure, updates to be executed for the hash table. The aggregated updates are distributed across a plurality of chunks in the data structure. An aspect also includes sorting, within each of the chunks, the updates according to respective bucket index values associated with the updates, and iteratively executing the sorted updates across each of the chunks in the data structure.

Abstract: An aspect includes splitting a table of buckets into a fixed number of domains. Each of the domains includes a corresponding subset of the buckets. An aspect also includes providing a spare bucket for each of the subsets of the buckets and providing a metadata structure for each of the domains. The metadata structure includes a head pointer that points to a first bucket of a corresponding subset of the buckets and a spare_bucket pointer that points to the spare bucket of the subset of the buckets. An aspect further includes providing a split-spare bucket pointer that interleaves, during updates to data, among the subset of buckets in the domain. Data subject to the updates is stored in the spare bucket for a corresponding one of the domains. An aspect also includes updating the head pointer and the spare_bucket pointer for corresponding domains in response to updating the data.

Abstract: A technique for managing data storage obtains a batch of chunks of data. The technique generates, using multiple pipelined instructions operating on the batch, a measure of data entropy for each of the chunks in the batch. The technique selectively compresses chunks in the batch based at least in part on the measures of data entropy generated for the respective chunks.

Abstract: A technique for managing deduplication performs partial-block matching opportunistically by leveraging information acquired during times when a storage system has available resources. The information identifies anchor blocks that are likely targets for partial-block matches, based on discovering that the anchor blocks belong to populations of blocks that have high similarity. When processing write requests, inline activities access anchor blocks that closely match newly arriving candidate blocks and perform partial-block deduplication against those anchor blocks.

Abstract: A technique for performing deduplication identifies representative sub-blocks within candidate blocks and performs sub-block matching to entries in a digest database. When a representative sub-block is matched to a differently-aligned target sub-block that belongs to a target block, the technique effectuates storage of the candidate block using the target block and a block adjacent to the target block.

Abstract: A technique for performing deduplication extends a match found in a digest database by looking forward and/or backward to adjacent data elements. The technique performs data comparisons between data blocks adjacent to a candidate block and corresponding data blocks adjacent to an identified target block. If the data comparisons indicate that an adjacent candidate block matches an adjacent target block, then the adjacent candidate block may be deduplicated to the adjacent target block without having to compute a hash value of the adjacent candidate block or to perform a separate lookup into the digest database.

Abstract: A technique for managing data storage begins at a predetermined offset relative to a chunk of data received for writing, and identifies a span of contiguous regions of the chunk that contain identical data. The technique replaces the span of contiguous regions of the chunk with a single instance of a region of the contiguous regions. The technique persistently stores a shortened version of the chunk with the single instance replacing the span of contiguous regions.

Abstract: Techniques for data processing may include: receiving a page P having a binary representation; determining a plurality of features for the page P; determining a plurality of feature hashes for the plurality of features, wherein each of the plurality of feature hashes corresponds to a different one of the plurality of features, wherein said each feature hash is determined using a hash function and in accordance with said different one of the plurality of features and an index associated with said different one of the plurality of features; and determining, in accordance with the plurality of feature hashes for the plurality of features, a similarity hash for the page P.

Abstract: Techniques for data processing may include: receiving a data block P having a binary representation; determining features for the data block P; determining, using at least one table of precomputed hash values, feature hashes for the features, wherein each of the feature hashes corresponds to a different feature, wherein each of the feature hashes is one of the precomputed hash values of the at least one table; and determining, in accordance with the feature hashes, a similarity hash for the data block P. Each feature may be a byte of P. The at least one table may be a single 3 dimensional or multiple 2 dimensional tables. Each row of a table of precomputed hash values may correspond to a single precomputed hash value. The row may include byte entries where each byte entry includes a single bit value of a precomputed hash.

Abstract: Techniques for determining unique ownership of data blocks includes selecting a first data block referenced by a first logical address of a first logical device, wherein first metadata for the first data block includes a reference count and a result used in determining sole unique ownership of the first data block; determining C1, an expected value of the result of the first metadata when the first logical device is a sole unique owner of the first data block; and performing first processing that determines whether the first logical device is the sole unique owner of the first data block. The first processing includes: determining whether C1 equals a current value of the result of the first metadata; and determining that the first logical device is the sole unique owner of the first data block if C1 equals the current value of the result of the first metadata.