Abstract: A method, computer program product, and computer system for identifying, by a computing device, a list of objects containing a plurality of physical layer blocks (PLBs). One or more next PLBs of the plurality of PLBs may be allocated from a selected free object of the list of objects. One or more additional free objects from the list of objects may be generated. Garbage collection may be performed between an inactive object of the plurality of objects and the selected free object.

Abstract: A method of data reduction in a block-based data storage system includes selecting a starting position in a block based on a deterministic function of block data content. Then for an unaligned block beginning at the selected starting position, a block digest (e.g., block hash) is generated and compared with stored block digests of stored data blocks. If there is a match, and the stored block matches the unaligned block, then a reference to the stored block is stored in place of the unaligned block, and otherwise the unaligned block and a corresponding digest are stored. The storing of references to already stored blocks, without the constraint of observing aligned-block boundaries, realizes increased savings of physical storage space.

Abstract: A technique of performing data reduction includes, upon detecting a match between similarity hashes of a candidate dataset and a target dataset, evaluating an adjacent candidate dataset and an adjacent target dataset for similarity with each other and, in response to confirming such similarity, performing data reduction of the adjacent candidate dataset with reference to the adjacent target dataset.

Abstract: Determining and using deduplication estimates may include: determining two deduplication sample indexes (DSIs) for two logical device sets each including one or more logical devices, determining a Jaccard Similarity for the two DSIs, wherein the Jaccard Similarity denotes a measurement of similarity and mutual deduplication between the two logical device sets; determining, in accordance with one or more criteria, whether the two logical device sets should be located in different data storage systems or a same data storage system that performs data deduplication, wherein the one or more criteria uses the Jaccard Similarity in determining whether to locate the two logical device sets in the same data storage system or the different data storage systems; and responsive to determining that the two logical device sets should be located in the same data storage system, locating the two logical device sets in the same data storage system.

Abstract: Techniques for data processing may include: receiving a page P having a binary representation; determining a plurality of features for the page P; determining a plurality of feature hashes for the plurality of features, wherein each of the plurality of feature hashes corresponds to a different one of the plurality of features, wherein said each feature hash is determined using a hash function and in accordance with said different one of the plurality of features and an index associated with said different one of the plurality of features; and determining, in accordance with the plurality of feature hashes for the plurality of features, a similarity hash for the page P.

Abstract: An aspect includes splitting a table of buckets into a fixed number of domains. Each of the domains includes a corresponding subset of the buckets. An aspect also includes providing a spare bucket for each of the subsets of the buckets and providing a metadata structure for each of the domains. The metadata structure includes a head pointer that points to a first bucket of a corresponding subset of the buckets and a spare_bucket pointer that points to the spare bucket of the subset of the buckets. An aspect further includes providing a split-spare bucket pointer that interleaves, during updates to data, among the subset of buckets in the domain. Data subject to the updates is stored in the spare bucket for a corresponding one of the domains. An aspect also includes updating the head pointer and the spare_bucket pointer for corresponding domains in response to updating the data.

Abstract: Techniques for data processing may include: receiving a data block P having a binary representation; determining features for the data block P; determining, using at least one table of precomputed hash values, feature hashes for the features, wherein each of the feature hashes corresponds to a different feature, wherein each of the feature hashes is one of the precomputed hash values of the at least one table; and determining, in accordance with the feature hashes, a similarity hash for the data block P. Each feature may be a byte of P. The at least one table may be a single 3 dimensional or multiple 2 dimensional tables. Each row of a table of precomputed hash values may correspond to a single precomputed hash value. The row may include byte entries where each byte entry includes a single bit value of a precomputed hash.

Abstract: A method, computer program product, and computing system for storing data in one or more buckets of a plurality of buckets in a first level. The data of the one or more buckets may be de-staged to a plurality of buckets in a second level. The one or more buckets may be invalidated in the first level. A range of valid buckets in the first level may be defined based upon, at least in part, invalidating the one or more buckets in the first level.

Abstract: An aspect of determining per volume exclusive capacity in a deduplication system includes setting a percentage of a population of pages for selection. For each of the pages, an aspect includes selecting a page in the population, providing a data segment facilitating multiple references of the segment by at least one storage entity, maintaining counts corresponding with each segment in the page, and determining exclusive ownership of the page based on the counts and a key value of one of a plurality of storage entities.

Abstract: Dictionary-based compression is performed to compress data units using a similar data unit as the base unit (i.e., dictionary) for each candidate data unit. Similarity may be determined between data units by applying a locality-sensitive hashing scheme to each candidate data unit to produce a hash value, and by determining whether there is a matching value in a hash index of hash values for existing data units on the system. If there is a matching hash value, the candidate data unit may be compressed using the data unit corresponding to the matching hash value as the dictionary. Only a representative portion of the data unit may be hashed to produce the hash value, the portion comprised of chunks of the data unit, where each chunk is a continuous, uninterrupted section of data. The chunks themselves may not be (in some embodiments likely are not) contiguous to one another.

Abstract: In a method used for dictionary-based compression in a block-based storage system, a stored block of data that is similar to a received block of data is identified. A dictionary based on the stored block of data is determined. The received block of data is compressed based on the dictionary based on the stored block of data. The compressed, received block of data is stored with an association to the stored block of data.

Abstract: A technique for validating metadata includes creating log entries for virtualization structures pointed to by mapping pointers in a mapping tree and processing the log entries in multiple passes. A current pass validates a current level of redirection and creates new log entries to be processed during a next pass. The new log entries represent a next level of redirection, and as many next passes are processed in sequence as there are next levels of redirection.

Abstract: A technique for managing data storage includes generating digests of chunks of data. The digests are arranged to provide similar results for similar chunks but to weight contributions from different regions of chunks differently based on their entropy. Chunks that differ in low-entropy regions tend to produce more similar digests than do chunks that differ in high-entropy regions. The technique further includes identifying similar chunks based on similarity of their digests and preforming data reduction on one chunk by reference to another, storing any differences between the chunks separately.

Abstract: Techniques for caching may include: determining an update to a first data page of a first cache on a first node, wherein a second node includes a second cache and wherein the second cache includes a copy of the first data page; determining, in accordance with one or more criteria, whether to send the update from the first node to the second node; responsive to determining, in accordance with the one or more criteria, to send the update, sending the update from the first node to the second node; and responsive to determining not to send the update, sending an invalidate request from the first node to the second node, wherein the invalidate request instructs the second node to invalidate the copy of the first data page stored in the second cache of the second node.

Abstract: At least one processing device is configured to select a virtual block corresponding to a physical block in a storage system, wherein the virtual block comprises a plurality of pages with one or more pages of the virtual block respectively pointing to one or more data portions in the physical block. The at least one processing device is also configured to identify a target physical block and one or more virtual blocks corresponding to the target physical block. The at least one processing device is further configured to move at least one data portion from the physical block to the target physical block. Still further, the at least one processing device is configured to redirect the at least one page of the virtual block corresponding to the at least one moved data portion to one of the one or more virtual blocks corresponding to the target physical block.

Abstract: Determining and using deduplication estimates may include: determining two deduplication sample indexes (DSIs) for two logical device sets each including one or more logical devices, determining a Jaccard Similarity for the two DSIs, wherein the Jaccard Similarity denotes a measurement of similarity and mutual deduplication between the two logical device sets; determining, in accordance with one or more criteria, whether the two logical device sets should be located in different data storage systems or a same data storage system that performs data deduplication, wherein the one or more criteria uses the Jaccard Similarity in determining whether to locate the two logical device sets in the same data storage system or the different data storage systems; and responsive to determining that the two logical device sets should be located in the same data storage system, locating the two logical device sets in the same data storage system.

Abstract: A method is used in improving write performance in a storage system. Data is stored on a first tier of storage. A modification to the data is stored on a second tier of storage, the second tier being higher than the first tier. Setting an indicator identifies which data is valid.

Abstract: A technique for performing deduplication extends a match found in a digest database by looking forward and/or backward to adjacent data elements. The technique performs data comparisons between data blocks adjacent to a candidate block and corresponding data blocks adjacent to an identified target block. If the data comparisons indicate that an adjacent candidate block matches an adjacent target block, then the adjacent candidate block may be deduplicated to the adjacent target block without having to compute a hash value of the adjacent candidate block or to perform a separate lookup into the digest database.

Abstract: A technique for managing data storage includes generating digests of chunks of data. The digests are arranged to provide similar results for similar chunks but to weight contributions from different regions of chunks differently based on their entropy. Chunks that differ in low-entropy regions tend to produce more similar digests than do chunks that differ in high-entropy regions. The technique further includes identifying similar chunks based on similarity of their digests and preforming data reduction on one chunk by reference to another, storing any differences between the chunks separately.

Abstract: A technique for managing deduplication performs partial-block matching opportunistically by leveraging information acquired during times when a storage system has available resources. The information identifies anchor blocks that are likely targets for partial-block matches, based on discovering that the anchor blocks belong to populations of blocks that have high similarity. When processing write requests, inline activities access anchor blocks that closely match newly arriving candidate blocks and perform partial-block deduplication against those anchor blocks.