Abstract: Techniques for data processing may include: determining one or more sub-blocks of a target block that match one or more sub-blocks of a candidate block; creating a shared sub-block mapping (SSM) structure having a plurality of entries, wherein each of the plurality of entries corresponds to a different one of the sub-blocks in the candidate block and wherein a value stored in said each entry, corresponding to one of the sub-blocks of the candidate block, identifies a sub-block of the target block matching said one sub-block of the candidate block; and storing the candidate block as a deduplicated block sharing at least one sub-block with the target block. The SSM structure may be stored as a metadata structure of the candidate block to identify deduplicated sub-blocks of the candidate block and to identify sub-blocks of the target block providing content for the deduplicated sub-blocks of the candidate block.

Abstract: Techniques for processing data may include: receiving a candidate data block; computing a distance using a distance function, wherein the distance denotes a measurement of similarity between the candidate data block and a target data block; and determining, using the distance, whether to perform data deduplication of the candidate data block with respect to the target data block to identify at least one sub-block of the candidate data block that is a duplicate of at least one sub-block of the target data block. The distance may be computed using a bit-wise logical exclusive-or operation of the contents of the candidate data block and the target data block. The distance may be computed using a bit-wise logical exclusive-or operation of digests computed for the candidate and target data blocks using a distance preserving hash function. The target and candidate block may be similar if the distance is less than a threshold.

Abstract: Techniques for processing data may include: receiving a candidate block; performing partial deduplication processing of the candidate block; receiving a second candidate block subsequent to performing partial deduplication processing for the candidate block; and performing first processing to determine whether to perform promotion processing for the entry, The partial deduplication processing may include: partially deduplicating at least one sub-block of the candidate block; and creating an entry in a deduplication database for the candidate block, wherein the entry includes a digest of the candidate block and the entry denotes a potential target block having the digest, and wherein the entry includes a counter that tracks a number of missed full block deduplications between the potential target block and subsequently processed candidate blocks. The promotion processing promotes the potential target block, having the first digest of the entry, to a new target block.

Abstract: Techniques for processing data may include: receiving a plurality of data chunks for a data set; performing data deduplication processing for the plurality of data chunks; determining, in accordance with one or more criteria, whether a frequency distribution of a frequency histogram of digest byte frequencies is sufficiently uniform; and responsive to determining that the frequency distribution of the frequency histogram is not sufficiently uniform, performing processing to update data deduplication settings for the data set. Updating the data deduplication settings may include using a stronger hash algorithm and/or a larger size digest when generating subsequent digests. The data deduplication processing may include: determining, using a current hash algorithm, a plurality of digests for the plurality of data chunks of the data set; and updating the frequency histogram of digest byte frequencies for the data set in accordance the plurality of digests.

Abstract: Techniques for data processing may include: receiving a candidate block including a plurality of uniformly-sized sub-blocks, wherein a tag is stored at a first location in the candidate block; performing data deduplication processing of the candidate block, wherein the data deduplication processing excludes content stored from a first offset to a second offset corresponding to the first location; determining whether at least one sub-block of the candidate block has been deduplicated by the data deduplication processing; and responsive to determining that at least one sub-block of the candidate block has been deduplicated, storing the candidate block as a deduplicated data block having at least one sub-block matching an existing target sub-block, wherein a tag descriptor describing the tag is stored and associated with the candidate block, such as in block-level metadata of the candidate block. The tag descriptor may include tag content and tag location information.

Abstract: A technique for performing in-line compression includes receiving data into a data log that temporarily holds the data and aggregating the data into batches, where each batch includes multiple blocks of received data. For each batch of data, a storage system performs a compression operation, which proceeds block-by-block, compressing each block and comparing a total compressed size of all blocks compressed so far against a budget. The storage system increments the budget for successive blocks, such that a per-block budget is greater for a first block in the batch than it is for a last block in the batch, thus allowing earlier blocks to meet budget even if they are relatively incompressible.

Abstract: Techniques for data processing may include: receiving a candidate block including a plurality of uniformly-sized sub-blocks, wherein a tag is stored at a first location in the candidate block; performing data deduplication processing of the candidate block, wherein the data deduplication processing excludes content stored from a first offset to a second offset corresponding to the first location; determining whether at least one sub-block of the candidate block has been deduplicated by the data deduplication processing; and responsive to determining that at least one sub-block of the candidate block has been deduplicated, storing the candidate block as a deduplicated data block having at least one sub-block matching an existing target sub-block, wherein a tag descriptor describing the tag is stored and associated with the candidate block, such as in block-level metadata of the candidate block. The tag descriptor may include tag content and tag location information.

Abstract: Techniques for data processing may include: receiving a data chunk and an associated digest; and performing data deduplication processing for the data chunk comprising: determining whether there is an existing entry in a deduplication digest cache for the digest; and responsive to determining there is no existing entry in the deduplication digest cache, performing processing including: determining whether there is an existing entry in a mapping structure for the digest, the mapping structure mapping digests to associated pages of related entries in a deduplication data store; and responsive to determining there is an existing entry in the mapping structure, performing second processing including: obtaining, from the existing entry, a page mapped to the digest; and loading the page of entries from the deduplication data store into the deduplication digest cache. At least some entries of the page may be prefetched and loaded into the deduplication digest cache prior to use.

Abstract: A method is used in managing data reduction in storage systems using machine learning. A value representing a data reduction assessment for a first data block in a storage system is calculated using a hash of the data block. The value is used to train a machine learning system to assess data reduction associated with a second data block in the storage system without performing the data reduction on the second data block, where assessing data reduction associated with the second data block indicates a probability as to whether the second data block can be reduced.

Abstract: A method is used in managing inline data compression and deduplication in storage systems. A block of data from data stored in a cache of a storage system is identified based on entropy. Entropy of the block of data is compared with a first threshold value. Based on the comparison, the block of data is either deduplicated or compressed without deduplication.

Abstract: There is disclosed techniques for use in managing data storage in storage systems. For example, in one embodiment, there is disclosed a method comprising receiving a request to store data of a data object in a storage system. The method also comprising determining that at least a portion of the data is to be stored in an uncompressed format in the storage system in response to receiving the request. The method also comprising storing at least a portion of the data in the uncompressed format in an allocation unit of a segment in the storage system such that the stored data in the uncompressed format emulates stored data in a compressed format based on the said determination.

Abstract: There is disclosed techniques for data reduction reporting in storage systems. For example in one embodiment, the techniques comprise update a counter denoting an amount of data reduction obtained when a data reduction feature associated with a data reduction requirement is enabled in the storage system when one or more allocation units in a storage system satisfy a data reduction requirement of the storage system. The techniques further comprise generating a snapshot of at least a portion of a file related to the said allocation units such that a first relationship exists when the snapshot is generated and a second relationship exists when the snapshot or the file is updated. The techniques also comprise determining a data reduction attributed to the data reduction feature based on an amount of data reduction denoted by the data reduction counter and an amount of sharing and/or unsharing of indirect blocks including mapping pointers in the relationship between the snapshot and the file.

Abstract: Techniques for determining data reduction options may include: receiving first data reduction information regarding compression and deduplication of chunks of a data set; determining, in accordance with the first data reduction information for the data set, first settings denoting whether compression and deduplication are enabled or disabled for the data set; receiving, during a first time period when the first settings are effective, writes directed to the data set; receiving second data reduction information regarding compression and deduplication of chunks of the data set modified by writes during the first time period; and determining, in accordance with the second plurality of data reduction statistics for the data set, second settings denoting whether compression and deduplication are enabled or disabled for the data set.

Abstract: Techniques for data processing may include: receiving a data chunk; determining a metric value denoting a degree of compressibility of the data chunk; selecting, in accordance with the metric value denoting the compressibility of the data chunk, a first size of a plurality of sizes, wherein each of the plurality of sizes denotes a different size of an amount of storage used for storing a value of said each size; and performing the data deduplication processing for the data chunk, wherein the data deduplication processing includes using a first hash value for the data chunk to determine whether the data chunk is a duplicate of another data chunk of a hash table, wherein the first hash value is stored in a storage location of the first size.

Abstract: Techniques for data processing a data set may comprise: performing first processing that forms a first compression unit, wherein the first compression unit includes a data chunks including a first data chunk having a first entropy value less than an entropy threshold, the first processing including: receiving a second data chunk; determining, in accordance with criteria, whether to add the second data chunk to the first compression unit; and responsive to determining to add the second data chunk to the first compression unit, adding the second data chunk to the first compression unit; and compressing the first compression unit as a single compressible unit. The second chunk may be added if its entropy value is less than the entropy threshold and if entropy values of the first and second chunks are similar. The second chunk may be added if the resulting compression unit provides sufficient storage/compression benefit.

Abstract: Techniques for data processing may include: receiving a data chunk of the data set; determining, in accordance with criteria including a compressibility ratio for the data set and a cost ratio of compression computation cost and entropy computation cost, whether to activate or deactivate entropy computation for the data set, wherein the compressibility ratio is ratio of a number of compressible data chunks of the data set and a number of uncompressible data chunks of the data set; and responsive to determining to activate entropy computation for the data set, performing first processing comprising: determining an entropy value for the data chunk; and determining, in accordance with the entropy value for the data chunk, whether to compress the data chunk.

Abstract: A technique for managing cache in a storage system that supports data deduplication renders each of a set of data blocks as multiple sub-blocks and loads a cache-resident digest database on a per-block basis, selectively creating new digest entries in the database for all sub-blocks in a block, but only for blocks that contain no duplicate sub-blocks. Sub-blocks of blocks containing duplicates are excluded. By limiting digest entries to sub-blocks of blocks that contain no duplicates, the storage system limits the size of the digest database, and thus of the cache, while also biasing the contents of the digest database toward entries that are likely to produce deduplication matches in the future.

Abstract: A technique for performing data deduplication operates at sub-block granularity by searching a deduplication database for a match between a candidate sub-block of a candidate block and a target sub-block of a previously-stored target block. When a match is found, the technique identifies a duplicate range shared between the candidate block and the target block and effects persistent storage of the duplicate range by configuring mapping metadata of the candidate block so that it points to the duplicate range in the target block.

Abstract: Techniques for scavenging blocks may include: determining, in accordance with a selected option, a set of candidate upper deck file systems, wherein at least a first of the candidate upper deck file systems has storage allocated from at least one block of a lower deck file system; and performing, in accordance with the selected option, scavenging of the set of candidate upper deck file systems to attempt to free blocks of the lower deck file system. Scavenging may include issuing a request to perform hole punching of a backed free block of the first candidate upper deck file system, wherein the backed free block has first provisioned storage that is associated with a block of the lower deck file system. The selected option may be one of multiple options each specifying a different candidate set of upper deck file systems upon which hole punching is performed when selected.

Abstract: A method is used in managing data reduction in storage systems using machine learning. A value representing a data reduction assessment for a first data block in a storage system is calculated using a hash of the data block. The value is used to train a machine learning system to assess data reduction associated with a second data block in the storage system without performing the data reduction on the second data block, where assessing data reduction associated with the second data block indicates a probability as to whether the second data block can be reduced.