Abstract: A technique for performing data deduplication operates at sub-block granularity by searching a deduplication database for a match between a candidate sub-block of a candidate block and a target sub-block of a previously-stored target block. When a match is found, the technique identifies a duplicate range shared between the candidate block and the target block and effects persistent storage of the duplicate range by configuring mapping metadata of the candidate block so that it points to the duplicate range in the target block.

Abstract: Techniques are provided for reporting space savings due to pattern matching in storage systems. For example, in one embodiment, an exemplary method comprises, when a given allocation unit in a storage system matches one or more predefined patterns, (i) setting a corresponding pattern flag for the given allocation unit, and (ii) incrementing at least one pattern counter; generating at least one snapshot of at least a portion of a file comprising the given allocation unit; and determining a range of data reduction attributed to pattern matching based on said at least one pattern counter, wherein one extreme of said range of data reduction attributed to pattern matching excludes said one or more predefined patterns in said at least one snapshot.

Abstract: A method is used in managing truncation of files of file systems. A request is received to delete a portion of a file of a file system. The file system includes a plurality of files. Metadata of the file is evaluated for determining a number of file system blocks associated with the portion of the file that are available for de-allocation. Storage space associated with the file system blocks is reported as available storage space to a user of the file.

Abstract: A method performed by a block-storage server, of storing data is described. The method includes (1) receiving, from a remote file server, data blocks to be written to persistent block storage managed by the block-storage server; (2) receiving, from the remote file server, metadata describing files to which the data blocks belong in a set of filesystems managed by the remote file server; and (3) selectively applying data reduction when storing the data blocks in the persistent block storage based, at least in part, on the received metadata. An apparatus, system, and computer program product for performing a similar method are also provided.

Abstract: A method performed by a block-storage server, of storing data is described. The method includes (1) receiving, from a remote file server, data blocks to be written to persistent block storage managed by the block-storage server; (2) receiving, from the remote file server, metadata describing a placement of the data blocks in a filesystem managed by the remote file server; and (3) organizing the data blocks within the persistent block storage based, at least in part, on the received metadata. An apparatus, system, and computer program product for performing a similar method are also provided.

Abstract: Techniques for CPU cache efficiency may include performing concurrent processing, such as for first and second data operations, in a synchronized manner that prevents loading the same data chunk into the CPU cache more than once. Processing may include synchronizing the first and second data operations with respect to a first data chunk to ensure that both the first and second data operation processing has completed prior to proceeding with performing such processing on a second data chunk. The first and second data operations may be any two of deduplication, encryption, and compression, performed inline as part of the data path. In one embodiment, the first and second data operations for the first data chunk may be performed in parallel or sequentially where neither data operation proceeds with another data chunk until processing of the first and second data operations is complete for the first data chunk.

Abstract: A technique for managing data storage includes generating entropy of blocks on a per-block basis and selectively performing inline compression on blocks based at least in part on their entropy. Entropy of a block provides a rough measure of the block's compressibility. Thus, using per-block entropy enables a storage system to steer compression decisions, e.g., whether to compress and/or how much to compress, flexibly and with high granularity, striking a balance between throughput and storage efficiency.

Abstract: A technique for performing data reduction applies deduplication principles when performing data compression, providing a form of enhanced compression. The technique obtains a chunk of data that contains multiple extents and applies deduplication actions to identify duplicate extents within the chunk. The technique marks duplicate extents in metadata. Such duplicate extents need not be compressed using conventional data compression, saving computational resources and considerable time.

Abstract: Techniques for generating data sets may include: receiving an initial buffer that achieves a compression ratio responsive to compression processing using a compression algorithm, the initial buffer including first content located at a first position in the initial buffer and including second content located at a second position in the initial buffer; and generating a data set of buffers using the initial buffer. The data set may be expected to achieve a specified deduplication ratio responsive to deduplication processing and to achieve the compression ratio responsive to compression processing using the compression algorithm. Generating the data set may include generating a first plurality of buffers where each buffer of the first plurality is not a duplicate of another buffer in the first plurality, and generating a second plurality of duplicate buffers. Each duplicate buffer may be a duplicate of a buffer in the first plurality of buffers.

Abstract: A technique for managing data storage includes generating entropy of blocks on a per-block basis and selectively performing inline compression on blocks based at least in part on their entropy. Entropy of a block provides a rough measure of the block's compressibility. Thus, using per-block entropy enables a storage system to steer compression decisions, e.g., whether to compress and/or how much to compress, flexibly and with high granularity, striking a balance between throughput and storage efficiency.

Abstract: Techniques for generating data sets may include: receiving an initial buffer that achieves a compression ratio responsive to compression processing using a compression algorithm, the initial buffer including first content located at a first position in the initial buffer and including second content located at a second position in the initial buffer; and generating a data set of buffers using the initial buffer. The data set may be expected to achieve a specified deduplication ratio responsive to deduplication processing and to achieve the compression ratio responsive to compression processing using the compression algorithm. Generating the data set may include generating a first plurality of buffers where each buffer of the first plurality is not a duplicate of another buffer in the first plurality, and generating a second plurality of duplicate buffers. Each duplicate buffer may be a duplicate of a buffer in the first plurality of buffers.

Abstract: A method is used in managing data reduction in storage systems using machine learning. A value representing a data reduction assessment for a first data block in a storage system is calculated using a hash of the data block. The value is used to train a machine learning system to assess data reduction associated with a second data block in the storage system without performing the data reduction on the second data block, where assessing data reduction associated with the second data block indicates a probability as to whether the second data block can be reduced.

Abstract: Techniques for processing data may include: receiving a plurality of data chunks for a data set; performing data deduplication processing for the plurality of data chunks; determining, in accordance with one or more criteria, whether a frequency distribution of a frequency histogram of digest byte frequencies is sufficiently uniform; and responsive to determining that the frequency distribution of the frequency histogram is not sufficiently uniform, performing processing to update data deduplication settings for the data set. Updating the data deduplication settings may include using a stronger hash algorithm and/or a larger size digest when generating subsequent digests. The data deduplication processing may include: determining, using a current hash algorithm, a plurality of digests for the plurality of data chunks of the data set; and updating the frequency histogram of digest byte frequencies for the data set in accordance the plurality of digests.

Abstract: Techniques for data processing may include: receiving a data chunk; determining a metric value denoting a degree of compressibility of the data chunk; selecting, in accordance with the metric value denoting the compressibility of the data chunk, a first size of a plurality of sizes, wherein each of the plurality of sizes denotes a different size of an amount of storage used for storing a value of said each size; and performing the data deduplication processing for the data chunk, wherein the data deduplication processing includes using a first hash value for the data chunk to determine whether the data chunk is a duplicate of another data chunk of a hash table, wherein the first hash value is stored in a storage location of the first size.

Abstract: Techniques for data processing may include: receiving a data chunk of the data set; determining, in accordance with criteria including a compressibility ratio for the data set and a cost ratio of compression computation cost and entropy computation cost, whether to activate or deactivate entropy computation for the data set, wherein the compressibility ratio is ratio of a number of compressible data chunks of the data set and a number of uncompressible data chunks of the data set; and responsive to determining to activate entropy computation for the data set, performing first processing comprising: determining an entropy value for the data chunk; and determining, in accordance with the entropy value for the data chunk, whether to compress the data chunk.

Abstract: A method is used in managing inline data compression in storage systems. A request is received to flush data of a file system to a storage device of a storage system. A compression ratio is determined for writing the data to the storage device in a compressed format. Based on the determination, a number of allocation units is determined in which a segment is apportioned. The segment comprises of a set of allocation units in the compressed format. Based on the determination, the data is written to the segment in the compressed format.

Abstract: Techniques for scavenging blocks may include: determining, in accordance with a selected option, a set of candidate upper deck file systems, wherein at least a first of the candidate upper deck file systems has storage allocated from at least one block of a lower deck file system; and performing, in accordance with the selected option, scavenging of the set of candidate upper deck file systems to attempt to free blocks of the lower deck file system. Scavenging may include issuing a request to perform hole punching of a backed free block of the first candidate upper deck file system, wherein the backed free block has first provisioned storage that is associated with a block of the lower deck file system. The selected option may be one of multiple options each specifying a different candidate set of upper deck file systems upon which hole punching is performed when selected.

Abstract: There is disclosed techniques for data reduction reporting in storage systems. For example in one embodiment, the techniques comprise update a counter denoting an amount of data reduction obtained when a data reduction feature associated with a data reduction requirement is enabled in the storage system when one or more allocation units in a storage system satisfy a data reduction requirement of the storage system. The techniques further comprise generating a snapshot of at least a portion of a file related to the said allocation units such that a first relationship exists when the snapshot is generated and a second relationship exists when the snapshot or the file is updated. The techniques also comprise determining a data reduction attributed to the data reduction feature based on an amount of data reduction denoted by the data reduction counter and an amount of sharing and/or unsharing of indirect blocks including mapping pointers in the relationship between the snapshot and the file.

Abstract: A technique for performing writes in a storage system that supports data compression tests incoming writes to determine whether the amount of data to be written is less than a predetermined threshold. If so, the storage system avoids compression and stores the amount of data as received without compressing it. If not, the storage system performs compression and stores the incoming data as a compressed extent.

Abstract: A method of accepting writes in a multilayered storage system is provided. The method includes (a) monitoring a rate of flushing of data from a first data storage component to a second data storage component; (b) setting an intake rate for the first data storage component based on the monitored flushing rate; and (c) throttling writes to the first data storage component based on the set intake rate. An apparatus, system, and computer program product for performing a similar method are also provided.