Abstract: Techniques for determining data reduction options may include: receiving data reduction statistics for a data set including a first value of a first statistic denoting an amount of data reduction obtained for the data set when compression is enabled, a second value of a second statistic denoting an amount of data reduction obtained for the data set when deduplication is enabled, and a third value of a third statistic denoting an overlap in data reduction contribution when both compression and deduplication are enabled; and determining, in accordance with the data reduction statistics, a first setting denoting a current data reduction option enabled for the data set. A Venn diagram provided on a user interface display may illustrate data reduction benefits for the data set based on the data reduction statistics. Data reduction benefits for the data set may be reassessed to determine whether to modify the current data reduction option.

Abstract: A technique for managing storage space in a data storage system implements data objects in respective files. In response to an instruction to convert a set of such files from thin to thick or from thick to thin, the data storage system checks whether it has enough available storage space to accommodate the set of files if conversion were to proceed. If so, conversion is performed, and each of the set of files is converted from thin to thick or from thick to thin, in accordance with the instruction.

Abstract: Embodiments are directed to techniques for simplifying and automating the process of transitioning a storage object to use inline compression either on the same machine or migrated to a new machine. This may be accomplished by determining the raw compressibility of the data of a storage obj ect, estimating the interaction between the compressibility of the data and a structure of the inline compression feature, and automatically performing the upgrade or migration if the expected compression savings exceeds a threshold. Some embodiments further speed the process and decrease the resources by determining the raw compressibility through sampling. Embodiments are directed to a method, apparatus, system, and computer program product for performing these techniques.

Abstract: Techniques for effectively measuring the amount of space saved due to deduplication and selectively turning deduplication off for a filesystem if the savings are too low are provided. The techniques include maintaining both a lower bound count and an upper bound count for block savings due exclusively to deduplication rather than due to snapshots and using an average of the lower and upper bounds to estimate the actual savings due to deduplication in order decide whether to continue operating deduplication or not.

Abstract: A method is used in managing data reduction in storage systems using machine learning. A value representing a data reduction assessment for a first data block in a storage system is calculated using a hash of the data block. The value is used to train a machine learning system to assess data reduction associated with a second data block in the storage system without performing the data reduction on the second data block, where assessing data reduction associated with the second data block indicates a probability as to whether the second data block can be reduced.

Abstract: Techniques for determining data reduction options may include: receiving data reduction statistics for a data set including a first value of a first statistic denoting an amount of data reduction obtained for the data set when compression is enabled, a second value of a second statistic denoting an amount of data reduction obtained for the data set when deduplication is enabled, and a third value of a third statistic denoting an overlap in data reduction contribution when both compression and deduplication are enabled; and determining, in accordance with the data reduction statistics, a first setting denoting a current data reduction option enabled for the data set. A Venn diagram provided on a user interface display may illustrate data reduction benefits for the data set based on the data reduction statistics. Data reduction benefits for the data set may be reassessed to determine whether to modify the current data reduction option.

Abstract: Techniques for scavenging blocks may include: determining, in accordance with a selected option, a set of candidate upper deck file systems, wherein at least a first of the candidate upper deck file systems has storage allocated from at least one block of a lower deck file system; and performing, in accordance with the selected option, scavenging of the set of candidate upper deck file systems to attempt to free blocks of the lower deck file system. Scavenging may include issuing a request to perform hole punching of a backed free block of the first candidate upper deck file system, wherein the backed free block has first provisioned storage that is associated with a block of the lower deck file system. The selected option may be one of multiple options each specifying a different candidate set of upper deck file systems upon which hole punching is performed when selected.

Abstract: A technique for facilitating file system operations in a data storage system provides multiple version family databases. Each version family database identifies members of a respective version family, i.e., a respective set of files in a file system that are related by file snapping. When a snap of a file is created, the file system updates a version family database for the version family that includes the file, such that the version family database identifies both the file and the snap.

Abstract: Techniques for a managing concurrent I/Os in a file system may include receiving a sequence of conflicting I/O lists of write data stored in a cache, the sequence specifying a sequential order in which the I/O lists are to be flushed to a file stored on non-volatile storage; determining a first I/O list of the sequence having a conflict with a second I/O list of the sequence, wherein the conflict between the first I/O list and the second I/O list is a first common block written to by both the first and second I/O lists; and performing first processing that modifies the first I/O list and the second I/O list to remove the conflict.

Abstract: Techniques for data processing may include: computing an entropy value for the chunk; determining, in accordance with the entropy value for the data chunk, whether the data chunk is compressible; and responsive to determining the data chunk is compressible based on the entropy value for the chunk, compressing the data chunk. The entropy value may be determined using counters for data items where the counters denote current frequencies of different allowable data items in the data chunk; and performing second processing using the counters to determine an entropy value for the data chunk, wherein said second processing includes selecting a precomputed binary logarithmic value from a table for each of the counters. The table may include integer representations of binary logarithmic values. The second processing may include loading multiple data items of the chunk into a register, extracting each data item from the register and incrementing a corresponding counter.

Abstract: Techniques for CPU cache efficiency may include performing concurrent processing, such as for first and second data operations, in a synchronized manner that prevents loading the same data chunk into the CPU cache more than once. Processing may include synchronizing the first and second data operations with respect to a first data chunk to ensure that both complete prior to proceeding with performing such processing on a second data chunk. The first and second data operations for a chunk may be deduplication digest computation and entropy computation performed inline as part of the data path. If the chunk cannot be deduplicated, the entropy value may be used with an adaptive entropy threshold to determine whether to compress the chunk inline. The entropy value may be determined using less than all bytes in the chunk. The chunk's entropy value may be determined based on a data set entropy value.

Abstract: Techniques for data processing may include: determining a data layout for a configuration of counters stored in registers, wherein each of the registers is configured to store at least two counters, and each counter is associated with a particular data item allowable in the data set and denotes a current frequency of the particular data item; receiving data items of a data chunk of the data set; for each data item received, performing processing including: determining a first of the counters corresponding to the data item, wherein the first counter is stored in a first of the registers and denotes a current frequency of the data item; and incrementing the first counter stored in the first register by one; and determining, in accordance with the counters stored in the registers, an entropy value for the data chunk.

Abstract: There are disclosed techniques for reporting space savings due to compression in storage systems. In one embodiment, the techniques comprise receiving a request to write data to a data storage system. The techniques also comprise performing a compression of the data associated with the request. The techniques further comprise determining a difference between a number of allocation units needed if no compression of the data associated with the request and a number of allocation units allocated to service the compressed data associated with the request. The techniques still further comprise providing a data reduction attributed to compression based on the difference.

Abstract: A technique for repairing damage to a file system stores compression headers in association with respective extents of in-line compressed data. A repair utility reads contents from the compression headers and applies the contents to repair damaged metadata structures that the file system uses to access the associated extents. In this fashion, the repair utility restores access to compressed data, which would otherwise be lost.

Abstract: Techniques for scavenging blocks may include: determining, in accordance with a selected option, a set of candidate upper deck file systems, wherein at least a first of the candidate upper deck file systems has storage allocated from at least one block of a lower deck file system; and performing, in accordance with the selected option, scavenging of the set of candidate upper deck file systems to attempt to free blocks of the lower deck file system. Scavenging may include issuing a request to perform hole punching of a backed free block of the first candidate upper deck file system, wherein the backed free block has first provisioned storage that is associated with a block of the lower deck file system. The selected option may be one of multiple options each specifying a different candidate set of upper deck file systems upon which hole punching is performed when selected.

Abstract: Embodiments are directed to techniques for simplifying and automating the process of transitioning a storage object to use inline compression either on the same machine or migrated to a new machine. This may be accomplished by determining the raw compressibility of the data of a storage obj ect, estimating the interaction between the compressibility of the data and a structure of the inline compression feature, and automatically performing the upgrade or migration if the expected compression savings exceeds a threshold. Some embodiments further speed the process and decrease the resources by determining the raw compressibility through sampling. Embodiments are directed to a method, apparatus, system, and computer program product for performing these techniques.

Abstract: A storage processor in a data storage system includes a compression selection component that selects a data compression component to be used to compress host I/O data that is flushed from a persistent cache of the storage processor based on a current fullness level of the persistent cache. The compression selection component selects compression components implementing compression algorithms having relatively lower compression ratios for relatively higher current fullness levels of the persistent cache, and selects compression components implementing compression algorithms having relatively higher compression ratios for relatively lower current fullness levels of the persistent cache.

Abstract: Techniques for a managing concurrent I/Os in a file system may include receiving a sequence of conflicting I/O lists of write data stored in a cache, the sequence specifying a sequential order in which the I/O lists are to be flushed to a file stored on non-volatile storage; determining a first I/O list of the sequence having a conflict with a second I/O list of the sequence, wherein the conflict between the first I/O list and the second I/O list is a first common block written to by both the first and second I/O lists; and performing first processing that modifies the first I/O list and the second I/O list to remove the conflict.

Abstract: In response to a cache flush event indicating that host data accumulated in a cache of a storage processor of a data storage system is to be flushed to a lower deck file system, an aggregation set of blocks is formed within the cache, and a digest calculation group is selected from within the aggregation set. Hardware vector processing logic is caused to simultaneously calculate crypto-digests from the blocks in the digest calculation group. If one of the resulting crypto-digests matches a previously generated crypto-digest, deduplication is performed that i) causes the lower deck file system to indicate the block of data from which the previously generated crypto-digest was generated and ii) discards the block that corresponds to the matching crypto-digest. Objects required by a digest generation component may be allocated in a just in time manner to avoid having to manage a pool of pre-allocated objects.

Abstract: Techniques for data processing a data set may comprise: performing first processing that forms a first compression unit, wherein the first compression unit includes a data chunks including a first data chunk having a first entropy value less than an entropy threshold, the first processing including: receiving a second data chunk; determining, in accordance with criteria, whether to add the second data chunk to the first compression unit; and responsive to determining to add the second data chunk to the first compression unit, adding the second data chunk to the first compression unit; and compressing the first compression unit as a single compressible unit. The second chunk may be added if its entropy value is less than the entropy threshold and if entropy values of the first and second chunks are similar. The second chunk may be added if the resulting compression unit provides sufficient storage/compression benefit.