Abstract: A method is used in managing data reduction in storage systems using machine learning. A value representing a data reduction assessment for a first data block in a storage system is calculated using a hash of the data block. The value is used to train a machine learning system to assess data reduction associated with a second data block in the storage system without performing the data reduction on the second data block, where assessing data reduction associated with the second data block indicates a probability as to whether the second data block can be reduced.

Abstract: Techniques for data processing may include: receiving a data chunk; determining a metric value denoting a degree of compressibility of the data chunk; selecting, in accordance with the metric value denoting the compressibility of the data chunk, a first size of a plurality of sizes, wherein each of the plurality of sizes denotes a different size of an amount of storage used for storing a value of said each size; and performing the data deduplication processing for the data chunk, wherein the data deduplication processing includes using a first hash value for the data chunk to determine whether the data chunk is a duplicate of another data chunk of a hash table, wherein the first hash value is stored in a storage location of the first size.

Abstract: A method of assigning I/O requests to CPU cores of a data storage system includes, in a first operating mode, assigning I/O requests to CPU cores based on port affinity while maintaining a current I/O completion count, and regularly performing a first test-and-switch operation that includes (i) for a sample interval, temporarily assigning the I/O requests to the CPU cores based on core availability while obtaining a sample I/O completion count, (ii) comparing the first sample I/O completion count to the current I/O completion count, and (iii) based on the sample I/O completion count being greater than the current I/O completion count, switching to a second operating mode. In the second operating mode, I/O requests are assigned to the CPU cores based on core availability, and similar operations are performed for periodically testing whether to switch to the first operating mode.

Abstract: A method of assigning I/O requests to CPU cores of a data storage system includes, in a first operating mode, assigning I/O requests to CPU cores based on port affinity while maintaining a current I/O completion count, and regularly performing a first test-and-switch operation that includes (i) for a sample interval, temporarily assigning the I/O requests to the CPU cores based on core availability while obtaining a sample I/O completion count, (ii) comparing the first sample I/O completion count to the current I/O completion count, and (iii) based on the sample I/O completion count being greater than the current I/O completion count, switching to a second operating mode. In the second operating mode, I/O requests are assigned to the CPU cores based on core availability, and similar operations are performed for periodically testing whether to switch to the first operating mode.

Abstract: One example method includes storing a user application input/output operation (IO) in a compression cache portion of a cache memory wherein the user application IO is associated with a data block, compressing the data block to produce a compressed data block, hashing the compressed data block to create a hash that uniquely corresponds to the data block, and storing the hash in a deduplication digest portion of a cache memory, wherein the deduplication digest portion includes a plurality of groups, each of which corresponds to a particular extent to which a data block is compressible. When a compressibility of the data block exceeds a threshold, the data block is retained in the compression cache portion, and when the compressibility of the data block does not exceed the threshold, the data block is flushed from the compression cache portion to disk.

Abstract: A method is used in managing data reduction in storage systems using machine learning. A value representing a data reduction assessment for a first data block in a storage system is calculated using a hash of the data block. The value is used to train a machine learning system to assess data reduction associated with a second data block in the storage system without performing the data reduction on the second data block, where assessing data reduction associated with the second data block indicates a probability as to whether the second data block can be reduced.

Abstract: Technology for dynamically adjusting a process scheduler in a storage processor of a data storage system. An average amount of host data contained in sets of host data processed by host I/O request processing threads is calculated. An average amount of time required for each host I/O request processing thread to execute to completely process the average amount of host data contained in a set of host data is also calculated. Operation of the process scheduler in the storage processor is then adjusted to cause the process scheduler to subsequently allocate the processor in the storage processor to host I/O request processing threads in timeslices having a duration that is at least as large as the average amount of time required for each host I/O request processing thread to execute to completely process the average amount of host data contained in a set of host data.

Abstract: Techniques for data processing may include: receiving a data chunk; determining a metric value denoting a degree of compressibility of the data chunk; selecting, in accordance with the metric value denoting the compressibility of the data chunk, a first size of a plurality of sizes, wherein each of the plurality of sizes denotes a different size of an amount of storage used for storing a value of said each size; and performing the data deduplication processing for the data chunk, wherein the data deduplication processing includes using a first hash value for the data chunk to determine whether the data chunk is a duplicate of another data chunk of a hash table, wherein the first hash value is stored in a storage location of the first size.

Abstract: A method is used in managing data reduction in storage systems using machine learning. A value representing a data reduction assessment for a first data block in a storage system is calculated using a hash of the data block. The value is used to train a machine learning system to assess data reduction associated with a second data block in the storage system without performing the data reduction on the second data block, where assessing data reduction associated with the second data block indicates a probability as to whether the second data block can be reduced.

Abstract: A method is used in managing data reduction in storage systems using machine learning. A value representing a data reduction assessment for a first data block in a storage system is calculated using a hash of the data block. The value is used to train a machine learning system to assess data reduction associated with a second data block in the storage system without performing the data reduction on the second data block, where assessing data reduction associated with the second data block indicates a probability as to whether the second data block can be reduced.

Abstract: One example method includes storing a user application input/output operation (IO) in a compression cache portion of a cache memory wherein the user application IO is associated with a data block, compressing the data block to produce a compressed data block, hashing the compressed data block to create a hash that uniquely corresponds to the data block, and storing the hash in a deduplication digest portion of a cache memory, wherein the deduplication digest portion includes a plurality of groups, each of which corresponds to a particular extent to which a data block is compressible. When a compressibility of the data block exceeds a threshold, the data block is retained in the compression cache portion, and when the compressibility of the data block does not exceed the threshold, the data block is flushed from the compression cache portion to disk.

Abstract: Techniques for CPU cache efficiency may include performing concurrent processing, such as for first and second data operations, in a synchronized manner that prevents loading the same data chunk into the CPU cache more than once. Processing may include synchronizing the first and second data operations with respect to a first data chunk to ensure that both complete prior to proceeding with performing such processing on a second data chunk. The first and second data operations for a chunk may be deduplication digest computation and entropy computation performed inline as part of the data path. If the chunk cannot be deduplicated, the entropy value may be used with an adaptive entropy threshold to determine whether to compress the chunk inline. The entropy value may be determined using less than all bytes in the chunk. The chunk's entropy value may be determined based on a data set entropy value.

Abstract: Techniques are described for compressing cache pages from an LRU (Least-Recently-Used) queue so that data takes longer to age off and be removed from the cache. This increases the likelihood that data will be available within the cache upon subsequent re-access, reducing the need for costly disk accesses due to cache misses.