Abstract: Techniques for processing data may include: receiving a candidate data block; computing a distance using a distance function, wherein the distance denotes a measurement of similarity between the candidate data block and a target data block; and determining, using the distance, whether to perform data deduplication of the candidate data block with respect to the target data block to identify at least one sub-block of the candidate data block that is a duplicate of at least one sub-block of the target data block. The distance may be computed using a bit-wise logical exclusive-or operation of the contents of the candidate data block and the target data block. The distance may be computed using a bit-wise logical exclusive-or operation of digests computed for the candidate and target data blocks using a distance preserving hash function. The target and candidate block may be similar if the distance is less than a threshold.

Abstract: A method is used in managing inline data de-duplication in storage systems. The method receives a request to write data at a logical address of a file in a file system of a storage system. The method determines whether the data can be de-duplicated to matching data residing on the storage system in a compressed format. Based on the determination, the method uses a block mapping pointer associated with the matching data to de-duplicate the data. The block mapping pointer includes a block mapping of a set of compressed data extents and information regarding location of the matching data within the set of compressed data extents.

Abstract: A technique for managing user data in a storage system includes accessing RAID metadata to identify user data that the storage system backs with broken RAID arrays. The technique further includes marking metadata that points to at least some of that user data to identify such user data as uncorrectable.

Abstract: Embodiments are directed to techniques for performing deduplication. A method includes (a) obtaining a digest of a data block logically-positioned within a filesystem, the digest providing a hash value of data of the data block, (b) searching a Most Wanted Digest Cache (MWDC) within system memory for the digest, (c) locating an entry in the MWDC using the digest, wherein this locating indicates that the data block has the same data as another data block located elsewhere within the filesystem, the other data block having been previously persistently-stored, the entry having been added to the MWDC in response to the other data block having been deduplicated at least a plurality number of times, (d) locating a mapping structure referenced by the entry located from the MWDC, the mapping structure providing metadata about the other data block, and (e) deduplicating the data block and the other data block with reference to the located mapping structure.

Abstract: A technique for managing a deduplication digest cache (DDC) includes assigning each digest entry of the DDC to one of multiple entry lists, scoring the entry lists based at least in part on the deduplicability of the data represented by the digest entries in the entry lists, and selecting for eviction from the DDC entries assigned to the lowest-scoring entry list. In this manner, entries assigned to entry lists that perform more poorly in terms of deduplicability tend to be evicted more quickly than entries assigned to entry lists that perform better.

Abstract: A technique for managing user data in a storage system includes accessing RAID metadata to identify user data that the storage system backs with broken RAID arrays. The technique further includes marking metadata that points to at least some of that user data to identify such user data as uncorrectable.

Abstract: A storage processor in a data storage system includes a compression selection component that selects a data compression component to be used to compress host I/O data that is flushed from a persistent cache of the storage processor based on a current fullness level of the persistent cache. The compression selection component selects compression components implementing compression algorithms having relatively lower compression ratios for relatively higher current fullness levels of the persistent cache, and selects compression components implementing compression algorithms having relatively higher compression ratios for relatively lower current fullness levels of the persistent cache.

Abstract: A technique for managing metadata in a data storage system includes receiving a set of data to be stored in a file system of the data storage system and calculating values of metadata blocks that the file system will use to organize the set of data in the file system. The technique aggregates values of the metadata blocks and sends them in a single, atomic transaction to a persistent cache. The transaction either succeeds, in which case the persistent cache stores the values of all of the metadata blocks, or it fails, in which case the persistent cache stores none of the values of the metadata blocks. Over time, the persistent cache flushes the values of the metadata blocks to a set of non-volatile storage devices that back the metadata blocks in the data storage system.

Abstract: A data storage system in which a transaction is generated that indicates at least one data block of a logical volume to be written to non-volatile data storage of a data, and in which the logical volume is accessible to multiple nodes in the data storage system. A system-wide lock is obtained for each data block indicated by the transaction. A new generation identifier is then created that is equal to a last transaction identifier that was created and stored during processing of a previously completed transaction. Each data block indicated by the transaction is stored into the non-volatile data storage of the data storage system together with the new generation identifier and the last transaction identifier is updated before each system-wide lock on each data block indicated by the transaction is released.

Abstract: A method is used in ensuring data integrity in data storage. A set of physical pages that include data for a virtual page is identified on a storage device. A set of reconstructed pages is generated from other storage devices, and each reconstructed page corresponds to a page in the set of physical pages. One or more pages from the set of physical pages and the set of reconstructed pages with data yielding a checksum that matches a checksum for the virtual page is identified, and data corresponding to the virtual page from the identified one or more pages is obtained.

Abstract: A method of assigning I/O requests to CPU cores of a data storage system includes, in a first operating mode, assigning I/O requests to CPU cores based on port affinity while maintaining a current I/O completion count, and regularly performing a first test-and-switch operation that includes (i) for a sample interval, temporarily assigning the I/O requests to the CPU cores based on core availability while obtaining a sample I/O completion count, (ii) comparing the first sample I/O completion count to the current I/O completion count, and (iii) based on the sample I/O completion count being greater than the current I/O completion count, switching to a second operating mode. In the second operating mode, I/O requests are assigned to the CPU cores based on core availability, and similar operations are performed for periodically testing whether to switch to the first operating mode.

Abstract: A technique for managing storage of compressed data includes generating and enforcing a minimum slot size requirement. The minimum slot size is based at least in part on collected performance metrics that indicate a degree of compressibility of data received, compressed, and written by the data storage system. As new data arrive, the new data are compressed and stored in slots at least as big as the minimum slot size, in many cases effecting an over-allocation of storage space and improving the likelihood that subsequent overwrites will fit into existing slots.

Abstract: A method, computer program product, and computing system for processing higher-level metadata portions to identify each occurrence of a pointer to a lower-level metadata portion; and generating a pointer record within a memory device that memorializes each occurrence of a pointer to a lower-level metadata portion and identifies a specific lower-level metadata portion to which the pointer is pointing, thus resulting in a plurality of pointer records.

Abstract: A method, computer program product, and computing system for maintaining a first cursor pointer for a memory system, wherein the first cursor pointer concerns a first search criteria and identifies an initial first cursor starting location; receiving a first search request for the memory system that satisfies the first search criteria; executing the first search request within an initial first search window starting at the initial first cursor starting location; and updating the first cursor pointer to identify an updated first cursor starting location based, at least in part, upon the initial first search window.

Abstract: A method, computer program product, and computer system for identifying a bit for an allocation unit. It may be determined if data has been modified on the allocation unit while degraded. A rebuild of the allocation unit may be executed when the bit is a first value. The rebuild of the allocation unit may be skipped when the bit is a second value.

Abstract: A method, computer program product, and computing system for processing memory page metadata received from a cache memory system within a data storage system to determine if the memory page metadata includes corruption due to a power failure event; if the memory page metadata includes post-acknowledgement data corruption, initiating a data recovery process to attempt to recover content associated with the post-acknowledgement data corruption; and if the memory page metadata includes pre-acknowledgement data corruption, reobtaining content associated with the pre-acknowledgement data corruption.

Abstract: A method, computer program product, and computing system for processing higher-level metadata portions to identify each occurrence of a pointer to a lower-level metadata portion; and generating a pointer record within a memory device that memorializes each occurrence of a pointer to a lower-level metadata portion and identifies a specific lower-level metadata portion to which the pointer is pointing, thus resulting in a plurality of pointer records.

Abstract: Coordinating a reclaiming of data storage space among processing nodes of a data storage system includes, by a first node in response to an event, performing local cleanup of first-node mapping data and issuing a request message to a second node, the request message identifying the data storage space to be reclaimed. The event may be a notification from a RAID component rebuilding a RAID volume. In response to the request message, the second node performs local cleanup of second-node mapping data and issues a completion message to the first node indicating that the second node has completed its part of the reclaiming of the data storage space. The first node responds to the completion message by marking the data storage space as being reclaimed and signaling to a source of the event that the data storage space is available for new use.

Abstract: A method, computer program product, and computing system for processing a data portion to divide the data portion into a plurality of data chunks; performing an entropy analysis on each of the plurality of data chunks to generate a plurality of data chunk entropies; and determining an average data chunk entropy from the plurality of data chunk entropies.

Abstract: Techniques for data processing may include: determining one or more sub-blocks of a target block that match one or more sub-blocks of a candidate block; creating a shared sub-block mapping (SSM) structure having a plurality of entries, wherein each of the plurality of entries corresponds to a different one of the sub-blocks in the candidate block and wherein a value stored in said each entry, corresponding to one of the sub-blocks of the candidate block, identifies a sub-block of the target block matching said one sub-block of the candidate block; and storing the candidate block as a deduplicated block sharing at least one sub-block with the target block. The SSM structure may be stored as a metadata structure of the candidate block to identify deduplicated sub-blocks of the candidate block and to identify sub-blocks of the target block providing content for the deduplicated sub-blocks of the candidate block.