Abstract: A technique for managing deduplication using a non-cryptographic hash function includes obtaining metadata associated with both a candidate block presented for deduplication and a target block having a hash-based match to the candidate block. The technique further includes checking for a mismatch between the candidate block and the target block based on the obtained metadata. In response to the checking determining a mismatch, the technique further includes abandoning deduplication of the candidate block, such that the candidate block is stored independently of the target block.

Abstract: A request to perform a first operation in a system that stores deduplicated data can be received. The system can include a data block stored at multiple logical address each referencing the data block. A reference count can be associated with the data block and can denote a number of logical addresses referencing the data block. Processing can be performed to service the request and perform the first operation, wherein the processing can include: acquiring a non-exclusive lock for a page that includes the reference count of the data block; storing, in a metadata log while holding the non-exclusive lock on the page, an entry to decrement the reference count of the data block; and releasing the non-exclusive lock on the page.

Abstract: A method, computer program product, and computing system for receiving a user data page for storing in a storage system. An unaligned hash representation of the received user data page may be generated. The generated unaligned hash representation may be identified within a hash table, thus defining an identified hash representation. A user data page associated with the identified hash representation may be identified, thus defining an identified user data page. A hash offset reference of the identified user data page may be compared with a hash offset reference of the received user data page. A deduplication operation may be performed on the received user data page based upon, at least in part, the comparison of the hash offset reference of the identified user data page with the hash offset reference of the received user data page.

Abstract: A method, computer program product, and computing system for loading a first chunk of data from a user data page into a first buffer. Each successive chunk of data from the user data page may be iteratively loaded into a second buffer. The first chunk of data in the first buffer may be iteratively compared with each successive chunk of data in the second buffer. In response to determining that the first chunk of data in the first buffer is equivalent to each successive chunk of data in the second buffer, the first chunk of data may be defined as a pattern.

Abstract: A request to perform a first operation in a system that stores deduplicated data can be received. The system can include a data block stored at multiple logical address each referencing the data block. A reference count can be associated with the data block and can denote a number of logical addresses referencing the data block. Processing can be performed to service the request and perform the first operation, wherein the processing can include: acquiring a non-exclusive lock for a page that includes the reference count of the data block; storing, in a metadata log while holding the non-exclusive lock on the page, an entry to decrement the reference count of the data block; and releasing the non-exclusive lock on the page.

Abstract: A method, computer program product, and computing system for receiving data for storage in a storage system. The data may be written to a head entry of a log buffer, wherein the log buffer includes a plurality of data entries for flushing to the storage system. At least a portion of the plurality of data entries of the log buffer may be flushed, via a plurality of threads, to the storage system based upon, at least in part, a tail entry of the log buffer. A queue of committed data entries may be updated, via each thread of the plurality of threads, with one or more data entries of the log buffer flushed to the storage system by each thread. A new tail entry of the log buffer may be determined, via a thread of the plurality of threads, based upon, at least in part, the queue of committed data entries.

Abstract: Updates are applied to a multi-entry bucket of a persistent multi-bucket hash table indexed by a hash index having a bucket portion and a collision portion, each entry of each bucket storing a corresponding value. The bucket is initially stored in a buffer and both a hash lookup structure and value lookup structure are generated for the bucket, the hash lookup structure usable to identify an entry of the bucket based on collision portion, the value lookup structure usable to identify an entry of the bucket based on value. For each update, a value of the update is applied to the value lookup structure to identify a corresponding entry, and the entry in the buffer is modified as required by the update. Subsequently the bucket in the buffer is persisted back to the hash table using the hash lookup structure.

Abstract: Updates are applied to a multi-entry bucket of a persistent multi-bucket hash table indexed by a hash index having a bucket portion and a collision portion, each entry of each bucket storing a corresponding value. The bucket is initially stored in a buffer and both a hash lookup structure and value lookup structure are generated for the bucket, the hash lookup structure usable to identify an entry of the bucket based on collision portion, the value lookup structure usable to identify an entry of the bucket based on value. For each update, a value of the update is applied to the value lookup structure to identify a corresponding entry, and the entry in the buffer is modified as required by the update. Subsequently the bucket in the buffer is persisted back to the hash table using the hash lookup structure.

Abstract: A method, computer program product, and computing system for defining a normal IO write mode for writing data to a storage system, the normal IO writing mode including: writing the data to a cache memory system, writing the data to a journal, in response to writing the data to the journal, sending an acknowledgment signal to a host device, and writing the data from the cache memory system to a storage array. A request may be received to enter a testing IO write mode. In response to receiving the request, the data may be written to the cache memory system. The writing of the data to the journal may be bypassed. The acknowledgment signal may be sent to the host device in response to writing the data to the cache memory system. The data may be written from the cache memory system to the storage array.

Abstract: A technique of performing data reduction includes, upon detecting a match between similarity hashes of a candidate dataset and a target dataset, evaluating an adjacent candidate dataset and an adjacent target dataset for similarity with each other and, in response to confirming such similarity, performing data reduction of the adjacent candidate dataset with reference to the adjacent target dataset.

Abstract: Technology is disclosed for selectively compressing data based on similarity of pages within the data that is to be compressed. At least one corresponding hash value is generated for each one of multiple candidate pages to be compressed. In response to the hash values generated for the candidate pages, the technology selects a set of similar candidate pages from the candidate pages. The set of similar candidate pages are a subset of the candidate pages that includes less than all the candidate pages. The set of similar candidate pages are compressed as a single unit, separately from one or more other ones of the candidate pages that were not selected to be included in the set of similar candidate pages.

Abstract: A method, computer program product, and computing system for defining a normal IO write mode for writing data to a storage system including: writing the data to a cache memory system of a first storage node, writing the data to a journal of the first storage node, sending a notification concerning the data to a second storage node, writing one or more metadata entries concerning the data to a journal of the second storage node, sending an acknowledgment signal to the host device, and writing the data to the storage array. A request may be received to enter a testing IO write mode. In response to receiving the request, the data may be written to the cache memory system. The writing of the data to the journal may be bypassed. The acknowledgment signal may be sent to the host device. The data may be written to the storage array.

Abstract: A method, computer program product, and computer system for identifying, by a computing device, content in a first bucket in a first cache. It may be determined that a first portion of the content in the first bucket is a duplicate, wherein a second portion of the content in the first bucket may be unique. The first portion of the content in the first bucket may be deduplicated from the first cache. The second portion of the content may be stored in a second bucket in a second cache.

Abstract: Techniques described herein elect how data is deduplicated in a storage system. A similarity hash signature for a data unit is calculated. A digest table is searched for a similarity hash signature within a predetermined distance of the similarity hash signature for the data unit. Based on the search, either a similarity hash signature or a strong hash signature of the data unit is added to the digest table.

Abstract: An aspect of implementing amortized execution of updates for a hash table includes aggregating, within a data structure, updates to be executed for the hash table. The aggregated updates are distributed across a plurality of chunks in the data structure. An aspect also includes sorting, within each of the chunks, the updates according to respective bucket index values associated with the updates, and iteratively executing the sorted updates across each of the chunks in the data structure.

Abstract: A method is used for improving performance in a storage system during deduplication. A processor receives a request to write data. A determination is made that the data can be deduplicated. A non-exclusive lock for a page in storage with a reference count of the data may be obtained, and an instruction to increment the reference count for the data may be stored in a metadata journal. The non-exclusive lock for the page may be released.

Abstract: Technology is disclosed for selectively compressing data based on similarity of pages within the data that is to be compressed. At least one corresponding hash value is generated for each one of multiple candidate pages to be compressed. In response to the hash values generated for the candidate pages, the technology selects a set of similar candidate pages from the candidate pages. The set of similar candidate pages are a subset of the candidate pages that includes less than all the candidate pages. The set of similar candidate pages are compressed as a single unit, separately from one or more other ones of the candidate pages that were not selected to be included in the set of similar candidate pages.

Abstract: Dictionary-based compression is performed to compress data units using a similar data unit as the base unit (i.e., dictionary) for each candidate data unit. Similarity may be determined between data units by applying a locality-sensitive hashing scheme to each candidate data unit to produce a hash value, and by determining whether there is a matching value in a hash index of hash values for existing data units on the system. If there is a matching hash value, the candidate data unit may be compressed using the data unit corresponding to the matching hash value as the dictionary. Only a representative portion of the data unit may be hashed to produce the hash value, the portion comprised of chunks of the data unit, where each chunk is a continuous, uninterrupted section of data. The chunks themselves may not be (in some embodiments likely are not) contiguous to one another.

Abstract: A technique for correcting errors in a data storage system operates while the data storage system remains online. The technique includes identifying an object for validation, scanning a plurality of pointers, and counting a number of pointers that point to the object. The technique further includes repairing a discrepancy between the count of pointers and a reference count stored in connection with the object.

Abstract: A technique for correcting errors in a data storage system operates while the data storage system remains online. The technique includes identifying an object for validation, scanning a plurality of pointers, and counting a number of pointers that point to the object. The technique further includes repairing a discrepancy between the count of pointers and a reference count stored in connection with the object.