Abstract: A method, computer program product, and computing system for expanding the lifetime of virtual entries in a data storage system. A deduplication function of a virtual layer of a data storage system is monitored, a reference count of a virtual entry is incremented when a data page is written to a physical layer of the data storage system and the reference count of the virtual entry is decremented when a data page is deleted from the virtual layer. The virtual entry in the virtual layer is maintained when the reference count reaches a predetermined value and the virtual entry is reclaimed when a predetermined action of the data storage system is to be performed.

Abstract: A method of managing end-of-life processing of data blocks includes performing a decref procedure for a chunk of data, the decref procedure decrementing reference counts for respective blocks of the chunk of data, and deferring performance of an end-of-life procedure for a plurality of blocks of the chunk of data whose reference counts become zero as a result of performing the decref procedure. The method further includes, upon performing a data consolidation procedure after performing the decref procedure, performing the end-of-life procedure on the plurality of blocks.

Abstract: A technique is directed to providing data storage resiliency. The technique includes, after losing access to a storage device that provides storage to a storage extent, performing a degraded write operation that writes data to a first line of the storage extent. The technique further includes setting a first indicator to indicate that the first line was written to while access to the storage device was lost and while a second indicator continues to indicate that a second line of the storage extent was not written to. The technique further includes, after regaining access to the storage device and in response to the first indicator indicating that the first line was written to in the degraded write operation, performing a resiliency operation that constructs additional data from the data written in the first line to increase data storage resiliency of the storage extent.

Abstract: Techniques coordinate deduplication among nodes. The techniques involve, in response to a lookup query to search a deduplication index for an entry that maps a fingerprint which is based on incoming data, detecting an invalid result. The techniques further involve, in response to detecting the invalid result, updating the deduplication index to include an entry that maps the fingerprint, the entry initially storing an “in-progress” flag. The techniques further involve, after the deduplication index is updated to include the entry that maps the fingerprint and after the incoming data is stored in a storage location, removing the “in-progress” flag from the entry.

Abstract: A method in a data storage system includes foreground processes that produce incorrect-high reference counts greater than corresponding numbers of actual references to associated data instances (e.g., virtual large blocks), representing loss of storage capacity. A background process corrects a reference count for a data instance by (1) copying the data instance to a new data instance with an initial reference count of zero and making the new data instance unavailable for reclaiming, (2) scanning a set of referencing structures for all references to the data instance, and for each reference (i) replacing the reference with a new reference to the new data instance, and (ii) incrementing the reference count of the new data instance, and (3) upon completing the scanning, making the new data instance available for eventual reclaiming based on its reference count reaching zero. The process restores reference count accuracy and reduces loss/leakage of storage capacity.

Abstract: A method of data deduplication during log flushing includes use of an in-memory deduplication index as a cached sub-set of entries from a full on-drive deduplication index. In a pre-flush process, the data is scanned and the on-drive index is accessed to identify matching deduplication entries, and identified entries are copied into the in-memory index. The subsequent flush includes, for each data unit, (1) accessing the in-memory index to identify a matching deduplication entry for the data unit, (2) upon identifying a matching entry, storing the data unit in a deduplicated manner as a reference to previously stored data, and (3) upon not identifying a matching entry, storing the data unit in a non-deduplicated manner including a new data instance, without accessing the on-drive index. Performance and efficiency are improved by accessing only the in-memory index and avoiding the penalties of accessing the on-drive index.

Abstract: In at least one embodiment, processing can include receiving, at a first node, a read request directed to a logical address LA owned by a second node. The first node can locally cache content and an address hint corresponding to LA. The first node can issue a request to the second node. The request can include a flag to suppress the second node from returning content stored at the target logical address. The first node can receive a response including an address used to read current content of LA1 from back-end non-volatile storage. The first node can determine whether the address matches the address hint cached on the first node. If the first node determines the address and address hint match, the cached content of LA stored on the first node is valid and can be returned in response to the read as current content stored at LA.

Abstract: System-level garbage collection is performed in an SSD-based data storage system employing log-structured writing of data. Respective reclaim unit sizes of the storage devices are identified, wherein each reclaim unit is a unit of programming and erasure used by device-level garbage collection to generate erased reclaim units for allocation for physical data storage. The data is arranged across the storage devices in first and second data units, each first data unit being of a first size U equal to the product of R and a stripe width W, each second data unit including a respective set of first data units which occupy a maximum-size reclaim unit of a storage device among those storing the first data units. The system-level garbage collection is operated with second-data-unit granularity to regularly generate empty second data units for allocation in the log-structured writing of data, reducing write amplification and device wear.

Abstract: Logical address slices and corresponding metadata pages of mapping information can be partitioned into sets. Each node can be assigned exclusive ownership of one of the sets. In at least one embodiment, for a read I/O which is received at a first node and directed to a logical address LA1 that is owned by a second node, the first node can request that the second owning node perform resolution processing for LA1. The second node can own both LA1 and corresponding metadata pages included in mapping information used to map LA1 to a corresponding physical location PA1 including content C1 stored at LA1. The second node can perform resolution processing for LA1 using the metadata pages corresponding to LA1 to either read and return C1 to the first node, or obtain and return PA1 to the first node where the first node can then read C1 directly using PA1.

Abstract: Techniques for providing a “bin-less” metadata page allocator for clustered systems with log-structured metadata storage. The techniques include providing a mapping structure in memory of a storage node of a clustered system. The mapping structure can have multiple layers configured to map logical addresses of metadata pages to physical addresses of storage drives within a storage array. The techniques include providing a translation table configured to translate logical addresses of metadata pages to corresponding current physical addresses of storage drives within the storage array. The techniques include, in response to an allocated logical address of a metadata page no longer being in-use, replacing its corresponding current physical address with a predefined value in the translation table, freeing the logical address, and inserting the freed logical address into a free logical address array. The techniques include allocating the freed logical address from the free logical address array.

Abstract: In at least one embodiment, processing can include receiving, at a first node, a read request directed to a logical address LA owned by a second node. The first node can locally cache content and an address hint corresponding to LA. The first node can issue a request to the second node. The request can include a flag to suppress the second node from returning content stored at the target logical address. The first node can receive a response including an address used to read current content of LA1 from back-end non-volatile storage. The first node can determine whether the address matches the address hint cached on the first node. If the first node determines the address and address hint match, the cached content of LA stored on the first node is valid and can be returned in response to the read as current content stored at LA.

Abstract: Logical address space portions and virtual layer blocks (VLBs) can be partitioned into multiple sets. Each of multiple nodes in a system can be assigned exclusive ownership of one of the multiple sets. In at least one embodiment, for a read I/O which is received at a first node and directed to a logical address LA1 that is owned by a second node, the first node can request that the second owning node perform resolution processing for LA1. The second node can return either a VLB address or a PLB address based on whether the second node owns a VLB used in mapping LA1 to a corresponding physical location PA1 which includes content C1 stored at LA1. The second node can set a flag in its response to indicate whether a returned address is a VLB address or a PLB address.

Abstract: Techniques described herein elect how data is deduplicated in a storage system. A similarity hash signature for a data unit is calculated. A digest table is searched for a similarity hash signature within a predetermined distance of the similarity hash signature for the data unit. Based on the search, either a similarity hash signature or a strong hash signature of the data unit is added to the digest table.

Abstract: Logical address slices and corresponding metadata pages of mapping information can be partitioned into sets. Each node can be assigned exclusive ownership of one of the sets. In at least one embodiment, for a read I/O which is received at a first node and directed to a logical address LA1 that is owned by a second node, the first node can request that the second owning node perform resolution processing for LA1. The second node can own both LA1 and corresponding metadata pages included in mapping information used to map LA1 to a corresponding physical location PA1 including content C1 stored at LA1. The second node can perform resolution processing for LA1 using the metadata pages corresponding to LA1 to either read and return C1 to the first node, or obtain and return PA1 to the first node where the first node can then read C1 directly using PA1.

Abstract: Techniques for providing a “bin-less” metadata page allocator for clustered systems with log-structured metadata storage. The techniques include providing a mapping structure in memory of a storage node of a clustered system. The mapping structure can have multiple layers configured to map logical addresses of metadata pages to physical addresses of storage drives within a storage array. The techniques include providing a translation table configured to translate logical addresses of metadata pages to corresponding current physical addresses of storage drives within the storage array. The techniques include, in response to an allocated logical address of a metadata page no longer being in-use, replacing its corresponding current physical address with a predefined value in the translation table, freeing the logical address, and inserting the freed logical address into a free logical address array. The techniques include allocating the freed logical address from the free logical address array.

Abstract: Logical address space portions and virtual layer blocks (VLBs) can be partitioned into multiple sets. Each of multiple nodes in a system can be assigned exclusive ownership of one of the multiple sets. In at least one embodiment, for a read I/O which is received at a first node and directed to a logical address LA1 that is owned by a second node, the first node can request that the second owning node perform resolution processing for LA1. The second node can return either a VLB address or a PLB address based on whether the second node owns a VLB used in mapping LA1 to a corresponding physical location PA1 which includes content C1 stored at LA1. The second node can set a flag in its response to indicate whether a returned address is a VLB address or a PLB address.

Abstract: Logical address space portions and virtual layer blocks (VLBs) can be partitioned into multiple sets. Each of multiple nodes in a system can be assigned exclusive ownership of one of the multiple sets. In at least one embodiment, for a read I/O which is received at a first node and directed to a logical address LA1 that is owned by a second node, the first node can request that the second owning node perform resolution processing for LA1. The second node can return either a VLB address or a PLB address based on whether the second node owns a VLB used in mapping LA1 to a corresponding physical location PA1 which includes content C1 stored at LA1. The second node can set a flag in its response to indicate whether a returned address is a VLB address or a PLB address.

Abstract: A method, computer program product, and computing system for maintaining a pointer quantity count for each of a plurality of virtual mapping groups, wherein each virtual mapping group is associated with a duplicative data portion within a storage platform and each pointer quantity count defines the number of pointers included within the corresponding virtual mapping group; and obtaining a hash value for a specific duplicative data portion associated with a specific virtual mapping group if a specific pointer quantity count for the specific virtual mapping group is reduced to zero.

Abstract: A method is used for improving performance in a storage system during deduplication. A processor receives a request to write data. A determination is made that the data can be deduplicated. A non-exclusive lock for a page in storage with a reference count of the data may be obtained, and an instruction to increment the reference count for the data may be stored in a metadata journal. The non-exclusive lock for the page may be released.

Abstract: A technique for managing deduplication using a non-cryptographic hash function includes obtaining metadata associated with both a candidate block presented for deduplication and a target block having a hash-based match to the candidate block. The technique further includes checking for a mismatch between the candidate block and the target block based on the obtained metadata. In response to the checking determining a mismatch, the technique further includes abandoning deduplication of the candidate block, such that the candidate block is stored independently of the target block.