Abstract: A data storage system performs updating of a set of global counters stored in persistent storage accessible to a plurality of processing nodes of a data storage system, the global counters storing respective global count values describing associated units of data storage shared by the processing nodes for performing data storage operations. The updating includes, by each processing node, using a respective set of in-memory delta counters storing respective delta count values to track changes to respective global count values due to data storage operations performed by the processing node with respect to the units of data storage. A summarization process is regularly executed to merge the delta count values of the delta counters of all processing nodes into the respective global counters.

Abstract: Techniques are provided for inline deduplication based on a number of physical blocks having common fingerprints among multiple entries of a buffer cache. One method comprises storing input/output operations in a first cache comprising a plurality of entries each corresponding to a physical storage entity comprising a plurality of physical blocks. A given entry is maintained in the first cache based on a first number of physical blocks of the given entry having a duplicate fingerprint with at least one physical block of another entry in the first cache. A second number can be determined of the physical blocks of each entry having a fingerprint in a second cache, and a first ratio is determined for two entries in the first cache using the second number and the first number. A comparison of the first ratios can be performed to sort and possibly evict entries in the first cache based on the comparison.

Abstract: Techniques for scavenging blocks may include: determining, in accordance with a selected option, a set of candidate upper deck file systems, wherein at least a first of the candidate upper deck file systems has storage allocated from at least one block of a lower deck file system; and performing, in accordance with the selected option, scavenging of the set of candidate upper deck file systems to attempt to free blocks of the lower deck file system. Scavenging may include issuing a request to perform hole punching of a backed free block of the first candidate upper deck file system, wherein the backed free block has first provisioned storage that is associated with a block of the lower deck file system. The selected option may be one of multiple options each specifying a different candidate set of upper deck file systems upon which hole punching is performed when selected.

Abstract: A method of correcting errors in a data storage system including a first node, a second node, and shared persistent storage (the first and second nodes being configured to process data storage requests) is provided. The method includes (a) reading cached pages from a first cache disposed within the first node, the cached pages being cached versions of respective persistent pages stored in the shared persistent storage; (b) in response to determining that one of the cached pages is corrupted, requesting that the second node return to the first node a corresponding remote page from a second cache disposed within the second node, the cached page and the remote page each caching a same persistent page of the shared persistent storage; and (c) in response to determining that the remote page received from the second node by the first node is not corrupted, correcting the cached page using the remote page.

Abstract: There is disclosed techniques for data reduction reporting in storage systems. For example in one embodiment, the techniques comprise update a counter denoting an amount of data reduction obtained when a data reduction feature associated with a data reduction requirement is enabled in the storage system when one or more allocation units in a storage system satisfy a data reduction requirement of the storage system. The techniques further comprise generating a snapshot of at least a portion of a file related to the said allocation units such that a first relationship exists when the snapshot is generated and a second relationship exists when the snapshot or the file is updated. The techniques also comprise determining a data reduction attributed to the data reduction feature based on an amount of data reduction denoted by the data reduction counter and an amount of sharing and/or unsharing of indirect blocks including mapping pointers in the relationship between the snapshot and the file.

Abstract: A technique for performing writes in a storage system that supports data compression tests incoming writes to determine whether the amount of data to be written is less than a predetermined threshold. If so, the storage system avoids compression and stores the amount of data as received without compressing it. If not, the storage system performs compression and stores the incoming data as a compressed extent.

Abstract: A technique for managing a datapath of a data storage system includes receiving a request to access target data and creating a transaction that includes multiple datapath elements in a cache, where the datapath elements are used for accessing the target data. In response to detecting that one of the datapath elements is invalid, the technique further includes processing the transaction in a rescue mode. The rescue mode attempts to replace each invalid datapath element of the transaction with a valid version thereof obtained from elsewhere in the data storage system. The technique further includes committing the transaction as processed in the rescue mode.

Abstract: A method, computer program product, and computer system for receiving, by a computing device, new data to write to a leaf. At least two timestamps of the leaf may be examined. It may be determined whether a time interval between the at least two timestamps of the leaf is greater than an age threshold. The new data may be written to a first tier storage device when the time interval between the at least two timestamps of the leaf is less than the age threshold; The new data may be written to a second tier storage device when the time interval between the at least two timestamps of the leaf is greater than the age threshold.

Abstract: Dirty pages of cached user data are persistently stored to page buffers that are allocated from a page buffer pool in a persistent data storage resource of a data storage system, and are indicated by page descriptors that are stored at a head of a temporally ordered page descriptor ring as the dirty pages are stored to the page buffers. The disclosed technology performs a flush operation by selecting a work-set of non-sequential page descriptors within the page descriptor ring, flushing dirty pages from page buffers indicated by the page descriptors in the work-set to non-volatile data storage drives of the data storage system, and storing, for each one of the page buffers indicated by the page descriptors in the work-set, an indication that the page buffer is available for re-use.

Abstract: Techniques are provided for inline deduplication based on a number of physical blocks having common fingerprints among multiple entries of a buffer cache. One method comprises storing input/output operations in a first cache comprising a plurality of entries each corresponding to a physical storage entity comprising a plurality of physical blocks. A given entry is maintained in the first cache based on a first number of physical blocks of the given entry having a duplicate fingerprint with at least one physical block of another entry in the first cache. A second number is determined of the physical blocks of each entry having a fingerprint in a second cache, and a first ratio is determined for two entries in the first cache using the second number and the first number. A comparison of the first ratios can be performed to sort and possibly evict entries in the first cache based on the comparison.

Abstract: A method of managing storage space for a metadata consistency checking procedure (MCCP) is provided. The method includes (a) tracking an amount of metadata and an amount of user data organized by the metadata; (b) provisioning a quantity of storage dedicated to the MCCP based, at least in part, on a ratio of the amount of metadata to the amount of user data; and (c) upon initiation of the MCCP, building tracking structures within the provisioned storage dedicated to the MCCP. An apparatus, system, and computer program product for performing a similar method are also provided.

Abstract: Techniques for updating persistent statistics on a multi-transactional and multi-node storage system. The techniques can be practiced in a distributed storage system including a plurality of storage nodes. Each storage node can include a persistent storage configured to accommodate a set of delta counter pages, as well as a global counter page for summarizing delta count values tracked by respective delta counters. The techniques can include, in each storage node, tracking, by each of a set of delta counters, changes to delta count values due to storage node operations performed on units of data storage, and summarizing, periodically or at least at intervals, the delta count values of the respective delta counters as global count values. The techniques can further include summarizing the global count values across the respective storage nodes in a count summarization report and sending the count summarization report to a client via a communications interface.

Abstract: Techniques for processing data may include: receiving a candidate data block; computing a distance using a distance function, wherein the distance is an entropy-based distance and denotes a measurement of similarity between the candidate data block and a target data block; and determining, using the distance, whether to perform data deduplication of the candidate data block with respect to the target data block to identify at least one sub-block of the candidate data block that is a duplicate of at least one sub-block of the target data block. If the distance is less than a threshold, it may be expected to have a matching sub-block between the candidate and target data blocks. The distance may be a difference between entropy values for the candidate and target data blocks. The first entropy value may be used to determine whether to compress or perform partial deduplication for the candidate data block.

Abstract: Techniques manage objects in data storage equipment. Such techniques involve receiving a request to create a new object for a particular object family (e.g., a collection of related data storage objects such as a production volume, snapshots, clones, snapshots of clones, etc.). Such techniques further involve deriving, for the particular object family, a total object count based on an active object count and a deleted object count for the particular object family. Such techniques further involve, in response to the request, performing an object management operation that (i) creates the new object when the total object count is less than a predefined total object count threshold and (ii) prevents creation of the new object when the total object count is not less than the predefined total object count threshold.

Abstract: A logical unit of non-volatile data storage is provided for persistently storing metadata. The logical unit is provided by allocating RAID (Redundant Array of Independent Disks) extents to the logical unit, and allocating logical block addresses for the logical unit from within a logical address space of the logical unit to store blocks of metadata written to the logical unit. In response to detecting that a stored indication of a last allocated logical block address for the logical unit has been lost, two RAID extents that were last allocated to the logical unit are identified, and a binary search is performed across logical block addresses within the two RAID extents that were last allocated to the logical unit to locate the last allocated logical block address for the logical unit within the logical address space of the logical unit.

Abstract: A logical unit of non-volatile data storage is provided for persistently storing metadata. The logical unit is provided by allocating RAID (Redundant Array of Independent Disks) extents to the logical unit, and allocating logical block addresses for the logical unit from within a logical address space of the logical unit to store blocks of metadata written to the logical unit. In response to detecting that a stored indication of a last allocated logical block address for the logical unit has been lost, two RAID extents that were last allocated to the logical unit are identified, and a binary search is performed across logical block addresses within the two RAID extents that were last allocated to the logical unit to locate the last allocated logical block address for the logical unit within the logical address space of the logical unit.

Abstract: There is disclosed a technique for managing data inconsistencies in file systems. For example, in one embodiment, the technique comprises detecting a data inconsistency relating to a file system block in a file system. The technique also comprises determining whether the file system block includes a mapping pointer associated with a first type of pattern. The technique further comprises performing recovery in connection with the file system block. The recovery including deciding whether to associate a mapping pointer of the file system block with one of a second type of pattern or a data loss state based on the said determination of whether the file system block includes a mapping pointer associated with a first type of pattern.

Abstract: A technique for performing data deduplication operates at sub-block granularity by searching a deduplication database for a match between a candidate sub-block of a candidate block and a target sub-block of a previously-stored target block. When a match is found, the technique identifies a duplicate range shared between the candidate block and the target block and effects persistent storage of the duplicate range by configuring mapping metadata of the candidate block so that it points to the duplicate range in the target block.

Abstract: A method of managing storage space for a metadata consistency checking procedure (MCCP) is provided. The method includes (a) tracking an amount of metadata and an amount of user data organized by the metadata; (b) provisioning a quantity of storage dedicated to the MCCP based, at least in part, on a ratio of the amount of metadata to the amount of user data; and (c) upon initiation of the MCCP, building tracking structures within the provisioned storage dedicated to the MCCP. An apparatus, system, and computer program product for performing a similar method are also provided.

Abstract: Techniques for processing data may include: receiving a candidate block; performing partial deduplication processing of the candidate block; receiving a second candidate block subsequent to performing partial deduplication processing for the candidate block; and performing first processing to determine whether to perform promotion processing for the entry, The partial deduplication processing may include: partially deduplicating at least one sub-block of the candidate block; and creating an entry in a deduplication database for the candidate block, wherein the entry includes a digest of the candidate block and the entry denotes a potential target block having the digest, and wherein the entry includes a counter that tracks a number of missed full block deduplications between the potential target block and subsequently processed candidate blocks. The promotion processing promotes the potential target block, having the first digest of the entry, to a new target block.