Abstract: A data storage system performs updating of a set of global counters stored in persistent storage accessible to a plurality of processing nodes of a data storage system, the global counters storing respective global count values describing associated units of data storage shared by the processing nodes for performing data storage operations. The updating includes, by each processing node, using a respective set of in-memory delta counters storing respective delta count values to track changes to respective global count values due to data storage operations performed by the processing node with respect to the units of data storage. A summarization process is regularly executed to merge the delta count values of the delta counters of all processing nodes into the respective global counters.

Abstract: A data storage system in which a transaction is generated that indicates at least one data block of a logical volume to be written to non-volatile data storage of a data, and in which the logical volume is accessible to multiple nodes in the data storage system. A system-wide lock is obtained for each data block indicated by the transaction. A new generation identifier is then created that is equal to a last transaction identifier that was created and stored during processing of a previously completed transaction. Each data block indicated by the transaction is stored into the non-volatile data storage of the data storage system together with the new generation identifier and the last transaction identifier is updated before each system-wide lock on each data block indicated by the transaction is released.

Abstract: Host data stored in one or more source physical extents of non-volatile data storage is identified as valid and determined to be infrequently written by host I/O requests, and is therefore compressed to generate a highly compressed version of the valid host data. The highly compressed version is then stored into at least one target physical extent. The valid host data may be initially compressed before it is stored in the source physical extent(s), and may be re-compressed to generate the highly compressed version. If the valid host data is also infrequently read, it may be recompressed using larger blocks of host data than were used to perform the initial compression. The performance tier of the target physical extent may be different from (e.g. lower than) the performance tier of the source physical extent. The technology may be embodied in a background process such as a garbage collector.

Abstract: A data storage system in which data storage space for storing data storage metadata is selectively allocated from non-volatile data storage devices contained in a first storage enclosure. The first storage enclosure is directly connected to at least one storage processor of the data storage system, and the data storage system further includes at least one secondary storage enclosure that is indirectly connected to the storage processor. Data storage metadata is generated within the data storage system, and the data storage metadata generated within the data storage system is stored into the data storage space that was selectively allocated for storing data storage metadata from the non-volatile data storage devices contained in the first storage enclosure.

Abstract: Techniques for providing a direct IO path to compressed data on storage media of a storage system. The techniques include triggering a transaction cache to perform a flush operation for updating mapping metadata for a storage object containing the compressed data. Having updated the mapping metadata for the storage object, the techniques further include issuing, by a copier module, an IO read request for the compressed data of the storage object to a namespace layer, which issues the IO read request to a mapping layer. The techniques further include forwarding the IO read request to a logical layer of the mapping layer, bypassing the transaction cache. The techniques further include reading, by the logical layer, the compressed data of the storage object from the storage media, and providing, via the mapping layer and the namespace layer, the compressed data to the copier module for transfer to a destination storage system.

Abstract: A method of responding to failures in a tiered storage system is provided. The method includes (a) rebuilding a set of failed storage extents belonging to a first storage tier; (b) receiving a notification that a particular storage extent has failed while rebuilding the set of failed storage extents belonging to the first tier; and (c) upon determining that the particular storage extent belongs to a second storage tier that has a higher priority than does the first storage tier: (1) pausing rebuilding the set of failed storage extents belonging to the first storage tier, (2) rebuilding the particular storage extent, and (3) resuming rebuilding the set of failed storage extents belonging to the first storage tier after rebuilding the particular storage extent. An apparatus, system, and computer program product for performing a similar method are also provided.

Abstract: Processing for a file system may include determining an inconsistency between a data log and inodes (index nodes) of the file system. Responsive to determining the inconsistency, recovery processing may be performed including first processing of the data log, second processing of the inodes of the file system, and third processing of the data log after said second processing. First processing may, for each entry of the data log that records a create snapshot operation to create a snapshot of a file having an associated inode of the file system that is a dirty snapshot inode, marking the associated inode as “snap validated”. Second processing may include freeing each dirty snapshot inode not marked as “snap validated”. Third processing may include freeing any entry of the data log where the entry references dirty snapshot inode freed by the second processing.

Abstract: A method, computer program product, and computer system for receiving, by a computing device, new data to write to a leaf. At least two timestamps of the leaf may be examined. It may be determined whether a time interval between the at least two timestamps of the leaf is greater than an age threshold. The new data may be written to a first tier storage device when the time interval between the at least two timestamps of the leaf is less than the age threshold; The new data may be written to a second tier storage device when the time interval between the at least two timestamps of the leaf is greater than the age threshold.

Abstract: Improved techniques of managing a data storage system involve selectively inserting block virtualization structures (BVS) in access paths between data blocks of a file system and block pointers pointing to the data blocks. A BVS provides metadata for supporting deduplication of data in that data block. In some arrangements, a file system may support selective insertion of such a BVS between a block pointer and data block according to a specified criterion. For example, such a file system might support insertion of BVS's between block pointers and those data blocks storing cold data for which access latency is not important to overall performance of the data storage system.

Abstract: A technique for managing data received into a cache operates in cycles. To process a current batch of compressed blocks during a current cache processing cycle, a storage system obtains a new mapping structure and a new segment of contiguous storage space. If the system can place some of the current batch of compressed blocks into previously-allocated space, the system does so and partially populates the new mapping structure with entries for mapping the other compressed blocks that were not placed. The system then asserts a hold on the new mapping structure, so that the mapping structure is retained in cache at the end of the current cache processing cycle, and more completely populates the new mapping structure with entries for other compressed blocks during a later cache processing cycle before releasing the hold.

Abstract: A method, computer program product, and computing system for identifying a potential deduplication candidate and a related deduplication target; executing a comparison operation with respect to the potential deduplication candidate and the related deduplication target to generate a comparison result; and determining a level of similarity between the potential deduplication candidate and the related deduplication target by processing the comparison result.

Abstract: A technique for performing data deduplication operates at sub-block granularity by searching a deduplication database for a match between a candidate sub-block of a candidate block and a target sub-block of a previously-stored target block. When a match is found, the technique identifies a duplicate range shared between the candidate block and the target block and effects persistent storage of the duplicate range by configuring mapping metadata of the candidate block so that it points to the duplicate range in the target block.

Abstract: Methods and apparatus are provided for reporting fragment filling in storage systems. An exemplary method comprises obtaining at least one compressed allocation unit of data in a storage system; reserving space for the at least one compressed allocation unit in a compressed segment based on a greater of (i) a size of the at least one compressed allocation unit, and (ii) a minimum target fragment length specified for at least one write operation class; and writing the at least one compressed allocation unit to the reserved space. An attempt is optionally first made to fill the at least one compressed allocation unit of data in a free extent and/or a hidden fragment of at least one compressed segment prior to reserving the space. The reserving the space is optionally only performed if the attempt to fill the at least one compressed allocation unit of data is not successful.

Abstract: A method, computer program product, and computer system for monitoring host IO latency. It may be identified that a rate of the host IO latency is at a one of a plurality of levels. At least one of a rate of background IOs and a rate of host IOs may be regulated based upon, at least in part, the rate of the host IO latency being at the one of the plurality of levels.

Abstract: A method, system, and computer program product for multi-generational virtual block compaction comprising identifying a first virtual block, the first virtual block being associated with a first generation number, determining a second virtual block as an appropriate target for live information in the first virtual block, creating an association between the second virtual block and the first virtual block, updating the live information in the first virtual block to be associated with the second virtual block, updating a generation information mapping associated with the first virtual block; and associating a second generation number with the first virtual block.

Abstract: In response to a cache flush event indicating that host data accumulated in a cache of a storage processor of a data storage system is to be flushed to a lower deck file system, an aggregation set of blocks is formed within the cache, and a digest calculation group is selected from within the aggregation set. Hardware vector processing logic is caused to simultaneously calculate crypto-digests from the blocks in the digest calculation group. If one of the resulting crypto-digests matches a previously generated crypto-digest, deduplication is performed that i) causes the lower deck file system to indicate the block of data from which the previously generated crypto-digest was generated and ii) discards the block that corresponds to the matching crypto-digest. Objects required by a digest generation component may be allocated in a just in time manner to avoid having to manage a pool of pre-allocated objects.

Abstract: A method of performing deduplication includes (1) receiving a write command that specifies a set of data, the set of data including multiple blocks of data, (2) hashing a subset of the set of data, yielding a representative digest of the set of data, and (3) performing deduplication on the set of data based at least in part on matching the representative digest to a digest already stored in a database which relates digests to locations of data from which the digests were produced. An apparatus, system, and computer program product for performing a similar method are also provided.

Abstract: There is disclosed techniques for use in managing data storage in storage systems. For example, in one embodiment, there is disclosed a method comprising receiving a request to store data of a data object in a storage system. The method also comprising determining that at least a portion of the data is to be stored in an uncompressed format in the storage system in response to receiving the request. The method also comprising storing at least a portion of the data in the uncompressed format in an allocation unit of a segment in the storage system such that the stored data in the uncompressed format emulates stored data in a compressed format based on the said determination.

Abstract: A method performed by a block-storage server, of storing data is described. The method includes (1) receiving, from a remote file server, data blocks to be written to persistent block storage managed by the block-storage server; (2) receiving, from the remote file server, metadata describing a placement of the data blocks in a filesystem managed by the remote file server; and (3) organizing the data blocks within the persistent block storage based, at least in part, on the received metadata. An apparatus, system, and computer program product for performing a similar method are also provided.

Abstract: A technique for performing data reduction applies deduplication principles when performing data compression, providing a form of enhanced compression. The technique obtains a chunk of data that contains multiple extents and applies deduplication actions to identify duplicate extents within the chunk. The technique marks duplicate extents in metadata. Such duplicate extents need not be compressed using conventional data compression, saving computational resources and considerable time.