Abstract: Embodiments are directed to techniques for implementing a deduplication system that minimizes disk accesses to an on-disk digest log when deduplicating consecutively-stored data. These techniques for performing deduplication utilize an in-memory temporal digest cache. When the on-disk digest log is accessed for a set of data and a match is found, the temporal digest cache is written with digests not only for the set of data but also for other data stored in a temporal relationship with the set of data. This temporal digest cache allows subsequent deduplication of temporally-related data to proceed faster without needing to repeatedly access the digest log on disk.

Abstract: Techniques for processing unaligned IO requests in data storage systems that provide optimization of inline compression. The disclosed techniques employ an unaligned IO cache, which is used by a data storage system to process unaligned IO requests containing data with sizes that are not multiples of a predetermined block size. By employing the unaligned IO cache while processing such unaligned IO requests, the data storage system can reduce the number of read-modify-write sequences required to process a sequential load of unaligned IO requests, thereby reducing the burden on processing resources of the data storage system.

Abstract: A method of protecting a filesystem against metadata corruption while maintaining high performance is provided. The method includes (a) receiving a storage command directing a computing device to store a block of file data to the filesystem and (b) in response: (1) writing the block to a data portion spanning a set of drives using a single-fault tolerant level of RAID, wherein writing the block includes performing exactly two write operations to the set of drives; and (2) writing filesystem metadata associated with the block to a metadata portion spanning a subset of the set of drives using a double-fault tolerant data redundancy scheme different than the single-fault tolerant level of RAID, the subset including at least three drives, the data and metadata portions being non-overlapping, wherein writing the metadata includes performing at least three write operations to the subset of the set of drives.

Abstract: A storage technique includes selectively tagging a set of write requests by applying tags to write requests that are pursuant to copy operations and not applying tags to at least some write requests that are not pursuant to copy operations. For each of the set of write requests, the technique further includes performing a decision operation configured to (i) direct the write request down a first processing path that supports inline deduplication in response to the write request satisfying a set of criteria and (ii) direct the write request down a second processing path that does not support inline deduplication in response to the write request failing to satisfy the set of criteria. The set of criteria include whether the write request has a tag applied, such that the decision operation preferentially directs write requests down the first processing path in response to the write requests being tagged.

Abstract: Embodiments are directed to techniques for performing deduplication. A method includes (a) obtaining a digest of a data block logically-positioned within a filesystem, the digest providing a hash value of data of the data block, (b) searching a Most Wanted Digest Cache (MWDC) within system memory for the digest, (c) locating an entry in the MWDC using the digest, wherein this locating indicates that the data block has the same data as another data block located elsewhere within the filesystem, the other data block having been previously persistently-stored, the entry having been added to the MWDC in response to the other data block having been deduplicated at least a plurality number of times, (d) locating a mapping structure referenced by the entry located from the MWDC, the mapping structure providing metadata about the other data block, and (e) deduplicating the data block and the other data block with reference to the located mapping structure.

Abstract: A technique for storing data in a data storage system detects that a read is being performed pursuant to a data copy request. In response, the data storage system stores a digest of the data being read in an entry of a digest cache. Later, when a write pursuant to the same copy request arrives, the storage system obtains the entry from the digest cache and completes the write request without creating a duplicate copy of the data.

Abstract: A storage technique includes selectively tagging a set of write requests by applying tags to write requests that are pursuant to copy operations and not applying tags to at least some write requests that are not pursuant to copy operations. For each of the set of write requests, the technique further includes performing a decision operation configured to (i) direct the write request down a first processing path that supports inline deduplication in response to the write request satisfying a set of criteria and (ii) direct the write request down a second processing path that does not support inline deduplication in response to the write request failing to satisfy the set of criteria. The set of criteria include whether the write request has a tag applied, such that the decision operation preferentially directs write requests down the first processing path in response to the write requests being tagged.

Abstract: A technique writes data to a storage array. The technique involves operating storage circuitry in a “FILL HOLE” mode in which the circuitry writes a stream of first data portions within storage portions of used storage stripes of the array. The technique further involves, after operating the circuitry in the “FILL HOLE” mode and in response to a first event, transitioning the circuitry from the “FILL HOLE” mode to a “STRIPE WRITE” mode in which the circuitry writes a stream of second data portions within unused storage stripes of the array. The technique further involves, after operating the circuitry in the “STRIPE WRITE” mode and in response to a second event, transitioning the circuitry from the “STRIPE WRITE” mode back to the “FILL HOLE” mode in which the circuitry writes a stream of third data portions within storage portions of used storage stripes of the array.

Abstract: A method is provided for storing files in a filesystem of a data storage system. The method includes (a) storing data of each file of a set of files of the filesystem in a set of data allocation units (AUs) of the filesystem on the data storage system, each data AU of the set of data AUs having a common data AU size, and (b) storing metadata pertaining to each file of the set of files in a set of metadata AUs of the filesystem on the data storage system, each metadata AU of the set of metadata AUs having a common metadata AU size, the common data AU size being larger than the common metadata AU size. A computerized apparatus and a computer program product are also provided for performing a method similar to that described above.

Abstract: Managing a circular transaction log in a data storage system includes receiving requests specifying data to be written to a data object served by the data storage system, and adding log entries to the log containing data specified by the requests at or after a tail of the log and before a head of the log. In response to a trigger event, a size of the log is dynamically increased by inserting additional storage space into the log at or after the tail and before the head, to account for differences between the rate at which entries are added to the log and the rate at which entries are flushed from the log. The disclosed technique may further operate to find the slot position in which each log entry is stored, based on the identifier of the log entry, after inserting the additional storage space into the log.

Abstract: Embodiments are directed to techniques for allowing write operations to proceed on units smaller than a block. Merely decreasing the block size is not desirable, however, since many files are written in large chunks at once, and larger block sizes can increase speed and decrease the amount of overhead metadata required. Therefore, in order to maintain large block sizes while still obtaining the benefit of being able to make small writes when necessary, blocks may be divided into sub-blocks. Unaligned writes that do not fill up an entire block may be segregated and stored separately from aligned writes, the unaligned writes having finer granularity. These techniques may result in faster operation for unaligned writes. They also especially benefit systems using compression because they allow efficiently partially overwriting compressed blocks.

Abstract: Improved techniques for maintaining cache coherence in a consistent state are provided. These techniques implement a data storage system using a journaled mirrored cache that ensures that storage operations making up certain transactions be performed atomically, so that a system failure does not result in data loss. The improved techniques also allow for efficient communication of mirroring information.

Abstract: A method is used in managing copying of data in storage systems. A request is received to copy a portion of a source logical object to a target logical object. The source and target logical objects are subject to a deduplicating technique. The portion of the source logical object is copied to the target logical object by updating metadata of the target logical object. The target logical object shares the portion of the source logical object.

Abstract: A technique manages file system metadata and is performed in a data storage system that maintains a file system on behalf of a set of host computers. The technique involves modifying a first metadata block of the file system in response to a file system change, the first metadata block containing file system metadata. The technique further involves performing a compression operation to generate a compressed metadata block in volatile storage from the first metadata block, the compressed metadata block having a size that is smaller than a size of the first metadata block. The technique further involves writing the compressed metadata block from the volatile storage (volatile cache memory) to non-volatile storage (e.g., flash memory).

Abstract: A technique for controlling acceptance of host application data into a data log in a data storage system includes selectively accepting or refusing newly arriving host data into the data log based on a comparison between an oldest entry in the data log and an age threshold. The age threshold is dynamically updated based on system heuristics. As long as the oldest log entry is younger than the age threshold, the data log continues to accept newly arriving host application data, acknowledging IO requests to host applications as the data specified in those requests is entered into the log. However, when the oldest log entry is older than the age threshold, new log entries are temporarily refused entry into the data log. Instead, newly arriving data are placed in a pending list, where they are kept until the data log is again accepting new log entries.

Abstract: Techniques are described for compressing cache pages from an LRU (Least-Recently-Used) queue so that data takes longer to age off and be removed from the cache. This increases the likelihood that data will be available within the cache upon subsequent re-access, reducing the need for costly disk accesses due to cache misses.

Abstract: A method is used in managing log transactions in storage systems. A first portion of a transaction log space is allocated from a persistent journal associated with a file system to a first core of a multi-core storage processor and a second portion of the transaction log space is allocated from the persistent journal to a second core of the multi-core storage processor. A set of per-core bitmaps is used for allocating a set of transaction log entries from the persistent journal. Each per-core bitmap is associated with a respective core of the multi-core storage processor. A per-core bitmap of the set of per-core bitmaps associated with a core of the multi-core storage processor is used for allocating a transaction log entry for a write request performed by the core.

Abstract: A method is used in managing caches in storage systems. A set of block entries is reserved in a reserved block entries list of a storage system. The reserved block entries list is associated with a cache of the storage system configured to store data of the storage system. A portion of a memory of the storage system is reserved as the cache. Based on a criterion, a determination is made whether a cache entry is available for use in the cache of the storage system. Based on the determination, a block entry is selected for use from the reserved block entries list.

Abstract: A method is used in managing global caches in data storage systems. A cache entry of a global cache of a data storage system is accessed upon receiving a request to perform an I/O operation on a storage object. The cache entry is associated with the storage object. Accessing the cache entry includes holding a reference to the cache entry. A determination is made as to whether the I/O operation is associated with a sequential access. Based on the determination, releasing the reference to the cache entry is delayed.

Abstract: A technique for storing data in a log receives a set of sequential requests, each request specifying data to be written to a data object served by a data storage system. The data specified by respective ones of the requests is placed into respective lockable units of storage for the log, such that the data of no two sequential requests are stored within the same lockable unit. Each lockable unit is locked for access by a writer when being written, and unlocked when not being written, and is configured to store data specified by multiple requests. Completion of each of the requests is acknowledged in response to placing the data specified by the respective request in one of the lockable units of storage. The log may also be partitioned, with individual partitions being used to store data specified by requests having specific request sizes.