Abstract: Systems and methods for writing data are provided. A lock-free container and methods of writing to the lock-free container are disclosed. The container is associated with a tail pointer that identifies free space in the container. Threads writing to the container access the tail pointer and update an offset in the tail pointer to account for a size of a write to the container. Multiple threads can write to the same container without having to contend for a container lock.

Abstract: A computer-implemented method for compressing a data set, the method comprising receiving a first data block of the data set, selecting automatically by a compression management module a compression module from a plurality of compression modules to apply to the first data block based on projected compression efficacy or resource utilization, and compressing the first data block with the selected compression module to generate a first compressed data block.

Abstract: A computer-implemented method for compressing a data set, the method comprising receiving a first data block of the data set, selecting automatically by a compression management module a compression module from a plurality of compression modules to apply to the first data block based on projected compression efficacy or resource utilization, and compressing the first data block with the selected compression module to generate a first compressed data block.

Abstract: A method for storing data in a data storage system by partitioning the data into a plurality of data chunks and generating representative data for each of the plurality of chunks by applying a predetermined algorithm to each chunk of the plurality of chunks. Subsequently, the representative data is compared and sorted. Representative data for base data chunks and representative data for other data chunks that can be stored relative to the base data chunks are identified by evaluating the sorted set of representative data. Finally, each of the other data chunks identified as those that can be stored relative to a base data chunk are stored in the data storage system as the difference between the data chunk and a base data chunk.

Abstract: A computer-implemented method for compressing a data set, the method comprising receiving a first data block of the data set, selecting automatically by a compression management module a compression module from a plurality of compression modules to apply to the first data block based on projected compression efficacy or resource utilization, and compressing the first data block with the selected compression module to generate a first compressed data block.

Abstract: Techniques for transferring modified data with efficiency are described herein. According to one embodiment, the method starts with analyzing host write data to determine a host write pattern of a host writing to sectors at a primary storage, and then a data block size and a backup interval are determined based on at least one of the host write pattern, wherein a data block includes a number of sectors that is determined based on the host write pattern, and known characteristics of transferring dirty data blocks of the primary storage, wherein a dirty data block includes at least one sector that has been changed from a previous backup. Then dirty data blocks are transferred from the primary storage to a secondary storage as part of a backup.

Abstract: A cost function is determined for assigning first deduplicating storage units of a first storage system for replication onto second deduplicating storage units of a second storage system. One or more of the first storage units in the first storage system are assigned to one or more of the second storage units in the second storage system based on a minimized cost resulting from the cost function.

Abstract: Techniques for sanitizing a storage system are described herein. In one embodiment, for each file stored in the storage system, a list of fingerprints representing data chunks of the file is obtained. In such an embodiment, for each of the fingerprints, identifying a first container storing a data chunk corresponding to the fingerprint is identified, and determining a storage location of the first container in which the data chunk is stored is determined. In one embodiment, a bit in copy bit vector (CBV) is populated based on the identified container and the storage location. In one embodiment, after all of the bits corresponding to the data chunks of the first container have been populated in the CBV, data chunks represented by the CBV are copied from the first container to a second container, and records of the data chunks in the first container are erased.

Abstract: A computer-implemented method for performing garbage collection in a delta compressed data storage system selects a file recipe to traverse to identify live data chunks and a chunk identifier from the file recipe. The chunk identifier is added to a set of live data chunks. Delta references in the file recipe corresponding to the chunk identifier are added to the set of live data chunks. Data chunks in a data storage system not identified by the set of live data chunks are discarded.

Abstract: A computer-implemented method and system for performing garbage collection in a delta compressed data storage system selects a file recipe to traverse to identify live data chunks and selects a chunk identifier from the file recipe. The chunk identifier is added to a set of live data chunks. Delta references in an entry of an index corresponding to the chunk identifier are added to the set of live data chunks. Data chunks in a data storage system not identified by the set of live data chunks are then discarded.

Abstract: A computer-implemented method for indexing content stored in a cache memory device is disclosed. The method starts with maintaining a fingerprint index having a plurality of fingerprint entries, each mapping a fingerprint to a storage location of a cache memory device, where the cache memory device caches some of data blocks stored in a persistent storage device of a storage system, and where the fingerprint index is a partial index indexing a portion of data stored in the cache memory device. In response to receiving a request to insert a new fingerprint, the method continues with evicting one of the fingerprint entries according to a predetermined eviction algorithm and inserting the new fingerprint into the evicted fingerprint entry.

Abstract: A computer-implemented method for indexing content stored in a cache memory device is disclosed. The method starts with maintaining a file index having a plurality of extent entries, each extent entry corresponding to one of a plurality of file extents stored in a cache memory device that caches data stored in a persistent storage device of a storage system. The method continues with maintaining a fingerprint index having a plurality of fingerprint entries, each mapping a fingerprint to a data region of a file indexed in the file index, wherein each fingerprint indexed in the fingerprint index is retrieved from metadata stored in the persistent storage device of the storage system when one or more corresponding data chunks were accessed, and deduplicating and accessing the file extents stored in the cache memory device using the file index and the fingerprint index.

Abstract: Techniques for sanitizing a storage system are described herein. In one embodiment, for each of fingerprints representing data chunks stored in a first container of the storage system, a lookup operation in a live bit vector based on the fingerprint is performed to determine whether a corresponding data chunk is live. In one embodiment, a bit in a copy bit vector corresponding to the data chunk is populated based on the lookup operation. In one embodiment, after all of the bits corresponding to the data chunks of the first container have been populated in the CBV, data chunks represented by the CBV are copied from the first container to a second container, and records of the data chunks in the first container are erased.

Abstract: A computer-implemented method for caching content in a cache memory device is disclosed. The method starts with receiving a request for accessing a first data block associated with a first file, and a file manager provides access of the first data block in a persistent storage device of a storage system. The file manager then caches the first data block in a cache memory device including deduplicating the first data block, wherein at least some of data blocks stored in the cache memory device are deduplicated data blocks, and wherein at least one of the data blocks is referenced by different regions of an identical file or different files.

Abstract: A computer-implemented method and system for improving efficiency in a delta compression process in a data storage system selects a data chunk to delta compress and selects a set of candidate data chunks using a first selection mechanism. Throughput or resource utilization is monitored. A change is made to a second selection mechanism that increases similarity of the set of candidates with the selected data chunk to improve compression in response to determining high resource availability or high throughput level. A change is made to a third selection mechanism that increases throughput of the delta compression process in response to determining low resources availability or low throughput.

Abstract: A computer-implemented method and system for improving efficiency in a delta compression process in a data storage system selects a data chunk to delta compress and generates a sketch for the selected data chunk. The method and system search for a set of candidate data chunks with a matching sketch and rank the set of candidate data chunks by degree of sketch matching. The set of candidate data chunks are tie-braked using location status data for each candidate and the selected data chunk is delta compressed with a selected candidate data chunk. The delta compressed selected data chunk is then stored in a data storage system.

Abstract: Techniques for reducing overheads of primary storage transferring during a backup by transferring in an out-of-order manner are described herein. According to one embodiment, in response to a request at a primary storage for a backup of a plurality of data blocks, a transfer order of dirty data blocks is determined based on an access assessment of the dirty data blocks, wherein the dirty data blocks are data blocks of the plurality of data blocks that have been changed from a previous backup, and wherein the transfer order is different from a sequential logical order of the dirty data blocks provided by the primary storage. Then the dirty data blocks are transferred to a secondary storage in the determined order.

Abstract: A computer-implemented method for indexing content stored in a cache memory device is disclosed. The method starts with maintaining a file index having a plurality of extent entries, each extent entry corresponding to one of a plurality of file extents stored in a cache memory device that caches data stored in a persistent storage device of a storage system. In response to receiving a request to read a first file region of a first file, the method continues with retrieving first data block from the persistent storage device that contains the first data block, caching the first data block at a first storage location of the cache memory device; and creating a first extent entry in the file index having at least a first node, where the first node includes an address of the first storage location and a first bitmap indicating which data blocks are valid.

Abstract: A computer-implemented method for caching content in a cache memory device is disclosed. The method starts with receiving, at a cache manager, one or more data chunks to be cached in a cache memory device, where the one or more data chunks are retrieved from a persistent storage disk of a storage system in response to a read request of a region of a file. Then the one or more data chunks of a file extent is compressed using a predetermined compression algorithm, and the file extent is packed into a write-evict unit (WEU) maintained in a random-access memory (RAM) that has been open to store a plurality of file extents. In response to determining that the WEU is full, the cache manager writes the WEU from the RAM into the cache memory device.

Abstract: A computer-implemented method for performing garbage collection in a delta compressed data storage system selects a file recipe to traverse to identify live data chunks and a chunk identifier from the file recipe. The chunk identifier is added to a set of live data chunks. Delta references in the file recipe corresponding to the chunk identifier are added to the set of live data chunks. Data chunks in a data storage system not identified by the set of live data chunks are discarded.