Abstract: Techniques for improving data compression of a storage system in an online manner are described herein. According to one embodiment, in response to a sequence of data to be stored, the sequence of data is partitioned into a plurality of data chunks according to a predetermined chunking algorithm. A sketch for each of the data chunks is generated based on one or more features extracted from the data chunk. Each of the data chunks of the sequence of data is associated with one of a plurality of groups based on the sketch, wherein each group is represented by a sketch. The data chunks of each group are compressed and stored in a compression region of the storage systems, such that similar data chunks are compressed and stored in the same compression region.

Abstract: Exemplary methods for minimizing contention among multiple threads include maintaining a plurality of linked lists of elements, each linked list corresponding to one of a plurality of threads accessing cache entries, each element of each linked list corresponding to one of the cache entries, wherein each linked list comprises a head element and a tail element, the head element corresponding to a most recently used (MRU) cache entry among all cache entries accessed by a corresponding thread, and the tail element corresponding to a least recently used cache entry among all cache entries accessed by the corresponding thread. In response to a cache eviction request, determining a LRU cache entry among the plurality of cache entries based on values accessed from one or more of the tail elements of the linked lists, and evicting the determined LRU cache entry by populating the determined LRU cache entry with the new data.

Abstract: Techniques for sanitizing a storage system are described herein. In one embodiment, for each file stored in the storage system, a list of fingerprints representing data chunks of the file is obtained. In such an embodiment, for each of the fingerprints, identifying a first container storing a data chunk corresponding to the fingerprint is identified, and determining a storage location of the first container in which the data chunk is stored is determined. In one embodiment, a bit in copy bit vector (CBV) is populated based on the identified container and the storage location. In one embodiment, after all of the bits corresponding to the data chunks of the first container have been populated in the CBV, data chunks represented by the CBV are copied from the first container to a second container, and records of the data chunks in the first container are erased.

Abstract: In response to a request for stored data, retrieving an index entry, based on an identifier of the requested data, corresponding to an indexed block of storage containing the requested data. The index entry includes a start location of an indexed storage block and sub-block locators that identify the start of one or more sub-blocks within the indexed storage block. The sub-block containing the requested data is determined and the corresponding sub-block locator is read to find the starting location of the sub-block. Without reading the entire indexed storage block, the sub-block may be read from the starting location of the sub-block, decompressed, and the decompressed requested data read from the sub-block may be transmitted to the client. In this way, fewer I/O operations are needed that read the requested data, and memory needed for storing index information is minimized.

Abstract: Data replication with delta compression is disclosed. A primary system and a replica system are determined to both have an identical first data segment that is similar to a second data segment. The second data segment is encoded, wherein the encoding refers to the first data segment.

Abstract: Techniques for improving data compression of a deduplicated storage system are described herein. According to one embodiment, the similarity of a plurality of data chunks stored in one or more first storage areas of the storage system is determined based on a plurality of sketches, each describing characteristics of one of the data chunks. The data chunks are grouped into a plurality of groups of similar data chunks based on the similarity of the data chunks. The groups of similar data chunks are compressed, such that similar data chunks are compressed close to each other.

Abstract: A computer-implemented method for indexing content stored in a cache memory device is disclosed. The method starts with maintaining a fingerprint index having a plurality of fingerprint entries, each mapping a fingerprint to a storage location of a cache memory device, where the cache memory device caches some of data blocks stored in a persistent storage device of a storage system, and where the fingerprint index is a partial index indexing a portion of data stored in the cache memory device. In response to receiving a request to insert a new fingerprint, the method continues with evicting one of the fingerprint entries according to a predetermined eviction algorithm and inserting the new fingerprint into the evicted fingerprint entry.

Abstract: Techniques for improving data compression of a storage system are described herein. According to one embodiment, a first sequence of data is partitioned into a plurality of data chunks in a first sequence order according to a predetermined chunking algorithm. The similarity of the data chunks is determined based on data patterns of the data chunks. The data chunks are reorganized into a second sequence order based on the similarity of the data chunks, the second sequence order being different from the first sequence order. The reorganized data chunks are compressed in the second sequence order into a second sequence of data, such that similar data chunks are stored and compressed together within the second sequence of data.

Abstract: Techniques for sanitizing a storage system are described herein. In one embodiment, for each of fingerprints representing data chunks stored in a first container of the storage system, a lookup operation in a live bit vector based on the fingerprint is performed to determine whether a corresponding data chunk is live. In one embodiment, a bit in a copy bit vector corresponding to the data chunk is populated based on the lookup operation. In one embodiment, after all of the bits corresponding to the data chunks of the first container have been populated in the CBV, data chunks represented by the CBV are copied from the first container to a second container, and records of the data chunks in the first container are erased.

Abstract: A computer-implemented method for caching content in a cache memory device is disclosed. The method starts with receiving a request for accessing a first data block associated with a first file, and a file manager provides access of the first data block in a persistent storage device of a storage system. The file manager then caches the first data block in a cache memory device including deduplicating the first data block, wherein at least some of data blocks stored in the cache memory device are deduplicated data blocks, and wherein at least one of the data blocks is referenced by different regions of an identical file or different files.

Abstract: In one embodiment, in response to a list of a plurality of fingerprints representing data chunks to be replicated from a source storage system to a target storage system over a network, one of the fingerprints is selected as a representative fingerprint that matches selection criteria. The source storage system generates a fingerprint representation for each of the fingerprints that is smaller than the corresponding fingerprint. The representative fingerprint and the fingerprint representations are transmitted to the target storage system over the network, without transmitting all full fingerprints of the data chunks, to allow the target storage system identifying which of the data chunks are missing. In response to information received from the target storage system indicating one or more data chunks that are missing at the target storage system, the missing data chunks are then transmitted to the target storage system.

Abstract: Stream locality delta compression is disclosed. A previous stream indicated locale of data segments is selected. A first data segment is then determined to be similar to a data segment in the stream indicated locale.

Abstract: According to one embodiment, in response to a list of fingerprints representing data chunks of a first batch for replication, a second batch previously transmitted to the target storage system is identified based on at least a portion of the fingerprints of the first batch. Differential encoding information is generated representing a difference between fingerprints of the first batch and the second batch. The differential encoding information is transmitted to the target storage system, without transmitting all full fingerprints of the first batch, to allow the target storage system to determine which of the data chunks are missing at the target storage system. In response to information received from the target storage system indicating one or more data chunks that are missing at the target storage system, the missing data chunks are then transmitted to the target storage system.

Abstract: Stream locality delta compression is disclosed. A previous stream indicated locale of data segments is selected. A first data segment is then determined to be similar to a data segment in the stream indicated locale.

Abstract: Data replication with delta compression is disclosed. A primary system and a replica system are determined to both have an identical first data segment that is similar to a second data segment. The second data segment is encoded, wherein the encoding refers to the first data segment.

Abstract: A computer-implemented method for indexing content stored in a cache memory device is disclosed. The method starts with in response to receiving a first request for caching a first file extent associated with a first file in a cache memory device, generating a first fingerprint based on content of the first file extent. Then the method continues with searching in a fingerprint index based on the first fingerprint to determine whether the first file extent has been stored in the cache memory. In response to determining that a fingerprint entry matching the first fingerprint is found, the method then continues with associating a first identifier identifying the first file extent and the first file with a storage location of the cache memory device obtained from the matching fingerprint entry, without storing the first file extent in the cache memory device.

Abstract: A method for storing data in a data storage system by partitioning the data into a plurality of data chunks and generating representative data for each of the plurality of chunks by applying a predetermined algorithm to each chunk of the plurality of chunks. Subsequently, the representative data is compared and sorted. Representative data for base data chunks and representative data for other data chunks that can be stored relative to the base data chunks are identified by evaluating the sorted set of representative data. Finally, each of the other data chunks identified as those that can be stored relative to a base data chunk are stored in the data storage system as the difference between the data chunk and a base data chunk.

Abstract: Data replication with delta compression is disclosed. A primary system and a replica system are determined to both have an identical first data segment that is similar to a second data segment. The second data segment is encoded, wherein the encoding refers to the first data segment.

Abstract: Techniques for detecting unwanted data are described herein. In one embodiment, a request is received for storing a data object in a storage system from a client over a network, where the request includes first representative data representing the data object without including actual content of the data object. It is detected whether the data object contains unwanted content by comparing the first representative data with second representative data without accessing the actual content of the data object, where the second representative data represents the unwanted content. A response is transmitted to the client over the network indicating whether the data object is likely to contain the unwanted object based on comparison of the first and second representative data.

Abstract: Techniques for searching data in a storage system are described herein. In one embodiment, in response to a request for searching target data in a storage system, first representative data for the target data being searched are generated by applying a predetermined algorithm to at least a portion of the target data. The first representative data are searched and compared with second representative data representing one or more data sets stored in the storage system. It is indicated a likelihood that the target data or similar content has been found in the storage system based on the search and comparison.