Abstract: A system and method for generating synthetic data to simulate backing up data between a primary storage system and a protection storage system is presented. In one embodiment, a first track in a set of tracks is selected at random. Having selected a first track, at least a first block in the first track is modified. Subsequently, it is determined, based on a track run probability, whether to modify a second track that is consecutive to the first track or a third track that is selected randomly. Depending on the determination, at least one block is modified at either the second or third track. Other embodiments are also described herein.

Abstract: A computer-implemented method and system for improving efficiency in a delta compression process selects a data chunk to delta compress and generates a sketch for the selected data chunk. A set of candidate data chunks with a matching sketch is searched for. The set of candidate data chunks with at least a minimum degree of similarity is ranked by location status data. Tie-breaking of the set of candidate data chunks is done using a degree of sketch similarity for each candidate and the selected data chunk is delta compressed with a selected candidate data chunk.

Abstract: A computer-implemented method and system for improving efficiency in a delta compression process selects a data chunk to delta compress and generate matching criteria for the selected data chunk. The method and system searches for a similar data chunk using the matching criteria and loads a best match data chunk from the search. Locality data for the best match data chunk is determined and a plurality of data chunks with matching locality data are pre-fetched.

Abstract: A computer-implemented method and system for performing garbage collection in a delta compressed data storage system selects a file recipe to traverse to identify live data chunks and selects a chunk identifier from the file recipe. The chunk identifier is added to a set of live data chunks. Delta references in the file metadata corresponding to the chunk identifier are added to the set of live data chunks. Data chunks in a data storage system not identified by the set of live data chunks are then discarded.

Abstract: A computer-implemented method for indexing content stored in a cache memory device is disclosed. The method starts with in response to receiving a first request for caching a first file extent associated with a first file in a cache memory device, generating a first fingerprint based on content of the first file extent. Then the method continues with searching in a fingerprint index based on the first fingerprint to determine whether the first file extent has been stored in the cache memory. In response to determining that a fingerprint entry matching the first fingerprint is found, the method then continues with associating a first identifier identifying the first file extent and the first file with a storage location of the cache memory device obtained from the matching fingerprint entry, without storing the first file extent in the cache memory device.

Abstract: A computer-implemented method and system for improving efficiency in a delta compression process in a data storage system selects a data chunk to delta compress and generates a sketch for the selected data chunk. The method and system search for a set of candidate data chunks with a matching sketch and rank the set of candidate data chunks by degree of sketch matching. The set of candidate data chunks are tie-braked using location status data for each candidate and the selected data chunk is delta compressed with a selected candidate data chunk. The delta compressed selected data chunk is then stored in a data storage system.

Abstract: A method for storing data in a data storage system by partitioning the data into a plurality of data chunks and generating representative data for each of the plurality of chunks by applying a predetermined algorithm to each chunk of the plurality of chunks. Subsequently, the representative data is compared and sorted. Representative data for base data chunks and representative data for other data chunks that can be stored relative to the base data chunks are identified by evaluating the sorted set of representative data. Finally, each of the other data chunks identified as those that can be stored relative to a base data chunk are stored in the data storage system as the difference between the data chunk and a base data chunk.

Abstract: Embodiments of this invention are directed to a system and method for characterizing and modeling a virtual synthetic file system workload. In one embodiment, a virtual synthetic system is adapted to select a first location in a prior generation dataset of a first cluster and generate a first offset using a distance distribution function. Thereafter, the virtual synthetic system selects a second location in the prior generation dataset of a second cluster, wherein the second location is offset from the first cluster by the first offset. Finally, the virtual synthetic system modifies each cluster selected on the prior generation dataset thereby creating a next generation dataset. This process is repeated to generate multiple generations of a dataset. Other embodiments are also described herein.

Abstract: Techniques for detecting unwanted data are described herein. In one embodiment, a request is received for storing a data object in a storage system from a client over a network, where the request includes first representative data representing the data object without including actual content of the data object. It is detected whether the data object contains unwanted content by comparing the first representative data with second representative data without accessing the actual content of the data object, where the second representative data represents the unwanted content. A response is transmitted to the client over the network indicating whether the data object is likely to contain the unwanted object based on comparison of the first and second representative data.

Abstract: Techniques for searching data in a storage system are described herein. In one embodiment, in response to a request for searching target data in a storage system, first representative data for the target data being searched are generated by applying a predetermined algorithm to at least a portion of the target data. The first representative data are searched and compared with second representative data representing one or more data sets stored in the storage system. It is indicated a likelihood that the target data or similar content has been found in the storage system based on the search and comparison.

Abstract: Techniques for replicating data chunks in a storage system are described herein. In one embodiment, in response to a request for replicating data chunks of a source storage system having a first average chunk size to a target storage system having a second average chunk size, a new chunk size is determined based on metadata of the data chunks in view of an average chunk size of the target storage system. The data chunks are resized based on the new chunk size to generate resized data chunks. The resized data chunks are transmitted from the source storage system to the target storage system for replication.

Abstract: A computer-implemented method and system for improving efficiency in a delta compression process in a data storage system selects a data chunk to delta compress and selects a set of candidate data chunks using a first selection mechanism. Throughput or resource utilization is monitored. A change is made to a second selection mechanism that increases similarity of the set of candidates with the selected data chunk to improve compression in response to determining high resource availability or high throughput level. A change is made to a third selection mechanism that increases throughput of the delta compression process in response to determining low resources availability or low throughput.

Abstract: Techniques for evaluating deduplication effectiveness of data chunks in a storage system are described herein. In one embodiment, metadata of first data chunks associated with a deduplicated storage system is examined, where the first data chunks have been partitioned according to a first chunk size. A second chunk size is calculated based on the examination of the metadata of first data chunks. Metadata of the first data chunks is merged according to the second chunk size to represent second data chunks to which the first data chunks would have been merged. A deduplication rate of the second data chunks is determined based on the merged metadata.

Abstract: A request for allocating a storage unit of a storage system is received to back up data of one or more clients. The storage system includes multiple storage units and each storage unit storing data that is deduplicated within each storage unit. In response to the request, one or more of the storage units are selected based on an amount of deduplicated data that would be stored in each of the storage units after storing the data of the one or more clients. The selected one or more storage units are allocated to the one or more clients to back up the data of the one or more clients.