Abstract: A DSM component is organized as a matrix of page. The data structure of a set of data structures occupies a column in the matrix of pages. A recovery file is maintained in a persistent storage. The recovery file consists of entries and each one of the entries corresponds to a column in the matrix of pages by a location of each one of the entries.

Abstract: A deduplication storage system and a backup application create a synthetic backup. Metadata instructions are provided to the deduplication storage system. Each of the metadata instructions specifies the data segment of an originating backup and a designated location of the data segment in the synthetic backup. A set of metadata instructions is transformed into a transformed set of metadata instructions.

Abstract: Data matches are calculated in a data deduplication system by matching input and repository digests using a digest based data matching process where the reference digests corresponding to a repository interval of data identified as similar to an input interval of data are loaded into two data structures. The two data structures include a sequential buffer containing digests in a sequence of occurrence in the data and a search structure for searching of the reference digests matching a version digest.

Abstract: Exemplary method, system, and computer program product embodiments for scalable data deduplication working with small data chunk in a computing environment are provided. In one embodiment, by way of example only, for each small data chunk, a signature is generated based on a combination of a representation of characters used in selecting data to be deduplicated. A c-spectrum of the small data chunk being a sequence of representations of different characters ordered by a frequency of occurrence in the small data chunk, and an f-spectrum of the small data chunk being a corresponding sequence of frequencies of the different characters in the small data chunk.

Abstract: Embodiments for distributed data deduplication in a grid of processors. Input data is received on a processor. The input data is partitioned into a plurality of similarity units. A corresponding deduplication metadata slice and owning processor for one of the similarity units is calculated. A representative value and corresponding digest values of the similarity unit are sent to the owning processor. The owning processor is used to search for the representative value in the deduplication metadata slice, and to send a specification and owning processors of calculated identical data sections to the processor. The processor is used to send nominal information of the calculated identical data sections to the owning processors of the data referenced by the calculated identical data sections.

Abstract: Embodiments for rebalancing distributed deduplication metadata by a processor. An input similarity unit of data and a corresponding input representative value are received on an ingesting processor. A corresponding deduplication metadata slice and owning processor are calculated for the input similarity unit. The input representative value and input digest values are sent to the owning processor. The owning processor is used to search for the input representative value in a corresponding deduplication metadata slice, and to forward the input representative value and input digest values to an additional processor, if the input representative value is not found by the owning processor and a rebalancing status of the owning processor is in-process. The additional processor is used to send a reply message to the owning processor that facilitates migration of the input representative value and corresponding input digest values to the owning processor, if the input representative value is found.

Abstract: Embodiments for storing data deduplication metadata in a grid of processors. Each of a plurality of slices of deduplication metadata is assigned to be stored by a corresponding processor in a grid of processors. Each slice of the plurality of slices includes at least one of a slice of a similarity index and groups of digests corresponding to those of a plurality of representative values in the slice of the similarity index. A hashing method is use to map between a plurality of input representative values and the plurality of slices of deduplication metadata.

Abstract: Embodiments for processing tracked blocks in a data storage implemented with data deduplication by a processor. Input snapshot data is partitioned into changed tracked blocks. The changed tracked blocks are grouped into enclosing similarity units. Similarity units that contain at least one input changed tracked block are processed for deduplication.

Abstract: Embodiments for calculating a representative value for an input similarity unit in data deduplication of snapshots data by a processor. A corresponding similarity unit of a previous snapshot is identified. A calculation based on digests of the input similarity unit and digests of the corresponding similarity unit is performed. A representative value is produced for the input similarity unit.

Abstract: A deduplication storage system and a backup application create a synthetic backup. Metadata instructions are provided to the deduplication storage system. Each of the metadata instructions specifies the data segment of an originating backup and a designated location of the data segment in the synthetic backup. Each of the metadata instructions are processed by locating those data sub-segments in the deduplication storage system specified by the data segment in each of the metadata instructions, and creating metadata references to each of the data sub-segments and adding the metadata references to metadata of the synthetic backup being created.

Abstract: Applying a content defined maximum size bound on blocks produced by content defined segmentation of data by calculating the size of the interval of data between a newly found candidate segmenting position and a last candidate segmenting position of the same or higher hierarchy level, and then using the intermediate candidate segmenting positions of that interval if the size of the interval exceeds the maximum size bound, or discarding the intermediate candidate segmenting positions of that interval if the size of the interval does not exceed the maximum size bound.

Abstract: Data file in the data deduplication system are associated with a file identifier defined to have a first part identifier for denoting a location of the data file in a storage, and a second part identifier for uniquely identifying the data file in the data deduplication system over time.

Abstract: For reducing digests storage consumption in a data deduplication system using a processor device in a computing environment, digest values are calculated for input data. The digest values are used to locate matches with data stored in a repository. The digest values are stored in the repository. The digest values of the data stored in the repository that is determined to be redundant with the input data are removed.

Abstract: Embodiments for deduplication of snapshots data in a data storage by a processor. Input snapshot data is partitioned into input similarity units. An input similarity unit is deduplicated with a corresponding similarity unit of a previous snapshot. Deduplication coverage is examined. If a deduplication coverage threshold is not met, a similarity search is conducted and the input similarity unit is deduplicated with a found similarity unit.

Abstract: Embodiments for combining input data matches in data deduplication of input data by a processor. Matches of input data are calculated using a plurality of independent deduplication processes referencing a plurality of repository data segments for the input data. A combined list of output data matches is calculated by removing those of the input data matches that are fully enclosed within other input data matches; and removing those of the input data matches determined to be smaller than a predetermined threshold for citing. A deduplication operation is performed on the combined list of output data matches. Each pair of the input data matches having an overlap section is processed in an ascending order of a position.

Abstract: For reducing digests storage consumption in a data deduplication system using a processor device in a computing environment, input data is partitioned into chunks, and the chunks are grouped into chunk sets. Digests are calculated for input data and stored in sets corresponding to the chunk sets. Similarity elements are calculated for the input data and the similarity elements are stored in a similarity search structure, and the number of similarity elements associated with a chunk set which are currently contained in the similarity search structure is maintained for each chunk set.

Abstract: Embodiments for data deduplication of an initial snapshot of a data set in a storage system by a processor. An intra-snapshot similarity index, inclusive of representations of the data inside the initial snapshot, is built. The intra-snapshot similarity index is used for deduplication of the initial snapshot. The intra-snapshot similarity index is merged with a global similarity index.

Abstract: Embodiments for deduplication of snapshots data in a data storage by a processor. Input snapshot data is partitioned into input similarity units. An input similarity unit is deduplicated with a corresponding similarity unit of a previous snapshot. Deduplication coverage is examined. If a deduplication coverage threshold is not met, a similarity search is conducted and the input similarity unit is deduplicated with a found similarity unit.

Abstract: Embodiments for calculating a representative value for an input similarity unit in data deduplication of snapshots data by a processor. A corresponding similarity unit of a previous snapshot is identified. A calculation based on digests of the input similarity unit and digests of the corresponding similarity unit is performed. A representative value is produced for the input similarity unit.

Abstract: Embodiments for reducing resource consumption of a similarity index in data deduplication by a processor. Only a latest generation of repository data is represented in the similarity index. Implicit deletion is applied in the similarity index. A subset of bytes of a full representative value is maintained in a similarity index entry.