Abstract: Technologies are presented for data deduplication that operates at relatively high throughput and with relatively less storage space than conventional techniques. Building upon content-dependent chunking (CDC) using Rabin fingerprints, data may be fingerprinted and stored in variable-size chunks. In some examples, data may be chunked on multiple levels, for example, two levels, variable size large chunks in the first level and fixed-size sub-chunks in the second level, in order to prevent sub-chunks common to two or more data chunks from not being deduplicated. For example, at a first level, a CDC algorithm may be employed to fingerprint and chunk data in content-dependent sizes (variable sizes), and at a second level the CDC chunks may be sliced into small fixed-size chunks. The sliced CDC chunks may then be used for deduplication.

Abstract: Technologies are presented for data deduplication that operates at relatively high throughput and with relatively less storage space than conventional techniques. Building upon content-dependent chunking (CDC) using Rabin fingerprints, data may be fingerprinted and stored in variable-size chunks. In some examples, data may be chunked on multiple levels, for example, two levels, variable size large chunks in the first level and fixed-size sub-chunks in the second level, in order to prevent sub-chunks common to two or more data chunks from not being deduplicated. For example, at a first level, a CDC algorithm may be employed to fingerprint and chunk data in content-dependent sizes (variable sizes), and at a second level the CDC chunks may be sliced into small fixed-size chunks. The sliced CDC chunks may then be used for deduplication.