Abstract: A computer-implemented method includes receiving a set of basis fingerprints corresponding to image chunks within a basis set of image regions wherein each image region within the basis set of image regions comprises one or more image chunks, and generating a fingerprint for each image chunk of a plurality of selected image chunks within an unprocessed region of a machine image to produce a plurality of sampled fingerprints. The method also includes determining a similarity metric for the unprocessed region from the sampled fingerprints and the basis fingerprints, comparing the similarity metric for the unprocessed region with a selected threshold, and including the unprocessed region within the basis set of image regions in response to determining that the similarity metric is less than the selected threshold. A corresponding computer program product and computer system are also disclosed herein.

Abstract: Calculating fingerprints for each one of a multiplicity of alignment combinations of fixed-size deduplication data blocks and comparing each of the fingerprints to stored deduplicated data fingerprints in a lookup database for determining a preferred deduplication data block alignment. A deduplication data block comprises each of the fixed-size deduplication data blocks.

Abstract: A method, including identifying, using a sampling ratio, a random number of logical data units. A hash is calculated for each of the identified logical data units, and a first histogram is computed indicating a duplication count of each of the calculated hashes. Based on respective frequencies of the calculated hashes, a second histogram is computed indicating observed frequencies of each of the duplication counts in the first histogram, and based on the sampling ratio and the second histogram, a target function is derived. A range of acceptable results is derived for the target function, and based on the range of the acceptable results, a set of plausible duplication frequency histograms is defined. A first given plausible duplication frequency histogram having a highest number of distinct logical data units is identified, and a second given plausible duplication frequency histogram having a lowest number of distinct logical data units is identified.

Abstract: Methods, computing systems and computer program products implement embodiments of the present invention that include partitioning a dataset into a full set of logical data units, and selecting a sample subset of the full set, the sample subset including a random sample of the full set based on a sampling ratio. A set of target hash values are selected from a full range of hash values, and, using a hash function, a respective unit hash value is calculated for each of the logical data units in the sample subset. A histogram is computed that indicates a duplication count of each of the unit hash values that matches a given target hash value, and based on the histogram, a number of distinct logical data units in the full set is estimated.

Abstract: Various embodiments for identifying data in a data deduplication system, by a processor device, are provided. In one embodiment, a method comprises efficiently identifying duplicate data in the data deduplication system by identifying fingerprint matches using a direct inter-region fingerprint lookup to search for the fingerprint matches in at least one of a plurality of metadata regions, the direct inter-region fingerprint lookup supplementing a central fingerprint index.

Abstract: Various embodiments for data management in a replicated storage environment, by a processor device, are provided. In one embodiment, a method comprises storing a plurality of data replicas under a plurality of heterogeneous compression algorithms, wherein one of the data replicas is optimized for a data operation.

Abstract: Methods, computing systems and computer program products implement embodiments of the present invention that include accessing, from a sequence of multiple data segments including a first data segment at a first location in the sequence followed by additional data segments having respective additional locations in the sequence, a current given data segment in the sequence. In some embodiments, data to be compressed is received and partitioned into the multiple data segments. The current data segment is compressed the current data segment using a first minimal match length, and a compression ratio is calculated for the compressed current data segment. Based on the compression ratio and the respective location of the current data segment, a second minimal match length is selected, a subsequent data segment that immediately follows the current data segment in the sequence is accessed, and the subsequent data segment is compressed using the second minimal match length.

Abstract: Methods, computing systems and computer program products implement embodiments of the present invention that include accessing, from a sequence of multiple data segments including a first data segment at a first location in the sequence followed by additional data segments having respective additional locations in the sequence, a current given data segment in the sequence. In some embodiments, data to be compressed is received and partitioned into the multiple data segments. The current data segment is compressed the current data segment using a first minimal match length, and a compression ratio is calculated for the compressed current data segment. Based on the compression ratio and the respective location of the current data segment, a second minimal match length is selected, a subsequent data segment that immediately follows the current data segment in the sequence is accessed, and the subsequent data segment is compressed using the second minimal match length.

Abstract: Exemplary method, system, and computer program product embodiments for efficient one-pass cache-aware compression are provided. In one embodiment, by way of example only, an output of a fast compressor to Huffman encoding for achieving the one-pass cache-aware compression by using a predetermined Huffman-tree upon determining by the fast compressor a final representation of each data byte.

Abstract: Calculating fingerprints for each one of a multiplicity of alignment combinations of fixed-size deduplication data blocks and comparing each of the fingerprints to stored deduplicated data fingerprints in a lookup database for determining a preferred deduplication data block alignment. A deduplication data block comprises each of the fixed-size deduplication data blocks.

Abstract: For boosting decompression in the presence of reoccurring Huffman trees, a previous Huffman tree is used to decompress a new data block if a match is identified between a compacted description of the Huffman tree and a recently used Huffman tree compaction.

Abstract: For boosting decompression in the presence of reoccurring Huffman trees, a previous Huffman tree is used to decompress a new data block if a match is identified between a compacted description of the Huffman tree and a recently used Huffman tree compaction.

Abstract: For boosting decompressionin the presence of reoccurring Huffman trees, a compacted description of a Huffman tree is compared to recently used Huffman tree compactions for identifying matches rather than reconstructing a Huffman tree from a header of a compressed block for each data block to avoid overhead of a Huffman tree reconstruction.

Abstract: Exemplary method, system, and computer program product embodiments for efficient one-pass cache-aware compression are provided. In one embodiment, by way of example only, an output of a fast compressor to Huffman encoding for achieving the one-pass cache-aware compression by using a predetermined Huffman-tree upon determining by the fast compressor a final representation of each data byte.

Abstract: For boosting decompression in the presence of reoccurring Huffman trees, a compacted description of a Huffman tree is compared to recently used Huffman tree compactions for identifying matches rather than reconstructing a Huffman tree from a header of a compressed block for each data block to avoid overhead of a Huffman tree reconstruction.