Abstract: One example method includes, beginning at a parent node of a data structure, calculating a first W-index for the parent node and a left child node, calculating a second W-index for the parent node and a right child node, when the first W-index exceeds a threshold ?, marking the left child node as a candidate to be included in a sub-cluster separate from a sub-cluster that includes the parent node, when the second W-index exceeds the threshold ?, marking the right child node as a candidate to be included in a sub-cluster separate from the sub-cluster that includes the parent node, and recursively calculating respective W-indices for all other parent nodes in the data structure until all child nodes of the data structure have been assigned to a sub-cluster.

Abstract: One example method includes generating a fingerprint:tag dictionary that includes a group of fingerprints and a group of tags, and the fingerprint:tag dictionary identifies, for each fingerprint, the tag or tags which include that fingerprint, computing a similarity matrix based on the fingerprint:tag dictionary, and the similarity matrix identifies, for each pair of tags in the fingerprint:tag dictionary, a relative similarity of the tags in the pair to each other, running a clustering algorithm to identify groups of similar tags in the similarity matrix, and deduplicating, based on the groups of similar tags, respective data associated with the fingerprints.

Abstract: One example method includes, beginning at a parent node of a data structure, calculating a first W-index for the parent node and a left child node, calculating a second W-index for the parent node and a right child node, when the first W-index exceeds a threshold ?, marking the left child node as a candidate to be included in a sub-cluster separate from a sub-cluster that includes the parent node, when the second W-index exceeds the threshold ?, marking the right child node as a candidate to be included in a sub-cluster separate from the sub-cluster that includes the parent node, and recursively calculating respective W-indices for all other parent nodes in the data structure until all child nodes of the data structure have been assigned to a sub-cluster.

Abstract: One example method includes receiving at a worker node, from each deduplication filesystem (DDFS) node in a group of DDFS nodes, respective parts of a fingerprint-file object dictionary, wherein each of the parts is created using a common fingerprint mask, merging, by the worker node, the parts, scanning the merged parts to update an intersection count between file objects stored at the DDFS nodes, generating, based on the scanning, an intersection matrix, and transferring the intersection matrix to a master node.

Abstract: One example method includes, beginning at a parent node of a data structure, calculating a first W-index for the parent node and a left child node, calculating a second W-index for the parent node and a right child node, when the first W-index exceeds a threshold ?, marking the left child node as a candidate to be included in a sub-cluster separate from a sub-cluster that includes the parent node, when the second W-index exceeds the threshold ?, marking the right child node as a candidate to be included in a sub-cluster separate from the sub-cluster that includes the parent node, and recursively calculating respective W-indices for all other parent nodes in the data structure until all child nodes of the data structure have been assigned to a sub-cluster.

Abstract: One example method includes receiving at a worker node, from each deduplication filesystem (DDFS) node in a group of DDFS nodes, respective parts of a fingerprint-file object dictionary, wherein each of the parts is created using a common fingerprint mask, merging, by the worker node, the parts, scanning the merged parts to update an intersection count between file objects stored at the DDFS nodes, generating, based on the scanning, an intersection matrix, and transferring the intersection matrix to a master node.

Abstract: One example method includes finding an Ln segment in a namespace, and adding the Ln segment to a fingerprint hashtable, scanning a container set and locating Ln in the container set, looking for Ln in the fingerprint hashtable, when Ln is found in the fingerprint hashtable, deleting Ln from the fingerprint hashtable, otherwise, retaining Ln in the fingerprint hashtable, when Ln is found in in the fingerprint hashtable, adding all fingerprints, pointed to by Ln, to the fingerprint hashtable, and when all levels of a tree that includes Ln have been checked, identifying as missing, any fingerprints still remaining in the fingerprint hashtable.

Abstract: An intelligent method of scheduling garbage collection (GC) in a storage system. A GC scheduler obtains capacity utilization and ingest rate of the storage system and calculate therefrom a predicted capacity utilization. When the predicted capacity utilization reaches a threshold, the GC scheduler invokes GC, but otherwise skips GC until such time as predicted capacity utilization reaches the threshold. The ingest rage may be calculated by performing linear fit on past data ingest. The GC scheduler may calculate predicted capacity utilization periodically according to preset period. The GC scheduler may calculate the predicted capacity utilization to a future date beyond the next period. The future date may be at least as far as the next period plus total ingest time.

Abstract: One example method includes generating a fingerprint:tag dictionary that includes a group of fingerprints and a group of tags, and the fingerprint:tag dictionary identifies, for each fingerprint, the tag or tags which include that fingerprint, computing a similarity matrix based on the fingerprint:tag dictionary, and the similarity matrix identifies, for each pair of tags in the fingerprint:tag dictionary, a relative similarity of the tags in the pair to each other, running a clustering algorithm to identify groups of similar tags in the similarity matrix, and deduplicating, based on the groups of similar tags, respective data associated with the fingerprints.

Abstract: An intelligent method of scheduling garbage collection (GC) in a storage system. A GC scheduler obtains capacity utilization and ingest rate of the storage system and calculate therefrom a predicted capacity utilization. When the predicted capacity utilization reaches a threshold, the GC scheduler invokes GC, but otherwise skips GC until such time as predicted capacity utilization reaches the threshold. The ingest rage may be calculated by performing linear fit on past data ingest. The GC scheduler may calculate predicted capacity utilization periodically according to preset period. The GC scheduler may calculate the predicted capacity utilization to a future date beyond the next period. The future date may be at least as far as the next period plus total ingest time.