Abstract: Systems and methods for estimating the number of workers needed to perform a garbage collection operation are disclosed. Similarity groups are used to identify segments associated with objects in a computing system. Using deletion records that identify objects to be deleted, the similarity groups impacted by the deletion records can be identified. The number of workers can be determined based on the impacted similarity groups. More specifically, the number of impacted similarity groups and/or workers can be evaluated in terms of memory requirements, input/output constraints and/or time requirements to estimate the number or workers needed to clean similarity groups impacted by a garbage collection operation.

Abstract: One example method includes receiving a write request that includes a data structure version to be written, wherein the data structure version is associated with a unique identifier, storing the data structure version in association with the unique identifier, receiving a read request for a most recent version of the data structure and, when the stored data structure version is not the most recent version of the data structure, examining respective unique identifiers of each of a group of other stored data structure versions to determine which stored data structure version is the most recent. Finally, the example method includes returning the most recent data structure version, notwithstanding that one or more other data structure versions existed at the time that the read request was received.

Abstract: A technique for transferring data over a network leverages a standard NAS (Network Attached Storage) protocol to augment its inherent file-copying ability with fingerprint matching, enabling the NAS protocol to limit its data copying over the network to unique data segments while avoiding copying of redundant data segments.

Abstract: A method, computer program product, and computing system for receiving a candidate data portion; calculating a distance-preserving hash for the candidate data portion; and performing an entropy analysis on the distance-preserving hash to generate a hash entropy for the candidate data portion.

Abstract: Techniques to manage usage of a flash-based storage are disclosed. In various embodiments, the execution time of the flash-based storage is divided into quanta. Within each quantum comprising at least a subset of quanta, flash erasures are allowed without restriction up to a prescribed erasure quota. Erasures are throttled within a slack range bound at a lower end by the erasure quota and at an upper end by an upper bound, including by dividing the slack range into two or more intervals and within each interval applying a corresponding erasure control policy, wherein the respective corresponding erasure control policies applied to successive intervals in the slack range become increasingly strict in a stepwise manner.

Abstract: Systems and methods for estimating the number of workers needed to perform a garbage collection operation are disclosed. Similarity groups are used to identify segments associated with objects in a computing system. Using deletion records that identify objects to be deleted, the similarity groups impacted by the deletion records can be identified. The number of workers can be determined based on the impacted similarity groups. More specifically, the number of impacted similarity groups and/or workers can be evaluated in terms of memory requirements, input/output constraints and/or time requirements to estimate the number or workers needed to clean similarity groups impacted by a garbage collection operation.

Abstract: Techniques for improving data compression of a storage system using coarse and fine grained similarity are described herein. According to one embodiment, region sketches for a plurality of regions of the set of data are generated, each region storing a plurality of data chunks. A region sketch index having a plurality of entries is maintained, each corresponding to one of the region sketches of the regions. The entries of the region sketch index are sorted based on the sketches of the regions, such that regions with an identical region sketch are positioned adjacent to each other within the region sketch index, representing similar regions. The data chunks of the similar regions that are identified based on the sorted entries of the region sketch index are reorganized to improve data compression of the data chunks of the similar regions.

Abstract: One embodiment is related to a method for applying deduplication and/or compression to data stored or to be stored at a non-volatile memory (NVM) cache comprising: identifying the NVM cache; determining whether deduplication or compression, or both, is to be applied to data stored or to be stored at the NVM cache; and applying either deduplication or compression, or both, to the data stored or to be stored at the NVM cache.

Abstract: Systems and methods for performing data protection operations including garbage collection operations and copy forward operations. For deduplicated data stored in a cloud-based storage or in a cloud tier that stores containers containing dead and live segments or dead and live regions such as compression regions, the dead compression regions are deleted by copying the live compression regions into new containers and then deleting the old containers. The copy forward is based on a recipe from a data protection system and is performed using a serverless approach.

Abstract: A method, computer program product, and computing system for encoding a candidate data portion to generate an encoded candidate data portion; identifying one or more portion similarities between the encoded candidate data portion and an encoded target data portion to position the one or more portion similarities with respect to the encoded target data portion, thus generating one or more portion similarity measurements; identifying one or more portion differences between the encoded candidate data portion and the encoded target data portion to generate one or more portion difference measurements; and combining the one or more portion similarity measurements and the one or more portion difference measurements to generate a candidate similarity measurement for the candidate data portion.

Abstract: A source site includes a controller, a set of source worker nodes, and a message queue connected between the controller and source worker nodes. The message queue receives messages and stores the messages for retrieval. A destination site includes a set of destination worker nodes. Tasks are generated to replicate changes to objects at the source site to the destination site. The controller pushes messages corresponding to the tasks onto the message queue. A source worker node retrieves a message corresponding to a task from the message queue for processing in conjunction with a destination worker node. The message is indicated as having been retrieved from the message queue.

Abstract: A method, computer program product, and computing system for performing an entropy analysis on each of a plurality of candidate data chunks associated with a potential candidate to generate a plurality of candidate data chunk entropies; performing an entropy analysis on each of a plurality of target data chunks associated with a potential target to generate a plurality of target data chunk entropies; identifying a candidate data chunk entropy limit, chosen from the plurality of candidate data chunk entropies, and a target data chunk entropy limit, chosen from the plurality of candidate data chunk entropies; and comparing a specific candidate data chunk associated with the candidate data chunk entropy limit to a specific target data chunk associated with the target data chunk entropy limit to determine if the specific candidate data chunk and the specific target data chunk are identical.

Abstract: A method, computer program product, and computing system for processing a data portion to divide the data portion into a plurality of data chunks; performing an entropy analysis on each of the plurality of data chunks to generate a plurality of data chunk entropies; and determining an average data chunk entropy from the plurality of data chunk entropies.

Abstract: Systems and methods for performing data protection operations including garbage collection operations and copy forward operations. For deduplicated data stored in a cloud-based storage or in a cloud tier that stores containers containing dead and live segments, the dead segments are deleted by copying live segments into new containers and then deleting the old containers. The copy forward is based on a recipe from a data protection system and is performed using a microservices that can be run as needed in the cloud.

Abstract: Systems and methods for performing data protection operations including garbage collection operations and copy forward operations. For deduplicated data stored in a cloud-based storage or in a cloud tier that stores containers containing dead and live regions such as compression regions, the dead segments in the dead compression regions are deleted by copying the live compression regions into new containers and then deleting the old containers. The copy forward is based on a recipe from a data protection system and is performed using a microservices based approach.

Abstract: 4th 115078 A method, computer program product, and computing system for encoding a candidate data portion to generate an encoded candidate data portion; identifying one or more portion similarities between the encoded candidate data portion and an encoded target data portion to position the one or more portion similarities with respect to the encoded target data portion, thus generating one or more portion similarity measurements; identifying one or more portion differences between the encoded candidate data portion and the encoded target data portion to generate one or more portion difference measurements; and combining the one or more portion similarity measurements and the one or more portion difference measurements to generate a candidate similarity measurement for the candidate data portion.

Abstract: Systems and methods for estimating the number of workers needed to perform a garbage collection operation are disclosed. Similarity groups are used to identify segments associated with objects in a computing system. Using deletion records that identify objects to be deleted, the similarity groups impacted by the deletion records can be identified. The number of workers can be determined based on the impacted similarity groups. More specifically, the number of impacted similarity groups and/or workers can be evaluated in terms of memory requirements, input/output constraints and/or time requirements to estimate the number or workers needed to clean similarity groups impacted by a garbage collection operation.

Abstract: Systems and methods enabling garbage collection operations and normal system operations concurrently. Concurrent operations are performed by configuring a similarity group to permit garbage collection and normal operations. This may include creating a new subgroup in a similarity group for write and deduplication purposes such that an impacted subgroup can be cleaned.

Abstract: Systems and methods for marking similarity groups impacted by a garbage collection operation are disclosed. Similarity groups are used to identify segments associated with objects in a computing system. Using deletion records that identify objects to be deleted, the similarity groups impacted by the deletion records can be identified. The live segments associated with the impacted similarity groups are also identified.

Abstract: Systems and methods enabling garbage collection operations and normal system operations concurrently. Concurrent operations are performed by configuring a similarity group to permit garbage collection and normal operations. This may include creating a new subgroup in a similarity group for write and deduplication purposes such that an impacted subgroup can be cleaned.