Abstract: A data management device includes a persistent storage and a processor. The persistent storage includes meta-data of data stored in a long term retention (LTR) storage. The processor obtains a file storage request for a file and deduplicates the file against segments stored in the LTR storage while performing garbage collection on the LTR storage. Performing garbage collection includes deleting segments of the data stored in the LTR storage using the meta-data. The meta-data is not stored in the LTR storage.

Abstract: A data management device includes a persistent storage and a processor. The persistent storage includes an object storage. The processor generates a collision free hash function based on segments stored in the object storage. The processor generates a hash vector using the collision free hash function. The processor deduplicates the segments using the hash vector. The processor stores the deduplicated segments in the object storage.

Abstract: In an embodiment, a system and method for supporting a seeding process with suspend and resume capabilities are described. A resumable seeding component in a data seeding module can be used to move data from a source tier to a target tier. A resumption context including a perfect hash function (PHF) and a perfect hash vector (PHV) persists a state of a seeding process at the end of each operation in the seeding process. The PHV represents data segments of the data using the PHF. The resumption context is loaded into memory upon resumption of the seeding process after it is suspended. Information in the resumable context is used to determine a last successfully completed operation, and a last copied container. The seeding process is resumed by executing an operation following the completed operation in the resumable context.

Abstract: A data management device includes a persistent storage and a processor. The persistent storage includes an object storage that stores segments. The processor generates a collision free hash function based on the segments, generates a hash vector using the collision free hash function, deduplicates a portion of the segments associated with to-be-migrated files using the hash vector, and migrates the to-be-migrated files using the deduplicated portion of the segments to a remote storage.

Abstract: In an embodiment, a method for validating data integrity of a seeding process is described. The seeding process for migrating data from a source tier to a target tier persists a perfect hash vector (PHV) to a disk when the seeding process is suspended for various reasons. The PHV includes bits for fingerprints for data segments corresponding to the data, and can be reloaded into memory upon resumption of the seeding process. One or more bits corresponding to fingerprints for copied data segments are reset prior to starting the copy phase in the resumed run. A checksum of the PHV is calculated after the seeding process completes copying data segments in the containers. A non-zero checksum of the PHV indicates that one or more data segments are missing on the source tier or the data segments are not successfully copied to the target tier. The missing data segments and/or one or more related files are reported to a user via a user interface.

Abstract: In an embodiment, a method for validating data integrity of a seeding process is described. The seeding process for migrating data from a source tier to a target tier persists a perfect hash vector (PHV) to a disk when the seeding process is suspended for various reasons. The PHV includes bits for fingerprints for data segments corresponding to the data, and can be reloaded into memory upon resumption of the seeding process. One or more bits corresponding to fingerprints for copied data segments are reset prior to starting the copy phase in the resumed run. A checksum of the PHV is calculated after the seeding process completes copying data segments in the containers. A non-zero checksum of the PHV indicates that one or more data segments are missing on the source tier or the data segments are not successfully copied to the target tier. The missing data segments and/or one or more related files are reported to a user via a user interface.

Abstract: In an embodiment, a system and method for supporting a seeding process with suspend and resume capabilities are described. A resumable seeding component in a data seeding module can be used to move data from a source tier to a target tier. A resumption context including a perfect hash function (PHF) and a perfect hash vector (PHV) persists a state of a seeding process at the end of each operation in the seeding process. The PHV represents data segments of the data using the PHF. The resumption context is loaded into memory upon resumption of the seeding process after it is suspended. Information in the resumable context is used to determine a last successfully completed operation, and a last copied container. The seeding process is resumed by executing an operation following the completed operation in the resumable context.

Abstract: In general, in one aspect, the invention relates to a method for managing persistent storage in a storage system. The method includes determining, using a first plurality of containers in the storage system, a locality threshold, and performing, using the locality threshold, a locality repair on a first container of a second plurality of containers in the storage system, wherein the second plurality of containers comprises the first plurality of container.

Abstract: A computer-implemented method is disclosed. The method starts with determining a first container of a storage system is invalid. The method continues with the storage system setting a data recovery state for the first container to be en-queue, which indicates that data of at least one of the data segments needs to be recovered from the first container, and executing a process to recover any container having an en-queue data recovery state, and for each of the containers, to recover any valid data segment from the corresponding container. The process includes scanning the data segments of the first container to find valid data segments, moving or replicating the valid data segments to a second container, and setting the data recovery state for the first container to be complete once all the valid data segments are moved or replicated to the second container.