Abstract: Destination namespace and file copying: a namespace service receives communication of namespace update for file from file's source. and communicates namespace update for file to an access object service identified for file. The access object service receives communication of fingerprints stream, corresponding to file's segments, from file's source, and identifies sequential fingerprints in fingerprints stream as fingerprints group. The access object service identifies group identifier for fingerprints group, and communicates fingerprints group to a deduplication service associated with group identifier range including group identifier. The deduplication service identifies fingerprints in fingerprints group which are missing from fingerprint storage, and communicates identified fingerprints to the access object service, which communicates request for file's segments, corresponding to identified fingerprints, to file's source.

Abstract: Systems and methods for verifying files in bulk in a file system. When files are represented by a segment tree, the levels of the segment trees are walked by level such that that multiple files are verified at the same time in order to identify missing segments. Then, a bottom up scan is performed using the missing segments to identify the files corresponding to the missing segments. The missing files can then be handled by the file system.

Abstract: Source file copying is described. An access object service receives a request to copy a file to a destination and identifies a similarity group identifier for a group of fingerprints corresponding to sequential segments in the file. The access object service communicates a request for the group of fingerprints to a deduplication service associated with a range of similarity group identifiers that include the similarity group identifier. The deduplication service communicates the group of fingerprints, retrieved from fingerprint storage, to the access object service, which communicates the group of fingerprints and the similarity group identifier to the destination.

Abstract: An access object service receives a communication of a stream of fingerprints, corresponding to segments of a file, from a source of the file, and identifies sequential fingerprints in the stream of fingerprints as a group of fingerprints. The access object service identifies a similarity group identifier for the group of fingerprints. The access object service communicates the group of fingerprints to a deduplication service associated with a range of similarity group identifiers that include the similarity group identifier. The deduplication service identifies fingerprints in the group of fingerprints which are missing from fingerprint storage. The deduplication service communicates the identified fingerprints to the access object service, which communicates a request for segments of the file, corresponding to the identified fingerprints, to the source of the file. The deduplication service receives a communication of the requested segments from the source of the file and stores the requested segments.

Abstract: One example method includes scanning a storage device to obtain data and metadata concerning operation of a computing system, analyzing the data and, based on the analyzing, deriving data groups that include some of the data, and deriving data relationships among some of the data, receiving, by an expert system, a query from a user, and the query specifies a sample object for the expert system to investigate, but the query does not indicate purpose of the user in submitting the query, analyzing the query, based on the data groups and data relationships, and based on the analyzing of the query, generating, by the expert system, query results that comprise a set of user-selectable investigation directions that relate to the sample object, and presenting, by the expert system, the set of user-selectable investigation directions to the user.

Abstract: One example method includes ingesting data to a data protection system, separating, by the data protection system, the ingested data into groups according to Recovery Point Objective (RPO) such that each group is associated with a different respective RPO, storing the groups in respective storage pools, and each of the storage pools is associated with a respective one of the RPOs, and performing a respective garbage collection (GC) process at each storage pool.

Abstract: Source namespace and file copying: a namespace service receives request to copy namespace to destination, identifies namespace update for file in namespace, and communicates namespace update for file to destination. An access object service receives request to copy file to destination and identifies group identifier for fingerprints group corresponding to sequential segments in file. The access object service communicates request for fingerprints group to deduplication service associated with group identifier range including group identifier. The deduplication service communicates fingerprints group, retrieved from fingerprint storage, to access object service, which communicates fingerprints group and identifier to destination. The access object service communicates request for file segments, corresponding to fingerprints missing in destination, from destination to deduplication service.

Abstract: Object service receives communication of fingerprints stream, corresponding to file segments, from file source, and identifies sequential fingerprints in fingerprints stream as fingerprints group. Object service identifies group identifier for fingerprints group, and communicates fingerprints group to deduplication service associated with group identifier range including group identifier. Deduplication service identifies fingerprints in fingerprints group which are missing from fingerprint storage, and communicates identified fingerprints to object service, which communicates request for file segments, corresponding to identified fingerprints, to file source. Deduplication service receives communication of requested segments from file source, and stores requested segments.

Abstract: Disclosed is a system for storage management comprising: receiving capacity information within a predetermined time period from a plurality of storage in a first appliance pool and a second appliance pool, each having a plurality of storage systems; tracking a first capacity limit of the first storage system in the first appliance pool based on the capacity information; performing a first capacity prediction of the first storage system based on the tacked result of the first storage system, wherein the first capacity prediction indicating when the first storage system will be out of space based on the capacity information and the tracked result; generating a first recommendation indicating when an extra capacity of the first storage system should be added based on one or more attributes including the first capacity prediction; and presenting the first recommendation with one or more options.

Abstract: Object service receives request to copy file to destination and identifies group identifier for fingerprints group corresponding to sequential segments in file. Object service communicates request for fingerprints group to deduplication service associated with group identifier range including group identifier. Deduplication service communicates fingerprints group, retrieved from fingerprint storage, to object service, which communicates fingerprints group and group identifier to destination. Object service communicates request for file segments, corresponding to fingerprints missing in destination, communicated from destination, to deduplication service, which communicates requested segments, retrieved from source storage, to object service, which communicates requested segments to destination.

Abstract: Systems and methods for performing data protection operations including garbage collection operations and copy forward operations. For deduplicated data stored in a cloud-based storage or in a cloud tier that stores containers containing dead and live segments or dead and live regions such as compression regions, the dead compression regions are deleted by copying the live compression regions into new containers and then deleting the old containers. The copy forward is based on a recipe from a data protection system and is performed using a serverless approach.

Abstract: Systems and methods for performing data protection operations including garbage collection operations and copy forward operations. For deduplicated data stored in a cloud-based storage or in a cloud tier that stores containers containing dead and live regions such as compression regions, the dead segments in the dead compression regions are deleted by copying the live compression regions into new containers and then deleting the old containers. The copy forward is based on a recipe from a data protection system and is performed using a microservices based approach.

Abstract: One example method includes receiving data, partitioning the data according to their respective similarity groups, and the similarity groups collectively define a range of similarity groups, deduplicating the data after the partitioning, packing unique data segments remaining after deduplicating into one or more compression regions, compressing the compression regions, and writing an object, that includes the compression regions, to a durable log. The deduplicating and compressing for a similarity group may be performed by a dedup-compression instances uniquely assigned to that similarity group.

Abstract: Improving performance of garbage collection (GC) processes in a deduplicated file system having a layered processing architecture that maintains a log structured file system storing data and metadata in an append-only log arranged as a monotonically increasing log data structure of a plurality of data blocks wherein a head of the log increases in chronological order and no allocated data block is overwritten. The storage layer reserves a set of data block IDs within the log specifically for the garbage collection process, and assigns data blocks from the reserved set to GC I/O processes requiring acknowledgment in a possible out-of-order manner relative to an order of data blocks in the log. It strictly imposes using in-order I/O acknowledgement for other non-GC processes using the storage layer, where these processes may be deduplication backup processes using a segment store layer at the same protocol level as the GC layer.

Abstract: Embodiments are directed to a cost-aware object selection for cloud garbage collection that deletes completely dead objects and also selects low-live objects up to a carefully selected liveness threshold value. This threshold is dynamically chosen per cloud garbage collection cycle by balancing costs including egress, input/output operations (IOPs), storage cost of cleaning partial live objects, and the storage cost incurred by leaving behind dead data by not cleaning the object. The threshold value is dynamically calculated to accommodate different cost models for different cloud providers and also caters to different costs for different storage tiers.

Abstract: A garbage collection assisted deduplication process determines whether or not data segments should be deduplicated or not based on the liveness of segment data in a region, and the number of segments subject to deduplication in the region. Ingested data is divided into a plurality of segments, and a fingerprint is calculated for each segment. An index table entry maps a fingerprint to a region and container ID, and a perfect hash vector is setup for this mapping. A percentage of live segments in the region relative to a liveness threshold is determined, as is a number of segments in the region subject to deduplication relative to a deduplication threshold. If a region is sufficiently live, deduplication is performed, but if the region is dead, deduplication is not performed. For a live region, if the number of deduplicated segments is too low, deduplication is not performed.

Abstract: A deduplicated file system includes a set of microservices including front-ends and back-ends. Assignments of files are balanced across front-ends. The files are represented by segment trees including multiple segment levels. Assignments of similarity groups are balanced across back-ends. Similarity groups are associated with segments at a lower-level of the segment trees that form the files. Front-ends are responsible for operations involving an upper-level of the trees. Back-ends are responsible for operations involving the lower-level of the trees. A mapping of file assignments to front-ends and of similarity group assignments to back-ends is stored. A request to perform a file system operation is received. The mapping is consulted to identify particular front and back-ends that should be responsible for handling and processing the request.

Abstract: Disclosed is a system for storage management comprising: receiving capacity information within a predetermined time period from a plurality of storage in a first appliance pool and a second appliance pool, each having a plurality of storage systems; tracking a first capacity limit of the first storage system in the first compliance pool based on the capacity information; performing a first capacity prediction of the first storage system based on the tacked result of the first storage system, wherein the first capacity prediction indicating when the first storage system will be out of space based on the capacity information and the tracked result; generating a first recommendation indicating when an extra capacity of the first storage system should be added based on one or more attributes including the first capacity prediction; and presenting the first recommendation with one or more options.

Abstract: Embodiments for allocating and reclaiming memory using dynamic buffer allocation for a slab memory allocator. The method keeps track of a count of a total number of worker threads and a count of a total number of quiesced threads, and determines if there is any free slab memory. If there is no free slab memory, the method triggers an out of memory event and increments the count of the total number of quiesced threads. It reclaims all objects currently allocated in an object pool, and allocates a buffer of a next smaller size than an original buffer until a sufficient amount of slab memory is freed.

Abstract: One example method includes generating a fingerprint:tag dictionary that includes a group of fingerprints and a group of tags, and the fingerprint:tag dictionary identifies, for each fingerprint, the tag or tags which include that fingerprint, computing a similarity matrix based on the fingerprint:tag dictionary, and the similarity matrix identifies, for each pair of tags in the fingerprint:tag dictionary, a relative similarity of the tags in the pair to each other, running a clustering algorithm to identify groups of similar tags in the similarity matrix, and deduplicating, based on the groups of similar tags, respective data associated with the fingerprints.