Abstract: Source file copying is described. An access object service receives a request to copy a file to a destination and identifies a similarity group identifier for a group of fingerprints corresponding to sequential segments in the file. The access object service communicates a request for the group of fingerprints to a deduplication service associated with a range of similarity group identifiers that include the similarity group identifier. The deduplication service communicates the group of fingerprints, retrieved from fingerprint storage, to the access object service, which communicates the group of fingerprints and the similarity group identifier to the destination.

Abstract: A distributed and deduplicating file system that stores backup data to object storage is described. A node, in a cluster of nodes, receives a write operation to back up a data file, and identifies backup data associated with the write operation. The node identifies deduplicated data by deduplicating the backup data. The cluster stores the deduplicated data to each of the buffers which corresponds to a node in the cluster. If the node determines that the size of information, which includes the deduplicated data, stored to each of the buffers has accumulated to become greater than or equal to an object threshold, then the node writes the information, which includes the deduplicated data, stored in each of the buffers to an object storage which stores data for the nodes in the cluster.

Abstract: A distributed and deduplicating file system that stores backup metadata to object storage is described. A node, in a cluster of nodes, receives backup data associated with a write operation to back up a data file, and identifies metadata associated with deduplicated data by deduplicating the backup data. If the access frequency associated with the metadata is greater than or equal to an access threshold, and the size of the metadata is less than or equal to a storage threshold, the cluster writes the metadata to each of the memory devices corresponding to the nodes in the cluster. If the access frequency associated with the metadata is less than the access threshold, or the size of the metadata is more than the storage threshold, the node writes information which includes the metadata to an object storage that stores data objects for the nodes in the cluster.

Abstract: Mapping information identifies ranges of files, a set of front-end microservices, and assignments of the ranges to the front-end microservices. Each front-end microservice is thereby responsible for a range of files. The files are represented by segment trees and the front-end microservices handle operations involving an upper-level of the segment trees. A file system request on a file is directed to a particular front-end microservice that is responsible for handling a particular range of files within which the file falls according to the mapping information. An indication is received from a container orchestration service that a number of front-end microservices has changed. The mapping information is updated based on the change in the number of front-end microservices.

Abstract: One example method includes ingesting data to a data protection system, separating, by the data protection system, the ingested data into groups according to Recovery Point Objective (RPO) such that each group is associated with a different respective RPO, storing the groups in respective storage pools, and each of the storage pools is associated with a respective one of the RPOs, and performing a respective garbage collection (GC) process at each storage pool.

Abstract: A deduplicated file system includes a set of microservices including front-ends and back-ends. Assignments of files are balanced across front-ends. The files are represented by segment trees including multiple segment levels. Assignments of similarity groups are balanced across back-ends. Similarity groups are associated with segments at a lower-level of the segment trees that form the files. Front-ends are responsible for operations involving an upper-level of the trees. Back-ends are responsible for operations involving the lower-level of the trees. A mapping of file assignments to front-ends and of similarity group assignments to back-ends is stored. A request to perform a file system operation is received. The mapping is consulted to identify particular front and back-ends that should be responsible for handling and processing the request.

Abstract: Embodiments are described for a multi-domain and multi-tier architecture for clustered network file systems. This system allows a user to create sub-cluster of physical nodes, called domains, and file system resources for the data placed in a domain are allocated only from the nodes in the domain. It limits the impact of system failures to the files within a domain. A file system redirection service manages a global namespace spanning the domains and redirects file accesses to the appropriate domain where it is stored. In each domain, there are different classes of storage, tiers, with different cost and performance characteristics. Files can be placed on a set of tiers depending on a storage level agreement (SLA) specified for a file. Tier examples include a higher performance tier consisting of SSDs and a lower performance tier of HDDs.

Abstract: Metadata of a file system is maintained in a key value store. The metadata includes a namespace represented as a B+ tree having pages written to the key value store. Identifiers to the pages include a snapshot identifier and a page number. The identifiers are stored as keys in the key value store. The snapshot identifiers are used to track pages that are shared between first and second snapshots. Upon a request to write to an existing page, a determination is made from the snapshot identifiers as to whether the existing page belongs to the first or second snapshot. If the existing page belongs to the second snapshot, the write to the existing page is allowed. If the existing page belongs to the first snapshot, a copy on write (COW) is performed to generate a new page for the write.

Abstract: A fencing service receives a cluster message based on a first service's removal from membership in a first node in a cluster, and generates an access key based on a generation identifier in the cluster message. The fencing service sends the access key to a shared storage, thereby enabling the shared storage to create signing keys based on the access key, and which are used to validate signed requests to access resources stored by the shared storage. A second service in a second node in the cluster receives the cluster message, and generates the access key based on the generation identifier. The second service accesses a resource previously accessed by the first service, by generating a signing key based on the access key and a request to access the resource stored by the shared storage, and then sending the request signed by the signing key to the shared storage.

Abstract: A cluster service receives a cluster message based on a removal of a first service from membership in a first node in a cluster, and requests a shared storage to generate and then provide a new access key, thereby enabling the shared storage to use the new access key to validate requests to access resources stored by the shared storage. The cluster service receives the new access key from the shared storage, and sends the new access key in at least some cluster messages to at least a second service in a second node in the cluster of nodes. The second service creates a key based on the new access key. The second service uses the key to create a request to access a resource stored by the shared storage, sends the request to the shared storage, and then accesses the resource, which was previously accessed by the first service.

Abstract: A system receives a stream of fingerprints, which correspond to segments of a data file, from a source of the data file, and identifies fingerprints in the stream of fingerprints as fingerprints formatted by a first fingerprint format or a second fingerprint format. If a fingerprint, in the stream of fingerprints, does not duplicate any fingerprint in a fingerprint cache, the system stores the non-duplicate fingerprint in a first storage tier corresponding to the first fingerprint format or a second storage tier corresponding to the second fingerprint format. The system requests a segment of the data file, which corresponds to the non-duplicate fingerprint, from the source of the data file. The system receives the requested segment of the data file from the source of the data file, and stores the received segment of the data file in destination storage.

Abstract: If a client is configured to create fingerprints for data file segments, the configured client identifies any format specified for fingerprints created by the configured client, and identifies fingerprint formats, including a first fingerprint format and/or a second fingerprint format, supported by a backup server associated with the configured client. If the fingerprint formats supported by the backup server includes any format specified for fingerprints created by the configured client, the configured client partitions the backup server to store fingerprints created by the configured client. The configured client creates fingerprints for segments of a data file identified for backup, and sends the created fingerprints to the configured backup server. The configured client identifies a request from the configured backup server for a segment, of the data file, which corresponds to a non-duplicate fingerprint; and sends the requested segment to the configured backup server, which stores the requested segment.

Abstract: Improving performance of garbage collection (GC) processes in a deduplicated file system having a layered processing architecture that maintains a log structured file system storing data and metadata in an append-only log arranged as a monotonically increasing log data structure of a plurality of data blocks wherein a head of the log increases in chronological order and no allocated data block is overwritten. The storage layer reserves a set of data block IDs within the log specifically for the garbage collection process, and assigns data blocks from the reserved set to GC I/O processes requiring acknowledgment in a possible out-of-order manner relative to an order of data blocks in the log. It strictly imposes using in-order I/O acknowledgement for other non-GC processes using the storage layer, where these processes may be deduplication backup processes using a segment store layer at the same protocol level as the GC layer.

Abstract: Destination namespace and file copying: a namespace service receives communication of namespace update for file from file's source. and communicates namespace update for file to an access object service identified for file. The access object service receives communication of fingerprints stream, corresponding to file's segments, from file's source, and identifies sequential fingerprints in fingerprints stream as fingerprints group. The access object service identifies group identifier for fingerprints group, and communicates fingerprints group to a deduplication service associated with group identifier range including group identifier. The deduplication service identifies fingerprints in fingerprints group which are missing from fingerprint storage, and communicates identified fingerprints to the access object service, which communicates request for file's segments, corresponding to identified fingerprints, to file's source.

Abstract: Embodiments are directed to a cost-aware object selection for cloud garbage collection that deletes completely dead objects and also selects low-live objects up to a carefully selected liveness threshold value. This threshold is dynamically chosen per cloud garbage collection cycle by balancing costs including egress, input/output operations (IOPs), storage cost of cleaning partial live objects, and the storage cost incurred by leaving behind dead data by not cleaning the object. The threshold value is dynamically calculated to accommodate different cost models for different cloud providers and also caters to different costs for different storage tiers.

Abstract: Improving performance of garbage collection (GC) processes in a deduplicated file system having a layered processing architecture that maintains a log structured file system storing data and metadata in an append-only log arranged as a monotonically increasing log data structure of a plurality of data blocks wherein a head of the log increases in chronological order and no allocated data block is overwritten. The storage layer reserves a set of data block IDs within the log specifically for the garbage collection process, and assigns data blocks from the reserved set to GC I/O processes requiring acknowledgment in a possible out-of-order manner relative to an order of data blocks in the log. It strictly imposes using in-order I/O acknowledgement for other non-GC processes using the storage layer, where these processes may be deduplication backup processes using a segment store layer at the same protocol level as the GC layer.

Abstract: An intelligent method of scheduling garbage collection (GC) in a storage system. A GC scheduler obtains capacity utilization and ingest rate of the storage system and calculate therefrom a predicted capacity utilization. When the predicted capacity utilization reaches a threshold, the GC scheduler invokes GC, but otherwise skips GC until such time as predicted capacity utilization reaches the threshold. The ingest rage may be calculated by performing linear fit on past data ingest. The GC scheduler may calculate predicted capacity utilization periodically according to preset period. The GC scheduler may calculate the predicted capacity utilization to a future date beyond the next period. The future date may be at least as far as the next period plus total ingest time.

Abstract: A method, apparatus, and system for redistributing files in a multi-node storage system to improve global deduplication storage savings is disclosed. A plurality of file cluster candidates are generated for a plurality of files stored at a multi-node storage system comprising a plurality of data nodes. A similarity index is determined for each of the plurality of file cluster candidates based on similarity of the files comprised in the file cluster candidate. A ranked recipe list comprising a plurality of recipes is generated. Each recipe is associated with one of the plurality of file cluster candidates, comprises a destination data node for the associated file cluster candidate, and is associated with a deduplication space savings. At least some of the plurality of files are moved between the plurality of data nodes based on the recipes in the ranked recipe list to improve deduplication space savings in the multi-node storage system.

Abstract: System generates data structure based on unique identifiers of objects in storages and sets indicators in positions corresponding to hashes of unique identifiers of objects. The system copies active objects from one storage to another, if number of active objects in storage does not satisfy threshold, and resets indicators in positions in data structure corresponding to hashes of unique identifiers of active objects copied to the other storage. The system generates another data structure based on unique identifiers created while generating data structure, positions in other data structure corresponding to hashes of the unique identifiers. System sets indicators in positions in the other data structure corresponding to hashes of unique identifiers of data objects in active storages while generating data structure. System resets indicators in positions in data structure corresponding to hashes of the unique identifiers corresponding to indicators set in positions of the other data structure.

Abstract: Destination namespace and file copying: a namespace service receives communication of namespace update for file from file's source. and communicates namespace update for file to an access object service identified for file. The access object service receives communication of fingerprints stream, corresponding to file's segments, from file's source, and identifies sequential fingerprints in fingerprints stream as fingerprints group. The access object service identifies group identifier for fingerprints group, and communicates fingerprints group to a deduplication service associated with group identifier range including group identifier. The deduplication service identifies fingerprints in fingerprints group which are missing from fingerprint storage, and communicates identified fingerprints to the access object service, which communicates request for file's segments, corresponding to identified fingerprints, to file's source.