Abstract: Embodiments are described for providing backup resources in a multi-tier network, by providing a plurality of different storage devices, each with different operating characteristics and costs for storage of data assets, and organizing the storage devices into a plurality of tiers based on respective operating characteristics, such as based on performance versus cost parameters. Certain Service Level Agreements (SLAs) dictating storage requirements for each of the data assets to a backup program are provided to the backup system, and a mapper maps each SLA to one or more tiers of the plurality of tiers based on the storage requirements of a respective SLA to the operating characteristics of each tier.

Abstract: A system receives a stream of fingerprints, which correspond to segments of a data file, from a source of the data file, and identifies fingerprints in the stream of fingerprints as fingerprints formatted by a first fingerprint format or a second fingerprint format. If a fingerprint, in the stream of fingerprints, does not duplicate any fingerprint in a fingerprint cache, the system stores the non-duplicate fingerprint in a first storage tier corresponding to the first fingerprint format or a second storage tier corresponding to the second fingerprint format. The system requests a segment of the data file, which corresponds to the non-duplicate fingerprint, from the source of the data file. The system receives the requested segment of the data file from the source of the data file, and stores the received segment of the data file in destination storage.

Abstract: A source site includes a controller, a set of source worker nodes, and a message queue connected between the controller and source worker nodes. The message queue receives messages and stores the messages for retrieval. A destination site includes a set of destination worker nodes. Tasks are generated to replicate changes to objects at the source site to the destination site. The controller pushes messages corresponding to the tasks onto the message queue. A source worker node retrieves a message corresponding to a task from the message queue for processing in conjunction with a destination worker node. The message is indicated as having been retrieved from the message queue.

Abstract: Files in a file system are assigned to front-end instances. The files are represented by segment trees. Similarity groups are assigned to back-end instances. The similarity groups include segments at a lower-level of the segment trees. The front-end instances are responsible for operations involving an upper-level of the segment trees. The back-end instances are responsible for operations involving the lower-level of the segment trees. Upon a failure of an instance, an attempt is made to restart the failed instance. If the restart is successful, file and similarity group assignments are retained. If the restart is not successful, file assignments are redistributed to other front-end instances when the failed instance is a front-end instance, and similarity group assignments are redistributed to other back-end instances when the failed instance is a back-end instance.

Abstract: Embodiments are described for migrating data and load balancing in a multi-domain network having a clustered network partitioned into a number of separate domains. Each domain contains resources like processors and memory, as well as deduplication and compression services. A single global namespace for the network is presented by combining all of the namespaces in the domains. A multi-domain manager component with an analytics engine facilitates analytics-driven migration of data among nodes in different domains, and a load balancer component monitors environmental operating conditions of each node in the clustered network, detects nodes having operating conditions exceeding defined tolerances, and migrates data and processes from nodes exceeding the defined tolerances to other nodes not exceeding the defined tolerances.

Abstract: A distributed and deduplicating file system that stores backup metadata to object storage is described. A node, in a cluster of nodes, receives backup data associated with a write operation to back up a data file, and identifies metadata associated with deduplicated data by deduplicating the backup data. If the access frequency associated with the metadata is greater than or equal to an access threshold, and the size of the metadata is less than or equal to a storage threshold, the cluster writes the metadata to each of the memory devices corresponding to the nodes in the cluster. If the access frequency associated with the metadata is less than the access threshold, or the size of the metadata is more than the storage threshold, the node writes information which includes the metadata to an object storage that stores data objects for the nodes in the cluster.

Abstract: Mapping information identifies ranges of files, a set of front-end microservices, and assignments of the ranges to the front-end microservices. Each front-end microservice is thereby responsible for a range of files. The files are represented by segment trees and the front-end microservices handle operations involving an upper-level of the segment trees. A file system request on a file is directed to a particular front-end microservice that is responsible for handling a particular range of files within which the file falls according to the mapping information. An indication is received from a container orchestration service that a number of front-end microservices has changed. The mapping information is updated based on the change in the number of front-end microservices.

Abstract: Embodiments are described for a system and method facilitating deduplication in a multi-tier storage system in which a file can have different portions written to different tiers. A process partition the data space of each tier to a number of similarity groups and distributes the similarity groups across file system services in a cluster. The distribution is done in such a way that for a given similarity group ID, the same file system service owns the similarity groups of every tier. This allows for efficient checks for deduplication as it can be done local to a node rather than requiring remote procedure calls.

Abstract: Systems and methods for cleaning a storage system. A deduplicated storage system is cleaned by identifying structures that include dead or unreferenced segments. This includes processing recipes to identify the segments that are no longer part of a live object recipe. Then, the dead segments are removed. This is accomplished by copying forward the live segments and then deleting, as a whole, the structure that included the dead segments.

Abstract: Embodiments are described for migrating data and load balancing in a multi-domain network having a clustered network partitioned into a number of separate domains. Each domain contains resources like processors and memory, as well as deduplication and compression services. A single global namespace for the network is presented by combining all of the namespaces in the domains. A multi-domain manager component with an analytics engine facilitates analytics-driven migration of data among nodes in different domains, and a load balancer component monitors environmental operating conditions of each node in the clustered network, detects nodes having operating conditions exceeding defined tolerances, and migrates data and processes from nodes exceeding the defined tolerances to other nodes not exceeding the defined tolerances.

Abstract: Embodiments are described for providing backup resources in a multi-tier network, by providing a plurality of different storage devices, each with different operating characteristics and costs for storage of data assets, and organizing the storage devices into a plurality of tiers based on respective operating characteristics, such as based on performance versus cost parameters. Certain Service Level Agreements (SLAs) dictating storage requirements for each of the data assets to a backup program are provided to the backup system, and a mapper maps each SLA to one or more tiers of the plurality of tiers based on the storage requirements of a respective SLA to the operating characteristics of each tier.

Abstract: If a client is configured to create fingerprints for data file segments, the configured client identifies any format specified for fingerprints created by the configured client, and identifies fingerprint formats, including a first fingerprint format and/or a second fingerprint format, supported by a backup server associated with the configured client. If the fingerprint formats supported by the backup server includes any format specified for fingerprints created by the configured client, the configured client partitions the backup server to store fingerprints created by the configured client. The configured client creates fingerprints for segments of a data file identified for backup, and sends the created fingerprints to the configured backup server. The configured client identifies a request from the configured backup server for a segment, of the data file, which corresponds to a non-duplicate fingerprint; and sends the requested segment to the configured backup server, which stores the requested segment.

Abstract: Embodiments are described for a multi-domain and multi-tier architecture for clustered network file systems. This system allows a user to create sub-cluster of physical nodes, called domains, and file system resources for the data placed in a domain are allocated only from the nodes in the domain. It limits the impact of system failures to the files within a domain. A file system redirection service manages a global namespace spanning the domains and redirects file accesses to the appropriate domain where it is stored. In each domain, there are different classes of storage, tiers, with different cost and performance characteristics. Files can be placed on a set of tiers depending on a storage level agreement (SLA) specified for a file. Tier examples include a higher performance tier consisting of SSDs and a lower performance tier of HDDs.

Abstract: An access object service receives a communication of a stream of fingerprints, corresponding to segments of a file, from a source of the file, and identifies sequential fingerprints in the stream of fingerprints as a group of fingerprints. The access object service identifies a similarity group identifier for the group of fingerprints. The access object service communicates the group of fingerprints to a deduplication service associated with a range of similarity group identifiers that include the similarity group identifier. The deduplication service identifies fingerprints in the group of fingerprints which are missing from fingerprint storage. The deduplication service communicates the identified fingerprints to the access object service, which communicates a request for segments of the file, corresponding to the identified fingerprints, to the source of the file. The deduplication service receives a communication of the requested segments from the source of the file and stores the requested segments.

Abstract: Systems and methods for performing data protection operations including garbage collection operations and copy forward operations. For deduplicated data stored in a cloud-based storage or in a cloud tier that stores containers containing dead and live segments, the dead segments are deleted by copying live segments into new containers and then deleting the old containers. The copy forward is based on a recipe from a data protection system and is performed using a microservices that can be run as needed in the cloud.

Abstract: Source file copying is described. An access object service receives a request to copy a file to a destination and identifies a similarity group identifier for a group of fingerprints corresponding to sequential segments in the file. The access object service communicates a request for the group of fingerprints to a deduplication service associated with a range of similarity group identifiers that include the similarity group identifier. The deduplication service communicates the group of fingerprints, retrieved from fingerprint storage, to the access object service, which communicates the group of fingerprints and the similarity group identifier to the destination.

Abstract: A distributed and deduplicating file system that stores backup metadata to object storage is described. A node, in a cluster of nodes, receives backup data associated with a write operation to back up a data file, and identifies metadata associated with deduplicated data by deduplicating the backup data. If the access frequency associated with the metadata is greater than or equal to an access threshold, and the size of the metadata is less than or equal to a storage threshold, the cluster writes the metadata to each of the memory devices corresponding to the nodes in the cluster. If the access frequency associated with the metadata is less than the access threshold, or the size of the metadata is more than the storage threshold, the node writes information which includes the metadata to an object storage that stores data objects for the nodes in the cluster.

Abstract: A distributed and deduplicating file system that stores backup data to object storage is described. A node, in a cluster of nodes, receives a write operation to back up a data file, and identifies backup data associated with the write operation. The node identifies deduplicated data by deduplicating the backup data. The cluster stores the deduplicated data to each of the buffers which corresponds to a node in the cluster. If the node determines that the size of information, which includes the deduplicated data, stored to each of the buffers has accumulated to become greater than or equal to an object threshold, then the node writes the information, which includes the deduplicated data, stored in each of the buffers to an object storage which stores data for the nodes in the cluster.

Abstract: Mapping information identifies ranges of files, a set of front-end microservices, and assignments of the ranges to the front-end microservices. Each front-end microservice is thereby responsible for a range of files. The files are represented by segment trees and the front-end microservices handle operations involving an upper-level of the segment trees. A file system request on a file is directed to a particular front-end microservice that is responsible for handling a particular range of files within which the file falls according to the mapping information. An indication is received from a container orchestration service that a number of front-end microservices has changed. The mapping information is updated based on the change in the number of front-end microservices.

Abstract: A deduplicated file system includes a set of microservices including front-ends and back-ends. Assignments of files are balanced across front-ends. The files are represented by segment trees including multiple segment levels. Assignments of similarity groups are balanced across back-ends. Similarity groups are associated with segments at a lower-level of the segment trees that form the files. Front-ends are responsible for operations involving an upper-level of the trees. Back-ends are responsible for operations involving the lower-level of the trees. A mapping of file assignments to front-ends and of similarity group assignments to back-ends is stored. A request to perform a file system operation is received. The mapping is consulted to identify particular front and back-ends that should be responsible for handling and processing the request.