Abstract: Source file copying is described. An access object service receives a request to copy a file to a destination and identifies a similarity group identifier for a group of fingerprints corresponding to sequential segments in the file. The access object service communicates a request for the group of fingerprints to a deduplication service associated with a range of similarity group identifiers that include the similarity group identifier. The deduplication service communicates the group of fingerprints, retrieved from fingerprint storage, to the access object service, which communicates the group of fingerprints and the similarity group identifier to the destination.

Abstract: A distributed and deduplicating file system that stores backup data to object storage is described. A node, in a cluster of nodes, receives a write operation to back up a data file, and identifies backup data associated with the write operation. The node identifies deduplicated data by deduplicating the backup data. The cluster stores the deduplicated data to each of the buffers which corresponds to a node in the cluster. If the node determines that the size of information, which includes the deduplicated data, stored to each of the buffers has accumulated to become greater than or equal to an object threshold, then the node writes the information, which includes the deduplicated data, stored in each of the buffers to an object storage which stores data for the nodes in the cluster.

Abstract: A distributed and deduplicating file system that stores backup metadata to object storage is described. A node, in a cluster of nodes, receives backup data associated with a write operation to back up a data file, and identifies metadata associated with deduplicated data by deduplicating the backup data. If the access frequency associated with the metadata is greater than or equal to an access threshold, and the size of the metadata is less than or equal to a storage threshold, the cluster writes the metadata to each of the memory devices corresponding to the nodes in the cluster. If the access frequency associated with the metadata is less than the access threshold, or the size of the metadata is more than the storage threshold, the node writes information which includes the metadata to an object storage that stores data objects for the nodes in the cluster.

Abstract: Mapping information identifies ranges of files, a set of front-end microservices, and assignments of the ranges to the front-end microservices. Each front-end microservice is thereby responsible for a range of files. The files are represented by segment trees and the front-end microservices handle operations involving an upper-level of the segment trees. A file system request on a file is directed to a particular front-end microservice that is responsible for handling a particular range of files within which the file falls according to the mapping information. An indication is received from a container orchestration service that a number of front-end microservices has changed. The mapping information is updated based on the change in the number of front-end microservices.

Abstract: A deduplicated file system includes a set of microservices including front-ends and back-ends. Assignments of files are balanced across front-ends. The files are represented by segment trees including multiple segment levels. Assignments of similarity groups are balanced across back-ends. Similarity groups are associated with segments at a lower-level of the segment trees that form the files. Front-ends are responsible for operations involving an upper-level of the trees. Back-ends are responsible for operations involving the lower-level of the trees. A mapping of file assignments to front-ends and of similarity group assignments to back-ends is stored. A request to perform a file system operation is received. The mapping is consulted to identify particular front and back-ends that should be responsible for handling and processing the request.

Abstract: Embodiments are described for a multi-domain and multi-tier architecture for clustered network file systems. This system allows a user to create sub-cluster of physical nodes, called domains, and file system resources for the data placed in a domain are allocated only from the nodes in the domain. It limits the impact of system failures to the files within a domain. A file system redirection service manages a global namespace spanning the domains and redirects file accesses to the appropriate domain where it is stored. In each domain, there are different classes of storage, tiers, with different cost and performance characteristics. Files can be placed on a set of tiers depending on a storage level agreement (SLA) specified for a file. Tier examples include a higher performance tier consisting of SSDs and a lower performance tier of HDDs.

Abstract: Metadata of a file system is maintained in a key value store. The metadata includes a namespace represented as a B+ tree having pages written to the key value store. Identifiers to the pages include a snapshot identifier and a page number. The identifiers are stored as keys in the key value store. The snapshot identifiers are used to track pages that are shared between first and second snapshots. Upon a request to write to an existing page, a determination is made from the snapshot identifiers as to whether the existing page belongs to the first or second snapshot. If the existing page belongs to the second snapshot, the write to the existing page is allowed. If the existing page belongs to the first snapshot, a copy on write (COW) is performed to generate a new page for the write.

Abstract: A deduplicated storage system storing objects receives a search term. Storage includes metadata and segments into which the objects have been split and deduplicated. Metadata includes fingerprint sequences according to which the segments should be assembled. A partial match is found when a prefix of the term is found at an end of a segment or a suffix is found at a beginning of the segment. The partial matches are recorded and merged to form full matches. Each full match includes a first fingerprint of a first segment in which the prefix was found and a second fingerprint of a second segment in which the suffix was found. A check is made of whether a fingerprint and a next fingerprint from a first sequence of fingerprints associated with a first object have been recorded in the database. If so, the first object is reported as having the search term.

Abstract: A system receives a stream of fingerprints, which correspond to segments of a data file, from a source of the data file, and identifies fingerprints in the stream of fingerprints as fingerprints formatted by a first fingerprint format or a second fingerprint format. If a fingerprint, in the stream of fingerprints, does not duplicate any fingerprint in a fingerprint cache, the system stores the non-duplicate fingerprint in a first storage tier corresponding to the first fingerprint format or a second storage tier corresponding to the second fingerprint format. The system requests a segment of the data file, which corresponds to the non-duplicate fingerprint, from the source of the data file. The system receives the requested segment of the data file from the source of the data file, and stores the received segment of the data file in destination storage.

Abstract: If a client is configured to create fingerprints for data file segments, the configured client identifies any format specified for fingerprints created by the configured client, and identifies fingerprint formats, including a first fingerprint format and/or a second fingerprint format, supported by a backup server associated with the configured client. If the fingerprint formats supported by the backup server includes any format specified for fingerprints created by the configured client, the configured client partitions the backup server to store fingerprints created by the configured client. The configured client creates fingerprints for segments of a data file identified for backup, and sends the created fingerprints to the configured backup server. The configured client identifies a request from the configured backup server for a segment, of the data file, which corresponds to a non-duplicate fingerprint; and sends the requested segment to the configured backup server, which stores the requested segment.

Abstract: A deduplicated storage system storing objects receives a search term. Storage includes metadata and segments into which the objects have been split and deduplicated. The metadata includes fingerprint sequences according to which the segments should be assembled. A partial match is found when a prefix of the term is found at an end of a segment or a suffix is found at a beginning of the segment. A fingerprint of the segment having the partial match is recorded. A first sequence of fingerprints associated with a first object is read to check whether any fingerprints in the first sequence have been recorded. When a fingerprint in the sequence has been recorded, a check of a next fingerprint in the sequence is made to see if it has been recorded as having the partial match. If the next fingerprint has been recorded, the first object is reported as having the term.

Abstract: Destination namespace and file copying: a namespace service receives communication of namespace update for file from file's source. and communicates namespace update for file to an access object service identified for file. The access object service receives communication of fingerprints stream, corresponding to file's segments, from file's source, and identifies sequential fingerprints in fingerprints stream as fingerprints group. The access object service identifies group identifier for fingerprints group, and communicates fingerprints group to a deduplication service associated with group identifier range including group identifier. The deduplication service identifies fingerprints in fingerprints group which are missing from fingerprint storage, and communicates identified fingerprints to the access object service, which communicates request for file's segments, corresponding to identified fingerprints, to file's source.

Abstract: Described is a system for detecting corruption in a deduplicated object storage system accessible by one or more microservices while minimizing costly read operations on objects. A similarity group verification path is selected by a controller module based upon detection of an object storage memory size condition. The similarity group verification path includes controller phases to verify whether objects have been corrupted without having to incur costly read operations.

Abstract: Destination namespace and file copying: a namespace service receives communication of namespace update for file from file's source. and communicates namespace update for file to an access object service identified for file. The access object service receives communication of fingerprints stream, corresponding to file's segments, from file's source, and identifies sequential fingerprints in fingerprints stream as fingerprints group. The access object service identifies group identifier for fingerprints group, and communicates fingerprints group to a deduplication service associated with group identifier range including group identifier. The deduplication service identifies fingerprints in fingerprints group which are missing from fingerprint storage, and communicates identified fingerprints to the access object service, which communicates request for file's segments, corresponding to identified fingerprints, to file's source.

Abstract: One example method includes receiving a write request that includes a data structure version to be written, wherein the data structure version is associated with a unique identifier, storing the data structure version in association with the unique identifier, receiving a read request for a most recent version of the data structure and, when the stored data structure version is not the most recent version of the data structure, examining respective unique identifiers of each of a group of other stored data structure versions to determine which stored data structure version is the most recent. Finally, the example method includes returning the most recent data structure version, notwithstanding that one or more other data structure versions existed at the time that the read request was received.

Abstract: Source file copying is described. An access object service receives a request to copy a file to a destination and identifies a similarity group identifier for a group of fingerprints corresponding to sequential segments in the file. The access object service communicates a request for the group of fingerprints to a deduplication service associated with a range of similarity group identifiers that include the similarity group identifier. The deduplication service communicates the group of fingerprints, retrieved from fingerprint storage, to the access object service, which communicates the group of fingerprints and the similarity group identifier to the destination.

Abstract: An access object service receives a communication of a stream of fingerprints, corresponding to segments of a file, from a source of the file, and identifies sequential fingerprints in the stream of fingerprints as a group of fingerprints. The access object service identifies a similarity group identifier for the group of fingerprints. The access object service communicates the group of fingerprints to a deduplication service associated with a range of similarity group identifiers that include the similarity group identifier. The deduplication service identifies fingerprints in the group of fingerprints which are missing from fingerprint storage. The deduplication service communicates the identified fingerprints to the access object service, which communicates a request for segments of the file, corresponding to the identified fingerprints, to the source of the file. The deduplication service receives a communication of the requested segments from the source of the file and stores the requested segments.

Abstract: Source namespace and file copying: a namespace service receives request to copy namespace to destination, identifies namespace update for file in namespace, and communicates namespace update for file to destination. An access object service receives request to copy file to destination and identifies group identifier for fingerprints group corresponding to sequential segments in file. The access object service communicates request for fingerprints group to deduplication service associated with group identifier range including group identifier. The deduplication service communicates fingerprints group, retrieved from fingerprint storage, to access object service, which communicates fingerprints group and identifier to destination. The access object service communicates request for file segments, corresponding to fingerprints missing in destination, from destination to deduplication service.

Abstract: Object service receives communication of fingerprints stream, corresponding to file segments, from file source, and identifies sequential fingerprints in fingerprints stream as fingerprints group. Object service identifies group identifier for fingerprints group, and communicates fingerprints group to deduplication service associated with group identifier range including group identifier. Deduplication service identifies fingerprints in fingerprints group which are missing from fingerprint storage, and communicates identified fingerprints to object service, which communicates request for file segments, corresponding to identified fingerprints, to file source. Deduplication service receives communication of requested segments from file source, and stores requested segments.

Abstract: Disclosed is a system for storage management comprising: receiving capacity information within a predetermined time period from a plurality of storage in a first appliance pool and a second appliance pool, each having a plurality of storage systems; tracking a first capacity limit of the first storage system in the first appliance pool based on the capacity information; performing a first capacity prediction of the first storage system based on the tacked result of the first storage system, wherein the first capacity prediction indicating when the first storage system will be out of space based on the capacity information and the tracked result; generating a first recommendation indicating when an extra capacity of the first storage system should be added based on one or more attributes including the first capacity prediction; and presenting the first recommendation with one or more options.