Abstract: Embodiments of an instant recall process and system for long-term data stored on a cloud storage tier. Embodiments include saving a content handle of a file in a cloud storage tier as an extended attribute in a single file system namespace; moving the file from the cloud storage tier to an active storage tier for data processing; recalling the file from the active storage tier to the cloud storage tier upon completion of the data processing; using the content handle from hidden metadata for a working copy of the file; and saving a hash of a segment reference as part of the extended attribute.

Abstract: Detecting a trend in latency for storage underlying a deduplicated file system includes generating a set of data points by recording when input/output (IO) requests were issued to the storage and recording time required to receive success responses from the storage. Least squares regression is performed on the data points to find a best-fit line through the data points. A slope of the best-fit line is calculated. A determination is made as to whether the slope is positive, a positive slope thereby indicating a trend of increasing latency of the storage. When the slope is determined to be positive, clients accessing the deduplicated file system are throttled.

Abstract: Replication of a filesystem or a mount point or share may replicate all data that it consists of irrespective of where the data is stored. Replication protects data irrespective of location. One method is to replicate the filesystem namespace as is while skipping the data outside of the appliance/machine so that replication cost and time are reasonable. The data outside of the machine, like cloud/tape data is protected differently. One example method includes a data protection operation configured to replication a namespace associated with multiple data tiers. During replication, data from one of the tiers is skipped while all of the namespace metadata is replicated. The recovery restores the namespace metadata and the data that was replicated from the other tier. This may be performed in connection with cyber security, for example when replicating multi-tier data to a vault.

Abstract: A method and system for storage load balancing based on virtual synthetics metadata. When storing data onto a storage cluster, data submitted thereto may often be distributed unevenly across the constituent storage nodes thereof. To address the issue, some form of load balancing (or re-distribution of data) across the storage nodes may be implemented. Existing load balancing techniques, however, tend to migrate data between storage nodes without consideration for the efficient utilization of available storage space on the storage node where the data ends up (or destination storage node). Accordingly, the disclosed method and system propose a load balancing mechanism whereby the migrated data dedupes well, thereby securing the efficient consumption of storage space on the destination storage node.

Abstract: A method and system for storage load balancing based on virtual synthetics metadata. When storing data onto a storage cluster, data submitted thereto may often be distributed unevenly across the constituent storage nodes thereof. To address the issue, some form of load balancing (or re-distribution of data) across the storage nodes may be implemented. Existing load balancing techniques, however, tend to migrate data between storage nodes without consideration for the efficient utilization of available storage space on the storage node where the data ends up (or destination storage node). Accordingly, the disclosed method and system propose a load balancing mechanism whereby the migrated data dedupes well, thereby securing the efficient consumption of storage space on the destination storage node.

Abstract: Replication of a filesystem or a mount point or share may replicate all data that it consists of irrespective of where the data is stored. Replication protects data irrespective of location. One method is to replicate the filesystem namespace as is while skipping the data outside of the appliance/machine so that replication cost and time are reasonable. The data outside of the machine, like cloud/tape data is protected differently. One example method includes a data protection operation configured to replication a namespace associated with multiple data tiers. During replication, data from one of the tiers is skipped while all of the namespace metadata is replicated. The recovery restores the namespace metadata and the data that was replicated from the other tier. This may be performed in connection with cyber security, for example when replicating multi-tier data to a vault.

Abstract: A multi-stream restore method for reducing restore time of file data to service an external read request for file data stored by a data deduplication system employs a per-file read-ahead cache that is populated by multiple parallel internal read-ahead streams. The number of parallel read-ahead streams is dynamically variable based upon processing conditions and a set of heuristically pre-determined criteria that characterize the file data including a data access pattern that is sequential and a processing load that is below a predetermined threshold. Processing conditions are periodically monitored and adjustments to the multi-stream data access are made to optimize performance.

Abstract: An index associates fingerprints of file segments to container numbers of containers within which the file segments are stored. At a start of migration, a boundary is created identifying a current container number. At least a subset of file segments at a source storage tier are packed into a new container to be written to a destination storage tier. A new container number is generated for the new container. The index is updated to associate fingerprints of the at least subset of file segments to the new container number. A request is received to read a file segment. The index is queried with a fingerprint of the file segment to determine whether the request should be directed to the source or destination storage tier based on a container number of a container within which the file segment is stored.

Abstract: A multi-stream restore method for reducing restore time of file data to service an external read request for file data stored by a data deduplication system employs a per-file read-ahead cache that is populated by multiple parallel internal read-ahead streams. The number of parallel read-ahead streams is dynamically variable based upon processing conditions and a set of heuristically pre-determined criteria that characterize the file data including a data access pattern that is sequential and a processing load that is below a predetermined threshold. Processing conditions are periodically monitored and adjustments to the multi-stream data access are made to optimize performance.

Abstract: Data containers are stored at a first cloud and include file segments managed by a deduplication file system. Metadata containers are written to a log, identified by a container ID generated sequentially, and include references to data containers. An index is maintained to map segment fingerprints to container IDs. Upon starting a migration to a second cloud, a checkpoint is created identifying a container ID at a head of the log. During migration, the index is updated to map fingerprints of migrated segments to new container IDs, and referencing new data containers having the migrated segments and written to the second cloud. A request to read a file is received. The index is examined to identify a container ID associated with a data container storing segments of the file. Based on the identified container ID and the checkpoint, the data container is accessed from the first or second cloud.

Abstract: A method for managing storage devices includes, in response to a first determination that a first cloud storage device is inaccessible to a local storage device, obtaining metadata objects from a second cloud storage device, identifying, based on a sync time analysis, at least one metadata object of the metadata objects to be deleted, initiating the deletion of the at least one metadata object on a second cloud storage device, wherein the second cloud storage device is accessible to the local storage device, and updating, after initiating the deletion of the at least one metadata object, a namespace on a local storage device.

Abstract: A data management device includes a persistent storage and a processor. The persistent storage includes an object storage. The processor generates a collision free hash function based on segments stored in the object storage. The processor generates a hash vector using the collision free hash function. The processor deduplicates the segments using the hash vector. The processor stores the deduplicated segments in the object storage.

Abstract: A method for managing backups includes receiving, via a first backup stream, first data associated with a first tagged backup stripe, wherein the first tagged backup stripe is associated with a first routing tag, directing, based on the first routing tag, the first data to a first backup instance, receiving, via a second backup stream, second data associated with a second tagged backup stripe, wherein the second tagged backup stripe is associated with the first routing tag, directing, based on the first routing tag, the second data to the first backup instance, and performing, in the first backup instance, a deduplication operation on the first data and the second data.

Abstract: A data management device includes a persistent storage and a processor. The persistent storage includes an object storage that stores segments. The processor generates a collision free hash function based on the segments, generates a hash vector using the collision free hash function, deduplicates a portion of the segments associated with to-be-migrated files using the hash vector, and migrates the to-be-migrated files using the deduplicated portion of the segments to a remote storage.

Abstract: Embodiments are described for performing file restores from remote high-latency storage tiers by reading available data from a local low-latency tier in a deduplication appliance. A request to restore a previously segmented and deduplicated file can be received by a storage appliance from an application, each segment having a fingerprint. The name of the file can be looked up in an index on the storage appliance, and a first batch of fingerprints of segments of the file can be retrieved from the index. Each fingerprint can be looked up in metadata in the index to determine whether the segment corresponding to the fingerprint is available locally and therefore need not be retrieved locally. A list of local and remote prefetch segments is generated, and a prefetch request is generated for each list, if non-empty. Use of the prefetch scheme can be dynamically turned on or off.

Abstract: A multi-stream restore method for reducing restore time of file data to service an external read request for file data stored by a data deduplication system employs a per-file read-ahead cache that is populated by multiple parallel internal read-ahead streams. The number of parallel read-ahead streams is dynamically variable based upon processing conditions and a set of heuristically pre-determined criteria that characterize the file data including a data access pattern that is sequential and a processing load that is below a predetermined threshold. Processing conditions are periodically monitored and adjustments to the multi-stream data access are made to optimize performance.

Abstract: Embodiments are described for performing a storage tier-specific file system operation in a file system that comprises a namespace that spans multiple tiers of storage. A file system command can be a tier-specific deletion of files in a multi-tier file system, or a move data operation from a first tier of storage to a second tier of storage, affecting both the first and second tier of storage. The file system command can generate a copy of the namespace for the first tier of storage and a snapshot of the data of the first tier of storage referenced in the file system operation. Functionality may be performed using one or more storage appliances, each of which can have its open processing system, memory and storage device(s). Storage tiers can include active tier, cloud tier, and remote or archive storage tier.

Abstract: A method for managing backups includes receiving, via a first backup stream, first data associated with a first tagged backup stripe, wherein the first tagged backup stripe is associated with a first routing tag, directing, based on the first routing tag, the first data to a first backup instance, receiving, via a second backup stream, second data associated with a second tagged backup stripe, wherein the second tagged backup stripe is associated with the first routing tag, directing, based on the first routing tag, the second data to the first backup instance, and performing, in the first backup instance, a deduplication operation on the first data and the second data.

Abstract: Embodiments of an instant recall process and system for long-term data stored on a cloud storage tier. Embodiments include saving a content handle of a file in a cloud storage tier as an extended attribute in a single file system namespace; moving the file from the cloud storage tier to an active storage tier for data processing; recalling the file from the active storage tier to the cloud storage tier upon completion of the data processing; using the content handle from hidden metadata for a working copy of the file; and saving a hash of a segment reference as part of the extended attribute.

Abstract: A method for managing storage devices includes, in response to a first determination that a first cloud storage device is inaccessible to a local storage device, obtaining metadata objects from a second cloud storage device, identifying, based on a sync time analysis, at least one metadata object of the metadata objects to be deleted, initiating the deletion of the at least one metadata object on a second cloud storage device, wherein the second cloud storage device is accessible to the local storage device, and updating, after initiating the deletion of the at least one metadata object, a namespace on a local storage device.