Abstract: Embodiments of small file restore process in deduplication file system wherein restoration requires issuing a read request within an I/O request to the file system. A prefetch queue processor creates an intent to read the file, rather than opening the file upon receiving the request. During this step, the application passes the file handle of the file, and the destination handle where the data must be read into. As the queue is processed, the prefetch for the request (handle/offset/length) is passed to the file server. The filesystem processes the request to equivalently ‘open’ the file, and bring the data into memory. As the read request for the same file reaches the filesystem the file data is read from memory for writing to the destination handle.

Abstract: Embodiments of file restores in a Data Domain (DD) file system implementing a DD Bandwidth Optimized Open Storage Technology (DDBoost) library that translates application read and write request to DDBoost application program interfaces (APIs). A prefetch queue processor creates an intent to read the file. The application passes the file handle of the file, and the destination handle where the data must be read into. As the queue is processed, the prefetch for the request (handle/offset/length) is passed to the file server. The filesystem processes the request to open the file to load into memory. As the read request for the same file reaches the filesystem the file data is read from memory for writing to the destination handle. An extended DDBoost API expression is defined to pass the current path and destination path to the application.

Abstract: Embodiments for reducing panic shutdown of components in a pipelined data processing system. Components are monitored for health, processing progress, and dependencies during normal system operation. A directed graph is generated showing non-circular dependencies of components in the pipeline. Deadlock of a particular component may or may not signal a panic condition depending on whether any of its presently downstream and depended on components are operating properly. The continuously monitored knowledge of proper operation of all downstream components is thus used to intelligently apply or defer panic alerts to keep the system operating uninterrupted from panic conditions that might soon or eventually be fixed by continued operation of the system pipeline.

Abstract: A multi-stream restore method for reducing restore time of file data to service an external read request for file data stored by a data deduplication system employs a per-file read-ahead cache that is populated by multiple parallel internal read-ahead streams. The number of parallel read-ahead streams is dynamically variable based upon processing conditions and a set of heuristically pre-determined criteria that characterize the file data including a data access pattern that is sequential and a processing load that is below a predetermined threshold. Processing conditions are periodically monitored and adjustments to the multi-stream data access are made to optimize performance.

Abstract: Embodiments of a system and method to track the locality of a file being restored at the time of prefetching; and a mechanism to dynamically adjust the prefetching parallelism, per read batch, optimally based on the locality and other heuristics, such as system load. A process tracks locality of data elements in a batched data stream, as corresponds to a number of different container IDs accessed by the data elements. The prefetch nominally works serially on the data elements, however, if the locality exceeds a threshold separating acceptable versus non-acceptable distribution of data accesses, each batch is divided into a number of smaller sub-batches that are then pre-fetched in parallel with one another.

Abstract: Embodiments for a prefetch mechanism that stores data in the memory allocated in the remote passive (remote) node in a High Availability active-passive node pair. A read-ahead cache that can host different parts of a file is created on both the active and the passive nodes. The system improves the performance of sequential restores by using memory available on a passive node to supplement the local (active node) memory to hold the prefetched data for the common case of large sequential restores. This extra memory overcomes issues when a current prefetch fails to keep up with the target restore performance. Filesystem redirection is thus accomplished by servicing an I/O request by sending it another node that itself caches the data.

Abstract: Embodiments of small file restore process in deduplication file system wherein restoration requires issuing a read request within an I/O request to the file system. The process places the small files in a prefetch queue such that a combined size of the small files meets or exceeds a size of the prefetch queue as defined by a prefetch horizon. A queue processor issues a read request for the first file in the queue, scans the prefetch queue to find a read request for a file at the prefetch horizon, and prefetches the file at the prefetch horizon. The prefetch queue essentially constitutes a hint from the client that a read I/O is imminent for purposes of filling the read-ahead cache and preventing a need to issue a blocking I/O operation.

Abstract: Embodiments for reducing panic shutdown of components in a pipelined data processing system. Components are monitored for health, processing progress, and dependencies during normal system operation. A directed graph is generated showing non-circular dependencies of components in the pipeline. Deadlock of a particular component may or may not signal a panic condition depending on whether any of its presently downstream and depended on components are operating properly. The continuously monitored knowledge of proper operation of all downstream components is thus used to intelligently apply or defer panic alerts to keep the system operating uninterrupted from panic conditions that might soon or eventually be fixed by continued operation of the system pipeline.

Abstract: Replication of a filesystem or a mount point or share may replicate all data that it consists of irrespective of where the data is stored. Replication protects data irrespective of location. One method is to replicate the filesystem namespace as is while skipping the data outside of the appliance/machine so that replication cost and time are reasonable. The data outside of the machine, like cloud/tape data is protected differently. One example method includes a data protection operation configured to replication a namespace associated with multiple data tiers. During replication, data from one of the tiers is skipped while all of the namespace metadata is replicated. The recovery restores the namespace metadata and the data that was replicated from the other tier. This may be performed in connection with cyber security, for example when replicating multi-tier data to a vault.

Abstract: Described is a system for leveraging incremental backup metadata of a deduplication storage system to perform an efficient data restore. The system may reference metadata stored as part of incremental backup infrastructure to determine only the changes to the data that occurred subsequent to the restore point. As a result, the system may reduce data read processing by retrieving only the changed data from the deduplication storage system. Moreover, the system may reduce the required bandwidth for the restore by providing only the changed data to a target device of the restore. Data retained by the client system such as the current version of the data may then be overwritten with the reduced set of data to complete the restore. As a result, the system may perform an efficient restore by reducing read accesses and bandwidth requirements, and thus, improve overall restore performance.

Abstract: A request is received to restore a file at a deduplicated storage system to a client. The file resides at the storage system as a synthetic file based on a base file at the storage system. The request includes an indication that the base file is also present at the client. Metadata generated during a backup of the file to the storage system is reviewed. The metadata includes references to data determined to be in the base file at the storage system, and references to other data determined to not be in the base file at the storage system. The other data determined to not be in the base file is read from the storage system and transmitted to the client. Upon receipt, the client assembles the requested file using the base file present at the client and the other data determined to not be in the base file.

Abstract: An index associates fingerprints of file segments to container numbers of containers within which the file segments are stored. At a start of migration, a boundary is created identifying a current container number. At least a subset of file segments at a source storage tier are packed into a new container to be written to a destination storage tier. A new container number is generated for the new container. The index is updated to associate fingerprints of the at least subset of file segments to the new container number. A request is received to read a file segment. The index is queried with a fingerprint of the file segment to determine whether the request should be directed to the source or destination storage tier based on a container number of a container within which the file segment is stored.

Abstract: Embodiments of an instant recall process and system for long-term data stored on a cloud storage tier. Embodiments include saving a content handle of a file in a cloud storage tier as an extended attribute in a single file system namespace; moving the file from the cloud storage tier to an active storage tier for data processing; recalling the file from the active storage tier to the cloud storage tier upon completion of the data processing; using the content handle from hidden metadata for a working copy of the file; and saving a hash of a segment reference as part of the extended attribute.

Abstract: Detecting a trend in latency for storage underlying a deduplicated file system includes generating a set of data points by recording when input/output (IO) requests were issued to the storage and recording time required to receive success responses from the storage. Least squares regression is performed on the data points to find a best-fit line through the data points. A slope of the best-fit line is calculated. A determination is made as to whether the slope is positive, a positive slope thereby indicating a trend of increasing latency of the storage. When the slope is determined to be positive, clients accessing the deduplicated file system are throttled.

Abstract: Replication of a filesystem or a mount point or share may replicate all data that it consists of irrespective of where the data is stored. Replication protects data irrespective of location. One method is to replicate the filesystem namespace as is while skipping the data outside of the appliance/machine so that replication cost and time are reasonable. The data outside of the machine, like cloud/tape data is protected differently. One example method includes a data protection operation configured to replication a namespace associated with multiple data tiers. During replication, data from one of the tiers is skipped while all of the namespace metadata is replicated. The recovery restores the namespace metadata and the data that was replicated from the other tier. This may be performed in connection with cyber security, for example when replicating multi-tier data to a vault.

Abstract: A method and system for storage load balancing based on virtual synthetics metadata. When storing data onto a storage cluster, data submitted thereto may often be distributed unevenly across the constituent storage nodes thereof. To address the issue, some form of load balancing (or re-distribution of data) across the storage nodes may be implemented. Existing load balancing techniques, however, tend to migrate data between storage nodes without consideration for the efficient utilization of available storage space on the storage node where the data ends up (or destination storage node). Accordingly, the disclosed method and system propose a load balancing mechanism whereby the migrated data dedupes well, thereby securing the efficient consumption of storage space on the destination storage node.

Abstract: A method and system for storage load balancing based on virtual synthetics metadata. When storing data onto a storage cluster, data submitted thereto may often be distributed unevenly across the constituent storage nodes thereof. To address the issue, some form of load balancing (or re-distribution of data) across the storage nodes may be implemented. Existing load balancing techniques, however, tend to migrate data between storage nodes without consideration for the efficient utilization of available storage space on the storage node where the data ends up (or destination storage node). Accordingly, the disclosed method and system propose a load balancing mechanism whereby the migrated data dedupes well, thereby securing the efficient consumption of storage space on the destination storage node.

Abstract: Replication of a filesystem or a mount point or share may replicate all data that it consists of irrespective of where the data is stored. Replication protects data irrespective of location. One method is to replicate the filesystem namespace as is while skipping the data outside of the appliance/machine so that replication cost and time are reasonable. The data outside of the machine, like cloud/tape data is protected differently. One example method includes a data protection operation configured to replication a namespace associated with multiple data tiers. During replication, data from one of the tiers is skipped while all of the namespace metadata is replicated. The recovery restores the namespace metadata and the data that was replicated from the other tier. This may be performed in connection with cyber security, for example when replicating multi-tier data to a vault.

Abstract: A multi-stream restore method for reducing restore time of file data to service an external read request for file data stored by a data deduplication system employs a per-file read-ahead cache that is populated by multiple parallel internal read-ahead streams. The number of parallel read-ahead streams is dynamically variable based upon processing conditions and a set of heuristically pre-determined criteria that characterize the file data including a data access pattern that is sequential and a processing load that is below a predetermined threshold. Processing conditions are periodically monitored and adjustments to the multi-stream data access are made to optimize performance.

Abstract: An index associates fingerprints of file segments to container numbers of containers within which the file segments are stored. At a start of migration, a boundary is created identifying a current container number. At least a subset of file segments at a source storage tier are packed into a new container to be written to a destination storage tier. A new container number is generated for the new container. The index is updated to associate fingerprints of the at least subset of file segments to the new container number. A request is received to read a file segment. The index is queried with a fingerprint of the file segment to determine whether the request should be directed to the source or destination storage tier based on a container number of a container within which the file segment is stored.