Abstract: Detecting a trend in latency for storage underlying a deduplicated file system includes generating a set of data points by recording when input/output (IO) requests were issued to the storage and recording time required to receive success responses from the storage. Least squares regression is performed on the data points to find a best-fit line through the data points. A slope of the best-fit line is calculated. A determination is made as to whether the slope is positive, a positive slope thereby indicating a trend of increasing latency of the storage. When the slope is determined to be positive, clients accessing the deduplicated file system are throttled.

Abstract: Embodiments for reducing panic shutdown of components in a pipelined data processing system. Components are monitored for health, processing progress, and dependencies during normal system operation. A directed graph is generated showing non-circular dependencies of components in the pipeline. Deadlock of a particular component may or may not signal a panic condition depending on whether any of its presently downstream and depended on components are operating properly. The continuously monitored knowledge of proper operation of all downstream components is thus used to intelligently apply or defer panic alerts to keep the system operating uninterrupted from panic conditions that might soon or eventually be fixed by continued operation of the system pipeline.

Abstract: Embodiments for reducing panic shutdown of components in a pipelined data processing system. Components are monitored for health, processing progress, and dependencies during normal system operation. A directed graph is generated showing non-circular dependencies of components in the pipeline. Deadlock of a particular component may or may not signal a panic condition depending on whether any of its presently downstream and depended on components are operating properly. The continuously monitored knowledge of proper operation of all downstream components is thus used to intelligently apply or defer panic alerts to keep the system operating uninterrupted from panic conditions that might soon or eventually be fixed by continued operation of the system pipeline.

Abstract: Detecting a trend in latency for storage underlying a deduplicated file system includes generating a set of data points by recording when input/output (IO) requests were issued to the storage and recording time required to receive success responses from the storage. Least squares regression is performed on the data points to find a best-fit line through the data points. A slope of the best-fit line is calculated. A determination is made as to whether the slope is positive, a positive slope thereby indicating a trend of increasing latency of the storage. When the slope is determined to be positive, clients accessing the deduplicated file system are throttled.

Abstract: System generates data structure based on unique identifiers of objects in object storages and sets indicators in positions that correspond to hashes of unique identifiers of active objects. If a first number of regions of active data objects in first data storage and second number of regions of active data objects in second data storage each fail to satisfy data threshold, then system creates model identifying locations and sizes of regions of active data objects in first data storage and regions of active data objects in second data storage. System resets indicators in positions in data structure which correspond to hashes of unique identifiers of active data objects associated with model and enables remote storage to use model to copy regions of active data objects in first data storage and second data storage to third data storage, and to delete first data storage and second data storage.

Abstract: Embodiments are described for reducing memory usage and disk I/O's for reading and writing streams in a deduplication storage system. Generations of backups of a stream can be tracked for the stream. In the first generation (first backup), stream data is ingested, segmented, and written to disk without looking up a fingerprint of each segment, to save disk I/O's and memory. After the first generation, an amount of stream data processed is tracked, and at dynamically tunable increments of data processed, a fingerprint index lookup is performed to disk. The fingerprint index is read into memory and subsequent fingerprint lookups are preformed to cache, until a next dynamically tunable fingerprint lookup to disk. The dynamically tunable increment of processed data can be randomly reset.

Abstract: Embodiments are directed to a method for preserving temporal locality in disk-based backup systems by receiving a plurality of save streams each comprising a data stream and a metadata stream, directing the data streams to a plurality of respective data containers, and the metadata streams to a single shared metadata container, and assigning metadata streams using the shared metadata container to their own respective compression region to preserve locality at the compression region level.

Abstract: In one aspect of the disclosure, progress identifiers (e.g., a plurality of flags) and/or generation numbers are utilized to determine a “correct” non-volatile random access memory (NVRAM) replay source to use during a retried switchover or early switchback. Initially, a switchover sequence is initiated. The switchover sequence includes a replay of logged information to storage devices of an aggregate to ensure data consistency of the aggregate. Progress of the switchover sequence is tracked using the progress identifiers and/or generation numbers. After an interruption during the switchover sequence, switchover is retried or an early switchback is performed. The retried switchover or early switchback decides whether to replay logged information, and, if so, determines an NVRAM replay source to replay logged information from based on states of the progress identifiers and/or comparison of the generation numbers.

Abstract: In one aspect of the disclosure, progress identifiers (e.g., a plurality of flags) and/or generation numbers are utilized to determine a “correct” non-volatile random access memory (NVRAM) replay source to use during a retried switchover or early switchback. Initially, a switchover sequence is initiated. The switchover sequence includes a replay of logged information to storage devices of an aggregate to ensure data consistency of the aggregate. Progress of the switchover sequence is tracked using the progress identifiers and/or generation numbers. After an interruption during the switchover sequence, switchover is retried or an early switchback is performed. The retried switchover or early switchback decides whether to replay logged information, and, if so, determines an NVRAM replay source to replay logged information from based on states of the progress identifiers and/or comparison of the generation numbers.

Abstract: In one aspect of the disclosure, progress identifiers (e.g., a plurality of flags) and/or generation numbers are utilized to determine a “correct” non-volatile random access memory (NVRAM) replay source to use during a retried switchover or early switchback. Initially, a switchover sequence is initiated. The switchover sequence includes a replay of logged information to storage devices of an aggregate to ensure data consistency of the aggregate. Progress of the switchover sequence is tracked using the progress identifiers and/or generation numbers. After an interruption during the switchover sequence, switchover is retried or an early switchback is performed. The retried switchover or early switchback decides whether to replay logged information, and, if so, determines an NVRAM replay source to replay logged information from based on states of the progress identifiers and/or comparison of the generation numbers.

Abstract: In one aspect of the disclosure, progress identifiers (e.g., a plurality of flags) and/or generation numbers are utilized to determine a “correct” non-volatile random access memory (NVRAM) replay source to use during a retried switchover or early switchback. Initially, a switchover sequence is initiated. The switchover sequence includes a replay of logged information to storage devices of an aggregate to ensure data consistency of the aggregate. Progress of the switchover sequence is tracked using the progress identifiers and/or generation numbers. After an interruption during the switchover sequence, switchover is retried or an early switchback is performed. The retried switchover or early switchback decides whether to replay logged information, and, if so, determines an NVRAM replay source to replay logged information from based on states of the progress identifiers and/or comparison of the generation numbers.