Abstract: A system configured to generate a macro-fingerprint from at least one predefined set of summaries is provided. The system includes data storage storing a first predefined set of summaries associated with a first region of data, each member of the first predefined set of summaries characterizing data within the first region of data; and at least one processor coupled to the data storage and configured to: read the first predefined set of summaries; select at least one first member from the first predefined set of summaries based on a value of the at least one first member; and store the at least one first member within a first macro-fingerprint. The first region of data may have a first size indicative of a quantity of data included in the first region of data. The macro fingerprints are created from previously created smaller (micro) fingerprints without having to reread the data.

Abstract: Described are computer-based methods and apparatuses, including computer program products, for detection and deduplication of backup sets exhibiting poor locality. A first set of summaries of a first data set are determined, each summary of the first set of summaries being indicative of a data pattern in the first data set. A second set of summaries of a second data set are determined, each summary of the second set of summaries being indicative of a data pattern in the second data set. A set of comparison metrics are calculated, each comparison metric being based on a first subset of summaries from the first set of summaries and a second subset of summaries from the second set of summaries. A locality metric is calculated based on the set of comparison metrics indicative of whether the first data set and second data set exhibit poor locality.

Abstract: A method for removing redundant data from a backup storage system is presented. In one example, the method may include identifying a first back-up data object, identifying a second back-up data object, detecting a first portion of the first back-up data object that is a copy of a second portion of the second back-up data object, and replacing the second portion with a pointer to the first portion.

Abstract: Described are computer-based methods and apparatuses, including computer program products, for dynamic deduplication. Data is processed using an algorithm that deduplicates the data based on a first set of parameters. A first moving average of a first deduplication performance metric is calculated for the algorithm over a time period. A second moving average of a second deduplication performance metric is calculated for the algorithm over the time period. It is determined that the first moving average satisfies a first criterion, the second moving average satisfies a second criterion, or both. The algorithm is reconfigured based on a second set of parameters to deduplicate data.

Abstract: A back-up storage system that emulates a sequential storage medium such as a tape and stores data on a randomly accessible storage medium, such as disk. The back-up storage system includes a randomly accessible storage medium, a front-end interface to emulate the sequential storage medium, to communicate with external devices using a protocol that supports the sequential storage medium, and to receive sequential-format data from the external devices, and a back-end interface to receive the sequential-format data from the front-end interface and to store the sequential-format data on the randomly-accessible storage medium.

Abstract: Described are computer-based methods and apparatuses, including computer program products, for removing redundant data from a storage system. In one example, a data delineation process delineates data targeted for de-duplication into regions using a plurality of markers. The de-duplication system determines which of these regions should be subject to further de-duplication processing by comparing metadata representing the regions to metadata representing regions of a reference data set. The de-duplication system identifies an area of data that incorporates the regions that should be subject to further de-duplication processing and de-duplicates this area with reference to a corresponding area within the reference data set.

Abstract: Described are computer-based methods and apparatuses, including computer program products, for system health monitoring. Backup set metadata is received, wherein the backup set metadata comprises information about backup data sets that are received by a backup storage system. One or more processes that process the backup set metadata through an emulated processing flow path are executed, wherein the one or more processes are also implemented in the backup storage system. Two or more potential processing states are determined within the emulated processing flow path. A reason code is determined for each backup set metadata entry of the backup set metadata indicative of a reason that the backup set metadata entry is in a processing state of the two or more potential processing states. A problem with the manner in which the backup set metadata is flowing through the emulated processing flow path is identified based on the reason codes.

Abstract: Described are computer-based methods and apparatuses, including computer program products, for removing redundant data from a storage system. In one example, a data delineation process delineates data targeted for de-duplication into regions using a plurality of markers. The de-duplication system determines which of these regions should be subject to further de-duplication processing by comparing metadata representing the regions to metadata representing regions of a reference data set. The de-duplication system identifies an area of data that incorporates the regions that should be subject to further de-duplication processing and de-duplicates this area with reference to a corresponding area within the reference data set.

Abstract: Described are computer-based methods and apparatuses, including computer program products, for removing redundant data from a storage system. In one example, a data delineation process delineates data targeted for de-duplication into regions using a plurality of markers. The de-duplication system determines which of these regions should be subject to further de-duplication processing by comparing metadata representing the regions to metadata representing regions of a reference data set. The de-duplication system identifies an area of data that incorporates the regions that should be subject to further de-duplication processing and de-duplicates this area with reference to a corresponding area within the reference data set.

Abstract: Described are computer-based methods and apparatuses, including computer program products, for removing redundant data from a storage system. In one example, a data delineation process delineates data targeted for de-duplication into regions using a plurality of markers. The de-duplication system determines which of these regions should be subject to further de-duplication processing by comparing metadata representing the regions to metadata representing regions of a reference data set. The de-duplication system identifies an area of data that incorporates the regions that should be subject to further de-duplication processing and de-duplicates this area with reference to a corresponding area within the reference data set.

Abstract: Backup set metadata is received, wherein the backup set metadata comprises information about backup data sets that are received by a backup storage system that stores the backup data sets. The manner in which the backup data sets flow through a processing flow path of the backup storage system is emulated. One or more processes that process the backup set metadata through an emulated processing flow path are executed, wherein the emulated processing flow path is indicative of the manner in which the backup data sets flow through the processing flow path of the backup storage system when the backup storage system stores the backup data sets. One or more timing statistics are calculated based on the flow of the backup set metadata through the emulated processing flow path.

Abstract: A method for removing redundant data from a backup storage system is presented. In one example, the method may include receiving the application layer data object, selecting a de-duplication domain from a plurality of de-duplication domains based at least in part on a data object characteristic associated with the de-duplication domain, determining that the application layer data object has the characteristic and directing the application layer data object to the de-duplication domain.

Abstract: Described are computer-based methods and apparatuses, including computer program products, for dynamic deduplication. Data is processed using an algorithm that deduplicates the data based on a first set of parameters. A first moving average of a first deduplication performance metric is calculated for the algorithm over a time period. A second moving average of a second deduplication performance metric is calculated for the algorithm over the time period. It is determined that the first moving average satisfies a first criterion, the second moving average satisfies a second criterion, or both. The algorithm is reconfigured based on a second set of parameters to deduplicate data.

Abstract: Described are computer-based methods and apparatuses, including computer program products, for detection and deduplication of backup sets exhibiting poor locality. A first set of summaries of a first data set are determined, each summary of the first set of summaries being indicative of a data pattern in the first data set. A second set of summaries of a second data set are determined, each summary of the second set of summaries being indicative of a data pattern in the second data set. A set of comparison metrics are calculated, each comparison metric being based on a first subset of summaries from the first set of summaries and a second subset of summaries from the second set of summaries. A locality metric is calculated based on the set of comparison metrics indicative of whether the first data set and second data set exhibit poor locality.

Abstract: Systems and methods for efficient storage of data are provided. For instance, a method that includes acts of receiving a data stream to be stored on a first storage system, the data stream comprising a data object and metadata that describes attributes of the data object, extracting the metadata from the data stream, storing the data object on the first storage system and storing the extracted metadata, separately from the data object, on a computer-readable medium is disclosed. Additionally, a storage system is disclosed that includes a first storage medium, an interface configured to receive a data stream including a data object to be stored on the first storage medium and a controller. The controller may be configured to extract metadata from the data stream, the metadata being descriptive of attributes of the data object, and to direct the extracted metadata to be stored on a computer-readable medium.

Abstract: Backup set metadata is received, wherein the backup set metadata comprises information about backup data sets that are received by a backup storage system that stores the backup data sets. The manner in which the backup data sets flow through a processing flow path of the backup storage system is emulated. One or more processes that process the backup set metadata through an emulated processing flow path are executed, wherein the emulated processing flow path is indicative of the manner in which the backup data sets flow through the processing flow path of the backup storage system when the backup storage system stores the backup data sets. One or more timing statistics are calculated based on the flow of the backup set metadata through the emulated processing flow path.

Abstract: Described are computer-based methods and apparatuses, including computer program products, for system health monitoring. Backup set metadata is received, wherein the backup set metadata comprises information about backup data sets that are received by a backup storage system. One or more processes that process the backup set metadata through an emulated processing flow path are executed, wherein the one or more processes are also implemented in the backup storage system. Two or more potential processing states are determined within the emulated processing flow path. A reason code is determined for each backup set metadata entry of the backup set metadata indicative of a reason that the backup set metadata entry is in a processing state of the two or more potential processing states. A problem with the manner in which the backup set metadata is flowing through the emulated processing flow path is identified based on the reason codes.

Abstract: Described are computer-based methods and apparatuses, including computer program products, for garbage collection. A garbage collection data structure is provided for deleting unused data objects. One or more object identifiers are stored in a first data structure in the garbage collection data structure. Each object identifier represents a data object about to be created but not yet assigned any references from other data objects. The first data structure prevents the data object from being deleted during creation of the data object but before one or more references are created to the data object. Data indicative of one or more objects is stored in a second data structure in the garbage collection data structure. The data includes one or more object identifiers, each object identifier representing a created data object. The data also includes one or more references to created data objects.

Abstract: Described are computer-based methods and apparatuses, including computer program products, for removing redundant data from a storage system. In one example, a data delineation process delineates data targeted for de-duplication into regions using a plurality of markers. The de-duplication system determines which of these regions should be subject to further de-duplication processing by comparing metadata representing the regions to metadata representing regions of a reference data set. The de-duplication system identifies an area of data that incorporates the regions that should be subject to further de-duplication processing and de-duplicates this area with reference to a corresponding area within the reference data set.

Abstract: Described are computer-based methods and apparatuses, including computer program products, for removing redundant data from a storage system. In one example, a data delineation process delineates data targeted for de-duplication into regions using a plurality of markers. The de-duplication system determines which of these regions should be subject to further de-duplication processing by comparing metadata representing the regions to metadata representing regions of a reference data set. The de-duplication system identifies an area of data that incorporates the regions that should be subject to further de-duplication processing and de-duplicates this area with reference to a corresponding area within the reference data set.