Abstract: Provided are techniques for data deduplication. A chunk of data and a mapping of boundaries between file data and meta data in the chunk of data are received. The mapping is used to split the chunk of data into a file data stream and a meta data stream and to store file data from the file data stream in a first file and to store meta data from the meta data stream in a second file, wherein the first file and the second file are separate files. The file data in the first file is deduplicated.

Abstract: Various techniques for improving the performance of restoring deduplicated data files from a server to a client within a storage management system are disclosed. In one embodiment, a chunk index is maintained on the client that tracks the chunks remaining on the client for each data file that is stored to and restored from the storage server. When a specific file is selected for restore from the storage server to the client, the client determines if any local copies of this specific file's chunks are stored in files already existing on the client data store. The file is then reconstructed from a combination of these local copies of the file chunks and chunks retrieved from the storage server. Therefore, only chunks that are not stored or are inaccessible to the client are retrieved from the server, reducing server-side processing requirements and the bandwidth required for data restore operations.

Abstract: The various embodiments of the present invention include techniques for securing the use of data deduplication activities occurring in a source-deduplicating storage management system. These techniques are intended to prevent fake data backup, target data contamination, and data spoofing attacks initiated by a source. In one embodiment, one technique includes limiting chunk querying to authorized users. Another technique provides detection of attacks and unauthorized access to keys within the target system. Additional techniques include the combination of validating the existence of data from the source by validating the data chunk, validating a data sample of the data chunk, or validating a hash value of the data chunk. A further embodiment involves the use of policies to provide authorization levels for chunk sharing and linking within the target. These techniques separately and in combination provide a comprehensive strategy to avoid unauthorized access to data within the target storage system.

Abstract: One aspect of the present invention includes a configuration of a storage management system that enables the performance of deduplication activities at both the client (source) and at the server (target) locations. The location of deduplication operations can then be optimized based on system conditions or predefined policies. In one embodiment, seamless switching of deduplication activities between the client and the server is enabled by utilizing uniform deduplication process algorithms and accessing the same deduplication index (containing information on the hashed data chunks). Additionally, any data transformations on the chunks are performed subsequent to identification of the data chunks. Accordingly, with use of this storage configuration, the storage system can find and utilize matching chunks generated with either client- or server-side deduplication.

Abstract: Various embodiments are provided for facilitation of data retention using logical objects. Following an operation creating a redundant copy of the data performed on a scheduled interval, a logical object containing a number of managed file versions, represented by a number of member objects for a recovery point, is created. The logical object is assigned a policy of a data retention policy construct associated with the recovery point. The logical object is adapted for reassignment between policies of the data retention policy construct associated with various recovery points. During the reassignment, the plurality of member objects representing the plurality of managed file versions are logically retained instead of performing a data copy operation to associate the plurality of managed file versions with another recovery point.

Abstract: The present invention provides for a system and method for assuring integrity of deduplicated data objects stored within a storage system. A data object is copied to secondary storage media, and a digital signature such as a checksum is generated of the data object. Then, deduplication is performed upon the data object and the data object is split into chunks. The chunks are combined when the data object is subsequently accessed, and a signature is generated for the reassembled data object. The reassembled data object is provided if the newly generated signature is identical to the originally generated signature, and otherwise a backup copy of the data object is provided from secondary storage media.

Abstract: The present invention provides for a system and method for assuring integrity of deduplicated data objects stored within a storage system. A data object is copied to secondary storage media, and a digital signature such as a checksum is generated of the data object. Then, deduplication is performed upon the data object and the data object is split into chunks. The chunks are combined when the data object is subsequently accessed, and a signature is generated for the reassembled data object. The reassembled data object is provided if the newly generated signature is identical to the originally generated signature, and otherwise a backup copy of the data object is provided from secondary storage media.

Abstract: One aspect of the present invention includes an optimized simultaneous storage operation for data objects onto a combination of deduplicated and non-deduplicated storage pools. In one embodiment, a data object is provided for storage onto destination storage pools in a storage management system, and placed into a source buffer. The data object is first divided into data chunks if the data object has not previously been chunked within the storage management system. The data object is then simultaneously copied from the source buffer to each destination storage pool (deduplicating and non-deduplicating) with the following operation. If the destination pool utilizes deduplication, then the individual data chunks are only transferred if copies of the individual data chunks do not already exist on the destination storage pool. If the destination pool does not utilize deduplication, then all chunks of the data object are transferred to the destination storage pool.

Abstract: One aspect of the present invention includes enabling data chunks to be shared among different storage pools within a storage management system, according the use of deduplication and storage information kept at the system level, and applied with policy-based rules that define the scope of deduplication. In one embodiment, the parameters of performing deduplication are defined within the policy, particularly which of the plurality of storage pools allow deduplication to which other pools. Accordingly, a data object may be linked to deduplicated data chunks existent within other storage pools, and the transfer of a data object may occur by simply creating references to existing data chunks in other pools provided the policy allows the pool to reference chunks in these other pools. Additionally, a group of storage pools may be defined within the policy to perform a common set of deduplication activities across all pools within the group.

Abstract: One aspect of the present invention includes retrieving and recovering data chunks from alternate data stores in a storage management system which utilizes deduplication. In one embodiment, deduplication information for data chunks of data objects is stored at a system-wide level to enable the transfer and access of data chunks stored among multiple storage pools. When a data object is accessed on a first storage pool that contains damaged or inaccessible data chunks, the undamaged and accessible chunks may be retrieved from the first storage pool, in addition to retrieving an undamaged copy of the damaged or inaccessible data chunks from alternate data storage pools. Thus, a complete data object can be retrieved or recovered with a combination of chunks from the first storage pool and other storage pools within the storage management system, without requiring the entire data object to be retrieved from a backup source.

Abstract: One aspect of the present invention includes enabling the efficient transfer of deduplicated data between storage pools in a storage management system without unnecessary reassembly and deduplication of data objects. In one embodiment, the storage management system tracks deduplication information for the data chunks of data objects within an index at the storage management system level, in addition to tracking storage information for each data object within another index at the storage management system level. The data chunk deduplication information is then accessible by any storage pool. Accordingly, transfers of the data objects and data chunks of the data object are easily facilitated, even between non-deduplicating and deduplicating storage pools.

Abstract: The present invention provides for an enhanced method and system for assuring integrity of deduplicated data objects stored within a storage system. A digital signature of the data object is generated to determine if the data object reassembled from a deduplicated state is identical to its pre-deduplication state. In one embodiment, generating the object signature of a data object before deduplication comprises generating an object signature from intermediate hash values computed from a hash function operating on each data chunk within the data object, the hash function also used to determine duplicate data chunks. In an alternative embodiment, generating the object signature of a data object before deduplication comprises generating an object signature on a portion of each data chunk of the data object.

Abstract: The present invention provides for a system and method for assuring integrity of deduplicated data objects stored within a storage system. A data object is copied to secondary storage media, and a digital signature such as a checksum is generated of the data object. Then, deduplication is performed upon the data object and the data object is split into chunks. The chunks are combined when the data object is subsequently accessed, and a signature is generated for the reassembled data object. The reassembled data object is provided if the newly generated signature is identical to the originally generated signature, and otherwise a backup copy of the data object is provided from secondary storage media.

Abstract: The present invention provides for a method, system, and computer program for the application of data deduplication according to a policy-based strategy of tiered data. The method operates by defining a plurality of data storage policies for data in a deduplication system, policies which may be arranged in tiers. Data objects are classified according to a selected data storage policy and are split into data chunks. If the selected data storage policy for the data object does not allow deduplication, the data chunks are stored in a deduplication pool. If the selected data storage policy for the data object allows deduplication, deduplication is performed. The data storage policy may specify a maximum number of references to data chunks, facilitating storage of new copies of the data chunks when the maximum number of references is met.

Abstract: Systems and methods for retrieving data from a storage system having a plurality of storage pools are provided. The system comprises a logic unit for processing configurable data retrieval instructions to determine a first storage pool from which target backup data is to be retrieved, in response to a data restore request; and a logic unit for retrieving the target backup data from the first storage pool to satisfy the restore request. The configurable data retrieval instructions are managed by a source external to the storage system with administrative authority to change the configurable data retrieval instructions to optimize data restoration from the storage system.

Abstract: Systems and methods for retrieving data from a storage system having a plurality of storage pools are provided. The method comprises processing configurable data retrieval instructions to determine a first storage pool from which target backup data is to be retrieved, in response to a data restore request; and retrieving the target backup data from the first storage pool to satisfy the restore request. The configurable data retrieval instructions are managed by a source external to the storage system with administrative authority to change the configurable data retrieval instructions to optimize data restoration from the storage system.

Abstract: A process is disclosed by which data is securely deleted in a transactionally consistent manner. This may be accomplished by committing a preparation transaction for a data object within a system managing the data object in order to return the system to an initial condition if necessary, attempting to commit an execution transaction with the data object only after committing the preparation transaction, and securely deleting any portion of the data object necessary to return the system to the initial condition if committing the execution transaction fails and to change the system to a completed condition only if committing the execution transaction succeeds. In a delete or move transaction an existing backup object may be assigned a new logically deleted state such that if the delete or move transaction fails, the data will be made accessible again.

Abstract: A method and system for integrating migration level two (ML2) and backup tape processing provide for the backup, archival, and/or restoration of ML2 tape files without the use of tape mounts. In this manner, the amount of data movement required to recover from data loss is significantly reduced. Tape files may be recovered to their original status, even if the file has migrated to ML2. The status of the data is preserved after such a recovery operation. In this manner, data movement is reduced because no remigration of data is required after a recovery.

Abstract: A data storage hierarchy which inherently allows for a level 1 storage file to be uniquely identified across an entire network is disclosed. A directory naming convention is employed which includes an internal identifier and a name for each subdivision of the network. Because each file can be uniquely identified across the network, a single level 1 storage space in a file space, or a directory therein, can be used for the entire network. Also, because of the inherent uniqueness of the naming system, common DASD control files otherwise required to map between level 1 storage files and their level 0 source files can be eliminated.