Abstract: A method and system is used in managing deduplication of data in storage systems. A first digest for a deduplication candidate is received. At least one neighboring digest segment of a first loaded digest segment is loaded. Whether the digest is located in the at least one neighboring digest segment is determined. If the digest is not located in the at least one neighboring digest segment, the digest is processed.

Abstract: A method and system is used in managing deduplication of data in storage systems. A first digest for a deduplication candidate is received. At least one stream associated with the deduplication candidate is detected. At least one neighboring digest segment of a first loaded digest segment associated with the at least one stream is loaded. Whether the digest is located in the at least one neighboring digest segment is determined. If the digest is not located in the at least one neighboring digest segment, the digest is processed.

Abstract: A fast snap is a consistent point-in-time copy or logical copy of a storage object that can be created more quickly than an ordinary snap. In order to generate a fast snap a storage array creates a copy or logical copy of all of the dirty data associated with a source device by copying or logically copying dirty pages cache into a separate record in volatile memory, instantiates an unpopulated snap device that is backed by persistent storage, and generates a mapping between the record and the unpopulated snap device. The mapping includes the information needed to populate the unpopulated snap device with the dirty pages from the record. Once the fast snap has been created it is possible to service IOs to the source device and the fast snap device because the data can be accessed from the record and the cache.

Abstract: A method is used in selecting iteration schemes for deduplication. An iteration scheme is selected. Based on the selection, sets of data are iterated. Based on the iteration, a deduplicating technique is applied to the sets of data. Based on the results of applying the deduplicating technique, the iteration scheme is changed. Based on an evaluation of a set of criteria, the iteration scheme is changed.

Abstract: A method, article of manufacture, and apparatus for efficiently backing up information are disclosed. In some embodiments, this comprises receiving data and analyzing the data. In some embodiments, the data may be manipulated. In some embodiments, the data may come from multiple sources. Chunk headers, data chunks and chunk maps are then created based on analyzing the data. Chunk headers store the data manipulation technique of the data chunk. In some embodiments, chunk headers store the size of the manipulated and un-manipulated data chunks. The chunk headers, data chunks and chunk maps are then organized into a datastream. The datastream is then stored in a storage device. In some embodiments, the received data may be discarded. In some embodiments, a chunk header is associated with a data chunk.

Abstract: An object management system (10) comprises a storage device (12); a first computer system (12) configured to store a first object, and a copy of the first object on the storage device; and a second computer system. The first computer system is configured to create a first data object including information about the stored copy of the first object, and the second computer system is configured to read the information about the stored copy of the first object from the first data object, and retrieve the stored copy of the first object from the storage device. The first object may be partitioned by the first computer system and stored as partitions on the storage device, and the first data object would contain information about the partitions. The second computer system would be configured to read information about the partitions from the first data object, select partitions from the storage device.

Abstract: A method, article of manufacture, and apparatus for efficiently recovering information are disclosed. In some embodiments, this comprises receiving a datastream and analyzing the datastream for chunk headers, data chunks and chunk maps. The chunk headers are then used to determine the data manipulation technique of the data chunks. In some embodiments, the chunk headers contain information about the size of the manipulated and un-manipulated data chunks. Data is then reconstructed from the datastream analysis and data manipulation technique determination. The reconstructed data is then stored in a computer readable medium. In some embodiments, the reconstructed data may be stored in multiple destinations. In some embodiments, the storage of the reconstructed data in multiple destinations may be simultaneous.

Abstract: A computer system configured to store a plurality of files, and a data file, which includes inode and migration or purge information associated with the files. The computer system searches the data file to identify files that are candidates for migration or purge. A plurality of pathnames associated with the files is stored in the data file or a separate data file, and the computer system obtains the pathnames of candidate files from the data file (or separate data file). When candidate files are identified, their inodes are used to look up their pathnames in the plurality of pathnames. The data file may include information about whether a file has been modified and its time of last access. To determine whether a file is a candidate for migration, its entry in the data file is checked to determine whether it has been modified. If it has, it is identified as a migration candidate.

Abstract: A method, article of manufacture, and apparatus for managing data commonality in a Merkle tree is disclosed. Reference counts are associated with a node in the Merkle tree. Data commonality is detected and the root of the detected data commonality is determined. If a node is the root node of the detected data commonality, the nodes reference count is incremented. When an object is deleted, the root node of the object is determined. The reference count of the node is decremented. If the count reaches zero, the node is removed from the Merkle tree, and its child nodes are decremented.

Abstract: An object management system comprises a computer system configured to store an object (72), which may be a file. The computer system is configured to divide the object (72) into partitions (70), select partitions, and provide the selected partitions to a storage device (14) for storage. The partitions are selected by determining which partitions need to be migrated device, based on whether the partitions have been modified. The system determines this by examining a data object associated with the object. The computer system may be configured to retrieve a selected partition from the storage device.

Abstract: The invention provides a system to reclaim space identified as no longer in use and comprises a vLUN, a thinly provisioned mapped LUN, a mapping layer, and a data reduction engine. Chunks of data are stored at logical chunk addresses (LCAs) in the vLUN and are mapped to corresponding physical chunk addresses (PCAs) in the thinly provisioned mapped LUN. The data reduction engine performs a data reduction process on a first logical chunk of data stored at a first LCA in the vLUN, where the first logical chunk has a size that is a nonzero integer multiple of the size of the storage extent of the thinly provisioned mapped LUN. After the data reduction process, the PCA associated with the first logical chunk is no longer needed, and the thinly provisioned mapped LUN is instructed to deallocate the PCA associated with the first logical chunk that is no longer needed.

Abstract: The invention provides a system to reclaim space identified as no longer in use and comprises a vLUN, a thinly provisioned mapped LUN, a mapping layer, and a data reduction engine. Chunks of data are stored at logical chunk addresses (LCAs) in the vLUN and are mapped to corresponding physical chunk addresses (PCAs) in the thinly provisioned mapped LUN. The data reduction engine performs a data reduction process on a first logical chunk of data stored at a first LCA in the vLUN, where the first logical chunk has a size that is a nonzero integer multiple of the size of the storage extent of the thinly provisioned mapped LUN. After the data reduction process, the PCA associated with the first logical chunk is no longer needed, and the thinly provisioned mapped LUN is instructed to deallocate the PCA associated with the first logical chunk that is no longer needed.

Abstract: A system configured to optimize access to stored chunks of data is provided. The system comprises a vLUN layer, a mapped LUN layer, and a mapping layer disposed between the vLUN and the mapped LUN. The vLUN provides a plurality of logical chunk addresses (LCAs) and the mapped LUN provides a plurality of physical chunk addresses (PCAs), where each LCA or PCA stores a respective chunk of data. The mapping layer defines a layout of the mapped LUN that facilitates efficient read and write access to the mapped LUN.

Abstract: An object management system (10) comprises a storage device (12); a first computer system (12) configured to store a first object, and a copy of the first object on the storage device; and a second computer system. The first computer system is configured to create a first data object including information about the stored copy of the first object, and the second computer system is configured to read the information about the stored copy of the first object from the first data object, and retrieve the stored copy of the first object from the storage device. The first object may be partitioned by the first computer system and stored as partitions on the storage device, and the first data object would contain information about the partitions. The second computer system would be configured to read information about the partitions from the first data object, select partitions from the storage device.

Abstract: An object management system (10) comprises a storage device (12); a first computer system (12) configured to store a first object, and a copy of the first object on the storage device; and a second computer system. The first computer system is configured to create a first data object including information about the stored copy of the first object, and the second computer system is configured to read the information about the stored copy of the first object from the first data object, and retrieve the stored copy of the first object from the storage device. The first object may be partitioned by the first computer system and stored as partitions on the storage device, and the first data object would contain information about the partitions. The second computer system would be configured to read information about the partitions from the first data object, select partitions from the storage device.

Abstract: An object management system (10) comprises a storage device (12); a first computer system (12) configured to store a first object, and a copy of the first object on the storage device; and a second computer system. The first computer system is configured to create a first data object including information about the stored copy of the first object, and the second computer system is configured to read the information about the stored copy of the first object from the first data object, and retrieve the stored copy of the first object from the storage device. The first object may be partitioned by the first computer system and stored as partitions on the storage device, and the first data object would contain information about the partitions. The second computer system would be configured to read information about the partitions from the first data object, select partitions from the storage device.

Abstract: A computer system configured to store a plurality of files, and a data file, which includes inode and migration or purge information associated with the files. The computer system searches the data file to identify files that are candidates for migration or purge. A plurality of pathnames associated with the files is stored in the data file or a separate data file, and the computer system obtains the pathnames of candidate files from the data file (or separate data file). When candidate files are identified, their inodes are used to look up their pathnames in the plurality of pathnames. The data file may include information about whether a file has been modified and its time of last access. To determine whether a file is a candidate for migration, its entry in the data file is checked to determine whether it has been modified. If it has, it is identified as a migration candidate.

Abstract: An object management system comprises a computer system configured to store an object (72), which may be file. The computer system is configured to divide the object (72) into partitions (70), select partitions, and provide the selected partitions to a storage device (14) for storage. The partitions are selected by determining which partitions need to be migrated device, based on whether the partitions have been modified. The system determines this by examining a data object associated with the object. The computer system may be configured to retrieve a selected partition from the storage device.