Abstract: Methods, computer program products, computer systems, and the like are disclosed that provide for scalable deduplication in an efficient and effective manner. For example, such methods, computer program products, and computer systems can include receiving a data object at an assigned node, determining whether the data object includes a sub-data object, and processing the sub-data object. The assigned node is a node of a plurality of nodes of a cluster, where the data object includes a data segment, and a signature. The signature is generated based, at least in part, on data of the data segment. The processing includes sending the sub-data object to a remote node. The remote node is another node of the plurality of nodes of the cluster.

Abstract: Various systems, methods, and processes for caching and referencing multiple fingerprints while data operations are ongoing are disclosed. A first fingerprint is generated based on a first fingerprinting process. The first fingerprint is stored in association with a second fingerprint, which is based on a second fingerprinting process. The first fingerprint and the second fingerprint are associated with the same data segment. Data operations such as a backup operation, a restore operation, or a replication operation can be performed while the conversion of the data segment from the second fingerprint to the first fingerprint is ongoing.

Abstract: Techniques for data backup and restoration are disclosed. In one embodiment, the techniques may be realized as a method including generating a first backup representing a database at a first time; after the first backup, generating a plurality of journal entries, each journal entry representing a change to the database made after the first time; and restoring the database from the first backup and the plurality of journal entries, the restored database including the changes represented by the entries.

Abstract: Methods, computer program products, and computer systems for the management of data references in an efficient and effective manner are disclosed. Such methods, computer program products, and computer systems include receiving a change tracking stream at the computer system, identifying a data object group, and performing a deduplication management operation on the data object group. The change tracking stream is received from a client computing system. The change tracking stream identifies one or more changes made to a plurality of data objects of the client computing system. The identifying is based, at least in part, on at least a portion of the change tracking stream. The data object group represents the plurality of data objects.

Abstract: Various systems and methods are disclosed for optimizing data segments used during inline data deduplication, among other features and functionality. In one embodiment, such functionality includes buffering data received from a node, where an offset value is associated with the data and a segment size is associated with the buffer, until the buffer becomes full or a trigger event occurs. In response to determining that the buffer is full or that a trigger event has occurred, determining whether the offset value is an integer multiple of the segment size. If the offset value is not an integer multiple of the segment size, the functionality generates a modified segment, and then determines whether that modified segment is a duplicate of data stored in a deduplicated data store. If the modified segment is not a duplicate, the modified segment is stored in the deduplicated data store.

Abstract: Disclosed herein are systems, methods, and processes to perform replication between heterogeneous storage systems. In one embodiment, a request to perform a replication operation is sent by a target server to a source server, where the target server and the source server use different protocols to store data. A plurality of instructions, which are associated with a replication stream, are received from the source server by the target server, where the plurality of instructions comprise an include instruction to include existing data and a write instruction to write new data. A replication stream, which is associated with a backup stream stored on the source server, is also received from the source server, where the replication stream and the backup stream share a common format. The target server synthesizes a new replicated backup image, where the synthesizing comprises performing the include instruction and the write instruction on the replication stream.

Abstract: Disclosed herein are methods, systems, and processes to improve backup performance after backup failure. It is determined whether a backup operation is successful. In response to a determination that the backup operation was unsuccessful, backup metadata associated with the backup operation is replaced with snapshot metadata associated with a snapshot that is taken at a first point in time that is prior to initiation of the backup operation. A rolled-back snapshot that permits another backup operation to be performed at a second point in time based on the snapshot metadata is then generated.

Abstract: Disclosed herein are systems, methods, and processes to perform replication between heterogeneous storage systems. Information associated with a backup stream is recorded during a backup operation by a source server and includes instructions. The instructions include an include instruction to include existing data and a write instruction to write new data during a replication operation. A request to perform the replication operation is received. In response to the request, the information is sent to a target server as part of performing the replication operation.

Abstract: Disclosed herein are methods, systems, and processes to share data storage-related services between multiple deduplication systems. A determination is made as to whether a shared file that exists in a library of shared files corresponds to a local file. In response to a determination that the shared file exists in the library of shared files, the local file is virtually segmented into data segments, and local virtual mapping metadata is generated based, at least in part, on the data segments. The local virtual mapping metadata is then stored.

Abstract: A method for grouping backup images in deduplication groups is described. In one embodiment, the method includes identifying a new backup image, obtaining metadata from one or more nodes in the new backup image, and comparing the metadata from the one or more nodes in the new backup image with information from one or more nodes in a backup image from a first deduplication group. Each of the one or more nodes include at least in part a file or a directory. The first deduplication group is one group from a plurality of deduplication groups.

Abstract: The disclosed computer-implemented method for reducing data fragmentation may include (1) identifying update data which updates an initial data set; (2) categorizing, using a database manager, the update data based on how the update data is expected to impact digital storage device resources; (3) storing the update data in a physical storage device in a physical order based on the category of the update data; and (4) updating, after storing the update data, an extent map indicating a mapping of logical locations of the update data to physical locations at which the update data is stored. Various other methods, systems, and computer-readable media are also disclosed.

Abstract: A computer-implemented method for improving the efficiency of recording data to tape may include (i) identifying a command to duplicate a data unit to tape storage after a previous version of the data unit has already been duplicated to tape storage, (ii) identifying metadata that distinguishes between segments of the data unit that have not changed since the previous version of the data unit and segments that have changed, (iii) reading the previous version of the data unit from tape storage and reading the segments of the data unit that have changed from a data sharing storage rather than tape storage, and (iv) combining, using the metadata, the segments read from tape storage that have not changed and the segments read from the data sharing storage that have changed to duplicate the data unit to tape storage. Various other methods, systems, and computer-readable media are also disclosed.

Abstract: A deduplication storage system and associated methods are described. The deduplication storage system may split data objects into segments and store the segments. A plurality of data segment containers may be maintained. Each of the containers may include two or more of the data segments. Maintaining the containers may include maintaining a respective logical size of each container. In response to detecting that the logical size of a particular container has fallen below a threshold level, the deduplication storage system may perform an operation to reclaim the storage space allocated to one or more of the data segments included in the particular container.

Abstract: A method for data locality control in a deduplication system is provided. The method includes forming a fingerprint cache from a backup image corresponding to a first backup operation. The method includes removing one or more fingerprints from inclusion in the fingerprint cache, in response to the one or more fingerprints having a data segment locality, in a container, less than a threshold of data segment locality. The container has one or more data segments corresponding to the one or more fingerprints. The method includes applying the fingerprint cache, with the one or more fingerprints removed from inclusion therein, to a second backup operation, wherein at least one method operation is executed through a processor.

Abstract: Disclosed herein are methods, systems, and processes to improve backup performance after backup failure. It is determined whether a backup operation is successful. In response to a determination that the backup operation was unsuccessful, backup metadata associated with the backup operation is replaced with snapshot metadata associated with a snapshot that is taken at a first point in time that is prior to initiation of the backup operation. A rolled-back snapshot that permits another backup operation to be performed at a second point in time based on the snapshot metadata is then generated.

Abstract: The disclosed computer-implemented method for reducing data fragmentation may include (1) identifying update data which updates an initial data set; (2) categorizing, using a database manager, the update data based on how the update data is expected to impact digital storage device resources; (3) storing the update data in a physical storage device in a physical order based on the category of the update data; and (4) updating, after storing the update data, an extent map indicating a mapping of logical locations of the update data to physical locations at which the update data is stored. Various other methods, systems, and computer-readable media are also disclosed.

Abstract: The disclosed computer-implemented method for data placement in container-based storage systems may include (i) identifying a file stored within a container-based storage system, where the container-based storage system stores the file as data segments within containers, (ii) receiving, in response to a write operation directed to the file, a request to store within the container-based storage system a new data segment generated by the write operation, (iii) describing the file in terms of a plurality of consecutive slabs, (iv) determining that the new data segment falls within a specified slab, and (v) fulfilling the request to store the new data segment within the container-based storage system by storing the new data segment in a designated container that corresponds to the specified slab in response to determining that the new data segment falls within the specified slab. Various other methods, systems, and computer-readable media are also disclosed.

Abstract: The disclosed computer-implemented method for healing images in deduplication storage may include (i) detecting that a data segment stored within deduplication storage is corrupted by executing an algorithm on the data segment to generate a numerical identifier for the data segment and determining that the numerical identifier does not match a previously stored numerical identifier for the data segment, (ii) checking, during a backup operation, whether a digital fingerprint of a data segment of an image being backed up matches a digital fingerprint of the corrupted data segment that was taken prior to corruption and that is already stored within the deduplication storage, and (iii) appending the data segment from the image being backed up to a container that already contains the corrupted data segment rather than replacing the corrupted data segment. Various other methods, systems, and computer-readable media are also disclosed.

Abstract: Various systems, methods, and processes for caching and referencing multiple fingerprints while data operations are ongoing are disclosed. A first fingerprint is generated based on a first fingerprinting process. The first fingerprint is stored in association with a second fingerprint, which is based on a second fingerprinting process. The first fingerprint and the second fingerprint are associated with the same data segment. Data operations such as a backup operation, a restore operation, or a replication operation can be performed while the conversion of the data segment from the second fingerprint to the first fingerprint is ongoing.

Abstract: A computer-implemented method for taking snapshots in a deduplicated virtual file system may include (1) maintaining a deduplicated virtual file system that stores, at an original location within a non-virtual file system, at least one configuration file storing metadata for a target file and an extent map for the target file, the extent map defining how to construct the target file from deduplicated data segments in a deduplicated storage system, (2) receiving a request to take a snapshot of the target file corresponding to the configuration file, (3) copying the configuration file storing metadata for the target file and the extent map for the target file into a snapshot location within the non-virtual file system, and (4) transmitting a file reference request to the deduplicated storage system to add a file reference within the deduplicated storage system. Various other methods, systems, and computer-readable media are also disclosed.