Abstract: A system and method for data replication is described. A destination storage system receives a message from a source storage system as part of a replication process. The message includes an identity of a first file, information about where the first file is stored in the source storage system, a name of a first data being used by the first file and stored at a first location of the source storage system, and a fingerprint of the first data. The destination storage system determines that a mapping database is unavailable or inaccurate, and accesses a fingerprint database using the fingerprint of the first data received with the message to determine whether data stored in the destination storage system has a fingerprint identical to the fingerprint of the first data.

Abstract: A system and method for data replication is described. A destination storage system receives a message from a source storage system as part of a replication process. The message includes an identity of a first file, information about where the first file is stored in the source storage system, a name of a first data being used by the first file and stored at a first location of the source storage system, and a fingerprint of the first data. The destination storage system determines that a mapping database is unavailable or inaccurate, and accesses a fingerprint database using the fingerprint of the first data received with the message to determine whether data stored in the destination storage system has a fingerprint identical to the fingerprint of the first data.

Abstract: Presented herein are mass data storage systems, file system protocols, non-transitory machine readable devices, and methods for storing data blocks in data file systems. Methods for compressing snapshot data in a data file system are disclosed which include: loading a snapshot file with one or more data blocks, the snapshot representing a state of the data file system at a point in time; determining if at least one of the snapshot data blocks is less than a predetermined byte value; responsive to a snapshot data block having a size that is less than the predetermined byte value, identifying a packed block configured to store data chunks from plural distinct snapshots and having available sufficient storage space to store the snapshot data block; and adding to the packed block the snapshot data block and lost-write context information corresponding to the snapshot data block.

Abstract: Presented herein are mass data storage systems, file system protocols, non-transitory machine readable devices, and methods for storing data blocks in data file systems. Methods for compressing snapshot data in a data file system are disclosed which include: loading a snapshot file with one or more data blocks, the snapshot representing a state of the data file system at a point in time; determining if at least one of the snapshot data blocks is less than a predetermined byte value; responsive to a snapshot data block having a size that is less than the predetermined byte value, identifying a packed block configured to store data chunks from plural distinct snapshots and having available sufficient storage space to store the snapshot data block; and adding to the packed block the snapshot data block and lost-write context information corresponding to the snapshot data block.

Abstract: Presented herein are mass data storage networks, file system protocols, non-transitory machine readable devices, and methods for storing data blocks in mass data storage systems. Methods for storing data blocks in a file system are disclosed which include: receiving by storage controller of the data storage system a request to write a data file to a system storage module; determining whether the data file includes a sub-K data chunk that is less than approximately four kilobytes; identifying a packed block that stores a plurality of sub-K data chunks and has sufficient storage space available to store the sub-K data chunk; and placing, by the storage controller in the packed block, the sub-K data chunk and a corresponding data length and a respective offset identifying a location of the sub-K data chunk in the packed block.

Abstract: Systems, methods, and computer program products implementing hybrid file structures for data storage are provided. One embodiment of a method performed in a computer-based storage system includes writing a file as data blocks in an array of storage devices. The method includes associating the data blocks with metadata related to at least one location in the array of storage devices for later access to the data blocks. The file is represented as a hierarchical data structure having a plurality of nodes. A first portion of nodes has a first span type, and a second portion of nodes has a second span type. The data structure includes a buftree. The first span type includes a fixed-span type. The second span type includes a variable-span type.

Abstract: Techniques to clone a writeable data object in non-persistent memory are disclosed. The writeable data object is stored in a storage structure in non-persistent memory that corresponds to a portion of a persistent storage. The techniques enable cloning of the writeable data object without having to wait until the writeable data object is saved to the persistent storage and without needing to quiesce incoming operations (e.g., reads and writes) to the writeable data object.

Abstract: A method for information management comprises intercepting an output from an application; distributing packets according to a routing scheme, wherein the packets are associated with the output, and wherein distributing the packets may occur when the application is associated with a first operating system, and may also occur when the application is associated with a second operating system; and storing the packets.

Abstract: A first plurality of block identifiers is sorted based, at least in part, on a measure of spatial locality. A second plurality of block identifiers is sorted based, at least in part, on the measure of spatial locality. At least the first plurality of block identifiers and the second plurality of block identifiers are incrementally merged into a third plurality of block identifiers based, at least in part, on the measure of spatial locality. A block of data corresponding to metadata associated with a plurality of block identifiers of the third plurality of block identifiers is updated.

Abstract: A system and method for logically organizing compressed data. In one aspect, a destination storage server receives a write request that includes multiple data blocks and specifies corresponding file block numbers. An extent-based file system executing on the storage server accesses intermediate block entries that each associates one of the file block numbers with a respective extent block number. The file system, in cooperation with a compression engine, compresses the data blocks into a set of one or more compressed data blocks. The file system stores the compressed data blocks at physical locations corresponding to physical block numbers and allocates, within an extent map, pointers from an extent ID to the extent block numbers, and pointers from the extent ID to the physical block numbers.

Abstract: Techniques for a method of estimating deduplication potential are disclosed herein. The method includes steps of selecting randomly a plurality of data blocks from a data set as a sample of the data set, collecting fingerprints of the plurality of data blocks of the sample, identifying duplicates of fingerprints of the sample from the fingerprints of the plurality of data blocks, estimating a total number of unique fingerprints of the data set depending on a total number of the duplicates of fingerprints of the sample based on a probability of fingerprints from the data set colliding in the sample, and determining a total number of duplicates of fingerprints of the data set depending on the total number of the unique fingerprints of the data set.

Abstract: A system and method for data replication is described. A destination storage system receives a message from a source storage system as part of a replication process. The message includes an identity of a first file, information about where the first file is stored in the source storage system, a name of a first data being used by the first file and stored at a first location of the source storage system, and a fingerprint of the first data. The destination storage system determines that a mapping database is unavailable or inaccurate, and accesses a fingerprint database using the fingerprint of the first data received with the message to determine whether data stored in the destination storage system has a fingerprint identical to the fingerprint of the first data.

Abstract: It is determined that a first data block contains the same data as a second data block. The first data block is associated with a first extent and the second data block is associated with a second extent. In response to determining that the first data block contains the same data as the second data block, the second data block is associated with the first extent and the first data block is disassociated with the second extent.

Abstract: Provided is a method and system for reducing duplicate buffers in buffer cache associated with a storage device. Reducing buffer duplication in a buffer cache includes accessing a file reference pointer associated with a file in a deduplicated filesystem when attempting to load a requested data block from the file into the buffer cache. To determine if the requested data block is already in the buffer cache, aspects of the invention compare a fingerprint that identifies the requested data block against one or more fingerprints identifying a corresponding one or more sharable data blocks in the buffer cache. A match between the fingerprint of the requested data block and the fingerprint from a sharable data block in the buffer cache indicates that the requested data block is already loaded in buffer cache. The sharable data block in buffer cache is used instead thereby reducing buffer duplication in the buffer cache.

Abstract: Systems, methods, and computer program products implementing hybrid file structures for data storage are provided. One embodiment of a method performed in a computer-based storage system includes writing a file as data blocks in an array of storage devices. The method includes associating the data blocks with metadata related to at least one location in the array of storage devices for later access to the data blocks. The file is represented as a hierarchical data structure having a plurality of nodes. A first portion of nodes has a first span type, and a second portion of nodes has a second span type. The data structure includes a buftree. The first span type includes a fixed-span type. The second span type includes a variable-span type.

Abstract: An extent-based storage architecture is implemented by a storage server. The storage server generates a new extent identifier for cloning a source extent identified by a source extent identifier and stored at a source data structure that includes a length value providing a length of the source extent, an offset value and a reference count value that provides a number of data containers that reference the source extent identifier. The storage server updates a data structure for a cloned version of the data container for storing the new extent identifier that points to the source extent identifier and includes an extent length value and offset value different from length value and the offset value of the source data structure.

Abstract: Information management is disclosed. A file output from an application to an operating system is intercepted before the file output arrives at the operating system. The file output is directed towards protected data. The intercepted file output is analyzed to determine whether a predetermined type of version of the protected data has been created. In the event it is determined that the predetermined type of version of the protected data has been created at least in part because the analyzed intercepted file output includes a modification to the protected data, the protected data automatically backed up, including by storing at least a portion of the file output as a backup version of the protected data. In the event it is determined that the predetermined type of version of the protected data has not been created, the protected data is not backed up.

Abstract: Techniques to clone a writeable data object in non-persistent memory are disclosed. The writeable data object is stored in a storage structure in non-persistent memory that corresponds to a portion of a persistent storage. The techniques enable cloning of the writeable data object without having to wait until the writeable data object is saved to the persistent storage and without needing to quiesce incoming operations (e.g., reads and writes) to the writeable data object.

Abstract: A method for efficiently handling partial write requests in a storage system includes allocating a new block of data for the new partial data, and allocating a record in an extent map to record the location of the new partial data block, the location of the old partial data block and the offset length for each data block. Data blocks can be repackaged in the background when system resources are available. A full, but misaligned write request is also efficiently handled by writing the new data to a newly allocated data block and allocating new records in an extent map to record information corresponding to two partial write operations.

Abstract: Overwriting part of compressed data without decompressing on-disk compressed data is includes by receiving a write request for a block of data in a compression group from a client, wherein the compression group comprises a group of data blocks that is compressed, wherein the block of data is uncompressed. The storage server partially overwrites the compression group, wherein the compression group remains compressed while the partial overwriting is performed. The storage server determines whether the partially overwritten compression group including the uncompressed block of data should be compressed. The storage server defers compression of the partially overwritten compression group if the partially overwritten compression group should not be compressed. The storage server compresses the partially overwritten compression group if the partially overwritten compression group should be compressed.