Abstract: A computer implemented method may include: storing key-value pairs in an index in persistent storage, where indirect nodes of the index include pointers, where each pointer identifies an index portion and includes a generation identifier for the identified index portion, where the index comprises a plurality of snapshots associated with a plurality of generations; receiving a request to read data of a particular snapshot of the index, wherein the particular snapshot is associated with a particular generation of the plurality of generations; in response to the request, performing a traversal starting from a particular root node associated with the particular generation; and providing the requested data based on the traversal.

Abstract: A computer implemented method may include: receiving write requests to add key-value pairs to an index; storing the key-value pairs in a buffer of an indirect node of the index; determining whether the buffer of the indirect node exceeds a threshold level; and in response to a determination that the buffer of the indirect node exceeds the threshold level, transferring the key-value pairs stored in the buffer of the indirect node to buffers of a plurality of child nodes, where each buffer of the plurality of child nodes is smaller than the buffer of the indirect node.

Abstract: A computer implemented method may include: storing key-value pairs in an index in persistent storage, where indirect nodes of the index include pointers, where each pointer identifies an index portion and includes a generation identifier for the identified index portion, where the index comprises a plurality of snapshots associated with a plurality of generations; receiving a request to read data of a particular snapshot of the index, wherein the particular snapshot is associated with a particular generation of the plurality of generations; in response to the request, performing a traversal starting from a particular root node associated with the particular generation; and providing the requested data based on the traversal.

Abstract: A computer implemented method may include: receiving a read request for a key-value pair in an index, wherein each indirect node of the index comprises a buffer and a Bloom filter, and wherein sizes of the Bloom filters vary across the levels according to a predefined function; responsive to a read request for the key-value pair, determining whether the Bloom filter of the indirect node indicates that the buffer of the indirect node includes the key-value pair; and responsive to a determination that the Bloom filter of the indirect node indicates that the buffer of the indirect node includes the key-value pair, searching the buffer of the indirect node for the key-value pair.

Abstract: A computer implemented method may include: receiving write requests to add key-value pairs to an index; storing the key-value pairs in a buffer of an indirect node of the index; determining whether the buffer of the indirect node exceeds a threshold level; and in response to a determination that the buffer of the indirect node exceeds the threshold level, transferring the key-value pairs stored in the buffer of the indirect node to buffers of a plurality of child nodes, where each buffer of the plurality of child nodes is smaller than the buffer of the indirect node.

Abstract: Examples herein relate to re-synchronizing data between an upstream volume and a downstream volume, including by communicating data to the downstream volume, from a start point to an end point of the upstream volume, maintaining a first cursor to indicate data, from the start point to the first cursor, that has been communicated to and committed to persistent storage of the downstream volume, and maintaining a second cursor to indicate data, from the first cursor to the second cursor, that has been communicated to the downstream volume and buffered without yet being confirmed as committed to persistent storage of the downstream volume.

Abstract: Examples described herein include receiving a first data write request from a host computing device, assigning a first identification to the first data write request, recording the first identification in in-flight data, transmitting the first data write request to a second storage array, receiving a first acknowledgement of the first data write request from the second storage array, and recording an indicator representing a group of acknowledgements. In some examples, the group of acknowledgements comprises the first acknowledgment.

Abstract: According to examples, a system may include an upstream volume controller having: a processor and a non-transitory machine-readable storage medium. The storage medium may include instructions executable by the processor to freeze an upstream volume, the upstream volume being in a replication set with a downstream volume, receive a snapshot creation request, create a snapshot of the upstream volume, and send one of a snapshot permit message or a snapshot abort message to a downstream volume processor. The instructions may also be executable by the processor to unfreeze the upstream volume responsive to at least one of the sending of the one of the snapshot permit message or the snapshot abort message or expiration of a timeout corresponding to a maximum time period during which the upstream volume is to remain frozen.

Abstract: According to examples, an apparatus may include a processor and a non-transitory machine-readable storage medium comprising instructions executable by the processor to assign a first object identifier and a data identifier to a first volume, the first object identifier being assigned exclusively to the first volume. The instructions may also be executable by the processor to identify an identifier of a second volume, determined whether the identifier of the second volume matches the data identifier, and based on a determination that the identifier of the second volume matches the data identifier of the first volume, configure a replication relationship between the first volume and the second volume.

Abstract: According to examples, an apparatus may include a processor that may store copies of full and incremental snapshots of a volume to an object-based storage system in a format that may include a volume object, a plurality of snapshot objects, and a plurality of data bucket objects. The volume object may point to each of the snapshot objects that each correspond to a snapshot created for the volume. Each of the snapshot objects may have a list of one or more of the data bucket objects for the corresponding snapshot, in which, for incremental snapshots, the corresponding snapshot object points to identifications of data bucket objects that represent changes made to the data in the volume since a prior snapshot and each data bucket object may have a list of a data block including data representing the snapshot corresponding to the snapshot object pointing to the data bucket object.

Abstract: According to examples, an apparatus may include a processor and a non-transitory machine-readable storage medium comprising instructions executable by the processor to assign a first object identifier and a data identifier to a first volume, the first object identifier being assigned exclusively to the first volume. The instructions may also be executable by the processor to identify an identifier of a second volume, determined whether the identifier of the second volume matches the data identifier, and based on a determination that the identifier of the second volume matches the data identifier of the first volume, configure a replication relationship between the first volume and the second volume.

Abstract: According to examples, a system may include an upstream volume controller having: a processor and a non-transitory machine-readable storage medium. The storage medium may include instructions executable by the processor to freeze an upstream volume, the upstream volume being in a replication set with a downstream volume, receive a snapshot creation request, create a snapshot of the upstream volume, and send one of a snapshot permit message or a snapshot abort message to a downstream volume processor. The instructions may also be executable by the processor to unfreeze the upstream volume responsive to at least one of the sending of the one of the snapshot permit message or the snapshot abort message or expiration of a timeout corresponding to a maximum time period during which the upstream volume is to remain frozen.

Abstract: Examples described herein include receiving a first data write request from a host computing device, assigning a first identification to the first data write request, recording the first identification in in-flight data, transmitting the first data write request to a second storage array, receiving a first acknowledgement of the first data write request from the second storage array, and recording an indicator representing a group of acknowledgements. In some examples, the group of acknowledgements comprises the first acknowledgment.

Abstract: Examples herein relate to re-synchronizing data between an upstream volume and a downstream volume, including by communicating data to the downstream volume, from a start point to an end point of the upstream volume, maintaining a first cursor to indicate data, from the start point to the first cursor, that has been communicated to and committed to persistent storage of the downstream volume, and maintaining a second cursor to indicate data, from the first cursor to the second cursor, that has been communicated to the downstream volume and buffered without yet being confirmed as committed to persistent storage of the downstream volume.

Abstract: A system can efficiently removes ranges of entries from a flat sorted data structure that represent stale fingerprints As part of fingerprint verification during deduplication, the system performs an attributes intersect range calculation (AIRC) procedure on the stale fingerprint data structure to compute a set of non-overlapping and latest consistency point (CP) ranges. During the AIRC procedure, an inode associated with a data container is selected and the FBN tuple of each deleted data block in the file is sorted in a predefined FBN order. The AIRC procedure then identifies the most recent fingerprint associated with a deleted data block. The set of non-overlapping and latest CP ranges is then used to remove stale fingerprints associated with that deleted block from the fingerprint database. A single pass through the fingerprint database identifies the set of non-overlapping and latest CP ranges, thereby improving efficiency of the storage system.

Abstract: Techniques for using a bloom filter in deduplication are described herein. A change log comprising a plurality of data blocks may be received. Values associated with the data blocks may be hashed and compared with a bloom filter. The comparison with the bloom filter identifies data blocks from the change log as unique data blocks or potential duplicate data blocks. A bit by bit comparison of the potential duplicate data blocks and previous data blocks may be performed to determine if any of the potential duplicate data blocks are identical to any of previous data blocks. Such data blocks of the change log that are identified as being identical may be deduplicated.

Abstract: A method performed in a network storage system, the method including receiving a plurality of data blocks at a secondary storage subsystem from a primary storage subsystem, generating a first log that includes a first plurality of entries, one entry for each of the data blocks, in which each entry of the first plurality of entries includes a name for a respective data block and a fingerprint of the respective data block, receiving metadata at the secondary storage subsystem from the primary storage subsystem, the metadata describing relationships between the plurality of blocks and a plurality of files, generating a second log that includes a second plurality of entries, and merging the first log with the second log to generate a change log.

Abstract: A storage server is coupled to a storage device that stores blocks of data, and generates a fingerprint for each data block stored on the storage device. The storage server creates a fingerprints datastore that is divided into a primary datastore and a secondary datastore. The primary datastore comprises a single entry for each unique fingerprint and the secondary datastore comprises an entry having an identical fingerprint as an entry in the primary datastore. The storage server merges entries in a changelog with the entries in the primary datastore to identify duplicate data blocks in the storage device and frees the identified duplicate data blocks in the storage device. The storage server stores the entries that correspond to the freed data blocks to a third datastore and overwrites the primary datastore with the entries from the merged data that correspond to the unique fingerprints to create an updated primary datastore.

Abstract: A system and method efficiently removes ranges of entries from a flat sorted data structure, such as a fingerprint database, of a storage system. The ranges of entries represent fingerprints that have become stale, i.e., are not representative of current states of corresponding blocks in the file system, due to various file system operations such as, e.g., deletion of a data block without overwriting its contents. A deduplication module of a file system executing on the storage system performs a fingerprint verification procedure to remove the stale fingerprints from the fingerprint database. As part of the fingerprint verification procedure, the deduplication module performs an attributes intersect range calculation (AIRC) procedure on the stale fingerprint data structure to compute a set of non-overlapping and latest consistency point (CP) ranges. During the AIRC procedure, an inode associated with a data container, e.g.

Abstract: A method performed in a network storage system, the method including receiving a plurality of data blocks at a secondary storage subsystem from a primary storage subsystem, generating a first log that includes a first plurality of entries, one entry for each of the data blocks, in which each entry of the first plurality of entries includes a name for a respective data block and a fingerprint of the respective data block, receiving metadata at the secondary storage subsystem from the primary storage subsystem, the metadata describing relationships between the plurality of blocks and a plurality of files, generating a second log that includes a second plurality of entries, and merging the first log with the second log to generate a change log.