Abstract: The disclosure herein describes deduplicating data chunks using chunk objects. A batch of data chunks is obtained from an original data object and a hash value is calculated for each data chunk. A first duplicate data chunk is identified using the hash value and a hash map. A chunk logical block address (LBA) of a chunk object is assigned to the duplicate data chunk. Payload data of the duplicate data chunk is migrated from the original data object to the chunk object, and a chunk map is updated to map the chunk LBA to a physical sector address (PSA) of the migrated payload data on the chunk object. A hash entry is updated to map to the chunk object and the chunk LBA. An address map of the original data object is updated to map an LBA of the duplicate data chunk to the chunk object and the chunk LBA.

Abstract: A system manages a log-structured file system (LFS) by: receiving an input/output (I/O) operation for the LFS, the I/O operation prompting a key to be added to a first node of a tree metadata structure, the tree mapping addresses in a first address space to addresses in a second address space; determining that addition of the key to the first node would exceed a maximum number of keys allowed in the first node; adding a second node to the tree based on the determining, the second node containing the key; moving a quantity of keys from the first node to the second node such that a total number of keys resulting in the second node is less than half of the maximum number of keys, minus one, configured to be stored in nodes of the LFS; and writing updates to the tree metadata structure within the LFS.

Abstract: Linked clone read performance when retrieving data from a clone is improved at least by aggregating block mapping metadata efficiently. Primary metadata for a child clone maps a logical block address (LBA) for data in a data region of the child clone to a physical sector address (PSA) for data in the data region of the child clone. At least a portion of primary metadata for a parent clone of the child clone is copied into archival metadata for the child clone. In response to a read request, data is returned from the child clone, parent clone, or another ancestor of the child clone based on whether or not a read request LBA is within the primary metadata for the child clone, or within the archival metadata.

Abstract: Aspects of the disclosure perform network scheduling in a distributed storage system. Example operations include: determining a network congestion condition at a first host; based on the network congestion condition, determining a packet delay time; based on a first data packet belonging to a first traffic class of a plurality of traffic classes, delaying transmitting the first data packet, from the first host across a network to a second host, by the packet delay time; and based on a second data packet belonging to a second traffic class, transmitting the second data packet from the first host to the second host without a delay. In some examples, the first traffic class comprises resync input/output operations (I/Os) and the second traffic class comprises non-resync traffic I/Os. Some examples delay packets differently, based on the destination host. Some examples adjust delays to drive the network congestion condition toward a target.

Abstract: Example methods and systems for accessing data in a log-structured file system having a plurality of snapshots of storage objects backed by a first-level copy-on-write (COW) B+ tree data structure and a plurality of second-level B+ tree data structures have been disclosed. One example method includes obtaining a first first-level mapping associated with a first snapshot from the plurality of snapshots based on a first logical block address, wherein each of the plurality of snapshots corresponds to each of the plurality of second-level B+ tree data structures, identifying a first second-level B+ tree data structure corresponding to one of the plurality of snapshots based on the first first-level mapping, obtaining a first second-level mapping based on the first logical block address in the first second-level B+ tree data structure, obtaining a first physical block address based on the first second-level mapping, and accessing data at the first physical block address.

Abstract: The disclosure herein describes deduplicating data chunks using chunk objects. A batch of data chunks is obtained from an original data object and a hash value is calculated for each data chunk. A first duplicate data chunk is identified using the hash value and a hash map. A chunk logical block address (LBA) of a chunk object is assigned to the duplicate data chunk. Payload data of the duplicate data chunk is migrated from the original data object to the chunk object, and a chunk map is updated to map the chunk LBA to a physical sector address (PSA) of the migrated payload data on the chunk object. A hash entry is updated to map to the chunk object and the chunk LBA. An address map of the original data object is updated to map an LBA of the duplicate data chunk to the chunk object and the chunk LBA.

Abstract: Chunks of data are identified and deduplication is performed on the chunks of data using associated cyclic redundancy check (CRC) values. A plurality of CRC values is obtained that is associated with consecutive data blocks stored in a payload data store. Cut point CRC values are identified in the plurality of CRC values and CRC chunks are identified based on those cut point CRC values, wherein each CRC chunk is bounded by two consecutive cut point CRC values. A CRC chunk hash value is generated for each CRC chunk. A pair of duplicate CRC chunks is identified using the CRC chunk hash values and a deduplication operation is performed in association with the identified pair of duplicate CRC chunks. Using existing CRC values during the identification of chunk cut points reduces the computing resource costs associated with performing that process using the data blocks.

Abstract: System and method for managing different classes of storage input/output (I/O) requests for a two-phase commit operation in a distributed storage system assigns reserved log sequence values to each of storage I/O requests of a first class, which are added to a two-phase commit queue. The reserved log sequence values of the storage I/O requests of the first class in the two-phase commit queue are assigned to some of the storage I/O requests of the second class, which are added to the two-phase commit queue.

Abstract: The disclosure herein describes performing resynchronization (“resync”) jobs in a distributed storage system based on a parallelism policy. A resync job is obtained from a queue and input/output (I/O) resources that will be used during execution of the resync job are identified. Available bandwidth slots of each I/O resource of the identified I/O resources are determined. The parallelism policy is applied to the identified I/O resources and the available bandwidth slots. Based on the application of the parallelism policy, a bottleneck resource of the I/O resources is determined and a parallel I/O value is calculated based on the available bandwidth slots of the bottleneck resource, wherein the parallel I/O value indicates a quantity of I/O tasks that can be performed in parallel. The resync job is executed using the I/O resources, the execution of the resync job including performance of I/O tasks in parallel based on the parallel I/O value.

Abstract: Solutions for rapid ransomware detection and recovery include: receiving a first set of in-memory changed data blocks; identifying, within the first set of in-memory changed data blocks, a second set of in-memory changed data blocks addressed for storage within a file index for a virtual machine (VM) disk; determining, relative to a change history of the file index, an anomalous condition; based on at least determining the anomalous condition, identifying a third set of blocks within the file index that are changed between two versions of the VM disk; determining that changes in the third set of blocks indicate ransomware; and based on at least determining that changes in the third set of blocks indicate ransomware, generating an alert. Machine learning (ML) models may perform anomaly/ransomware detection. Remediation activities may include disk restoration storing the VM memory.

Abstract: Linked clone read performance (e.g., retrieving data) is improved at least by minimizing the number of input/output (I/O) operations. For a child clone, a local logical extent and an inherited logical extent are generated. The local logical extent comprises a logical block address (LBA) for data in a data region of the child clone and a physical sector address (PSA) corresponding to the LBA for the data in the data region of the child clone. The inherited logical extent spans logical extents that are accessible to the child clone. The inherited logical extent comprises an LBA for data in a data region of an ancestor of the child clone and a corresponding identifier (ID) of the ancestor. Data for an LBA in a read request may be rapidly found in the child clone (local logical extent) or an ancestor (inherited logical extent).

Abstract: System and method for converting a storage object in a redo-log snapshot format to a single-container snapshot format in a distributed storage system uses a temporary snapshot object, which is created by taking a snapshot of the storage object, and an anchor object, which points to a root object of the storage object. For each object chain of the storage object, each selected object is processed for format conversion. For each selected object, difference data between the selected object and a parent object of the selected object is written to the anchor object, a child snapshot of the anchor object is created in the single-container snapshot format, and the anchor object is updated to point to the selected object. The data of the running point object of the storage object is then copied to the anchor object, and each processed object and the temporary snapshot object are removed.

Abstract: A method for implementing native snapshot capabilities on non-native snapshots includes generating a virtual disk container having both native snapshot functionality and functionality of the non-native snapshot, including redo-log snapshot functionality. Redo-log parent disk chains may be preserved, along with subsequent native snapshot data, in virtual disk container objects. A virtual root node of a copy-on-write data structure of a virtual disk container object enables snapshotting and traversal between a redo-log disk chain and a native snapshot disk chain. Throughout backup and reversion operations, the virtual disk container object may be reparented as necessary, and a constant running point may be maintained for the virtual disk container object.

Abstract: System and method for managing different classes of storage input/output (I/O) requests for a two-phase commit operation in a distributed storage system assigns reserved log sequence values to each of storage I/O requests of a first class, which are added to a two-phase commit queue. The reserved log sequence values of the storage I/O requests of the first class in the two-phase commit queue are assigned to some of the storage I/O requests of the second class, which are added to the two-phase commit queue.

Abstract: The disclosure herein describes enhancing data durability of a base component using a delta component. A delta component is generated based on the base component becoming unavailable. The delta component is configured to include unwritten storage space with an address space matching the base component and a tracking bitmap associated with data blocks of the address space of the delta component. Write operations targeted for the base component are routed to the delta component. Based on the routed write operations, bits associated with data blocks affected by the write operations are changed in the tracking bitmap. Based on the base component becoming available, data blocks affected by routed write operations are identified based on the tracking bitmap and the identified data blocks are synchronized from the delta component to the base component. The delta component is then removed.

Abstract: An example method of upgrading a distributed storage object from a first version to a second version includes: querying metadata of a first component configured according to the first version of the distributed storage object, the metadata defining extents of data on a disk group of the first component; populating, for a second component configured according to the second version of the distributed storage object, logical and middle maps based on the metadata such that initial entries in the logical map point to initial entries in the middle map, and the initial entries in the middle map point to physical addresses of the disk group of the first component; and reading the data from the disk group of the first component and writing the data to a disk group of the second component while updating the initial entries in the middle map.

Abstract: A method for reverse range lookup in an ordered data structure of keys, wherein each key comprises a logical block address (LBA) and a snapshot identifier (ID) of one of one or more snapshots in a snapshot hierarchy, is provided. The keys in the ordered data structure are in an order from smallest to largest LBA, wherein in the order, keys having a same LBA are ordered from smallest to largest snapshot ID. The method includes determining a range of LBAs and traversing the ordered data structure in reverse order until a key is found that: has an LBA and a snapshot ID that is less than or equal to the last LBA and the largest snapshot ID, respectively. The method further includes adding an extent corresponding to the located key to an extent result list if the snapshot ID of the located key is not within a deny list.

Abstract: A method for resumeable snapshot deletion is provided. A method for deletion of nodes maintained in an ordered data structure for a first snapshot includes processing the nodes maintained in the ordered data structure according to a defined order, setting a node path cursor with a pointer to a node and an indication of the deletion of the node; storing the node path cursor in a persistent storage; and during processing of the nodes: detecting a failure; after the failure, checking the pointer of the node path cursor; and resuming processing of the nodes starting from the first node indicated by the pointer.

Abstract: The present disclosure is related to methods, systems, and machine-readable media for managing extent sharing between snapshots using mapping addresses. A first mapping address can be assigned to a first extent responsive to a request to write the first extent. A second mapping address can be assigned to a second extent responsive to a request to write the second extent. A snapshot can be created. A snapshot mapping address, that is monotonically increased from the second mapping address, can be assigned to the snapshot. A third mapping address, that is monotonically increased from the second mapping address, can be assigned to a third extent of the snapshot responsive to a request to write the third extent.

Abstract: A method for deleting one or more snapshots using micro-batch processing is provided. The method includes receiving a request to delete the one or more snapshots, identifying one or more middle map extents exclusively owned by the one or more snapshots requested to be deleted, wherein metadata for the one or more snapshots is stored in one or more logical maps having logical map extents mapping logical block addresses (LBAs) to middle block addresses (MBAs) and a middle map having middle map extents mapping MBAs to physical block addresses (PBAs) of physical locations where data blocks are written, adding MBAs of the identified one or more middle map extents in a batch, determining a first micro-batch including a first subset of the MBAs in the batch, the first subset of MBAs being MBAs less than a first upper bound MBA, and using a first transaction to delete the middle map extents corresponding to the first subset of MBAs included in the first micro-batch.