Abstract: Components of a distributed data object are synchronized using streamlined tracking metadata. A target component of the distributed data object is detected as it becomes available and stale. A source component that is up-to-date and that mirrors the address space of the detected target component is identified. A set of mapped address ranges and a set of unmapped address ranges of the identified source component are obtained. A mapped address range of the target component that corresponds with an unmapped address range of the source component is identified. The identified mapped address range of the target component is then synchronized with the corresponding unmapped address range of the source component. Thus, unmapped address ranges are synchronized without using tracking metadata of the source component.

Abstract: A system manages a log-structured file system (LFS) by: receiving an input/output (I/O) operation for the LFS, the I/O operation prompting a key to be added to a first node of a tree metadata structure, the tree mapping addresses in a first address space to addresses in a second address space; determining that addition of the key to the first node would exceed a maximum number of keys allowed in the first node; adding a second node to the tree based on the determining, the second node containing the key; moving a quantity of keys from the first node to the second node such that a total number of keys resulting in the second node is less than half of the maximum number of keys, minus one, configured to be stored in nodes of the LFS; and writing updates to the tree metadata structure within the LFS.

Abstract: Workload-responsive segment cleaning of log structured filesystems (LFSs) is disclosed. When multiple independent LFSs overlap on spanning a set of storage disks (including non-volatile memory express storage), a global segment cleaner (GSC) for each disk coordinates the cleaning rates of the local segment cleaners (LSCs) for each LFS having a presence on that disk. LFSs send usage information to relevant GSCs that select usage thresholds to trigger cleaning and cleaning rates. When capacity fullness (e.g., segments having at least one used block) meets a threshold, segment cleaning is performed at a rate based on capacity fullness and an equilibrium cleaning rate. Cleaning rates speed up when storage is more full, to provide capacity for burst writing events, but slow down when less full, to reduce overhead burden. LFSs clean at the highest rate identified for every GSC's usage threshold an LFS meets.

Abstract: Workload-responsive distributed segment cleaning of log structured filesystems (LFSs) is disclosed. When multiple independent LFSs overlap on spanning a set of storage disks (including non-volatile memory express storage), a global segment cleaner (GSC) for each disk coordinates the cleaning rates of the local segment cleaners (LSCs) for each LFS having a presence on that disk. LFSs send usage information to relevant GSCs that select usage thresholds to trigger cleaning and cleaning rates. When capacity fullness (e.g., segments having at least one used block) meets a threshold, segment cleaning is performed at a rate based on capacity fullness and an equilibrium cleaning rate. Cleaning rates speed up when storage is more full, to provide capacity for burst writing events, but slow down when less full, to reduce overhead burden. LFSs clean at the highest rate identified for every GSC's usage threshold an LFS meets.

Abstract: Aspects of the disclosure perform network scheduling in a distributed storage system. Example operations include: determining a network congestion condition at a first host; based on the network congestion condition, determining a packet delay time; based on a first data packet belonging to a first traffic class of a plurality of traffic classes, delaying transmitting the first data packet, from the first host across a network to a second host, by the packet delay time; and based on a second data packet belonging to a second traffic class, transmitting the second data packet from the first host to the second host without a delay. In some examples, the first traffic class comprises resync input/output operations (I/Os) and the second traffic class comprises non-resync traffic I/Os. Some examples delay packets differently, based on the destination host. Some examples adjust delays to drive the network congestion condition toward a target.

Abstract: An example method of resynchronizing a first replica of an object and a second replica of an object in an object storage system, includes: determining, by storage software in response to the second replica transitioning from failed to available, a stale sequence number for the second replica, the storage software having associated the stale sequence number with the second replica when the second replica failed; querying, by the storage software, block-level metadata for the object using the stale sequence number, the block-level metadata relating logical blocks of the object with sequence numbers for operations on the object; determining, by the software as a result of the querying, a set of the logical blocks each related to a sequence number being the same or after the stale sequence number; and copying, by the storage software, data of the set of logical blocks from the first replica to the second replica.

Abstract: An example method of managing a log-structured file system (LFS) on a storage device includes: receiving, at storage software executing on a host, an operation that overwrites a data block, the data block included in a segment of the LFS; determining from first metadata stored on the storage device, a change in utilization of the segment from a first utilization value to a second utilization value; modifying second metadata stored on the storage device to change a relation between the segment and a first bucket to be a relation between the segment and a second bucket, the first utilization value included in a range of the first bucket and the second utilization value included in a range of the second bucket; and executing a garbage collection process for the LFS that uses the second metadata to identify for garbage collection a set of segments in the second bucket.

Abstract: Example methods and systems for accessing data in a log-structured file system having a plurality of snapshots of storage objects backed by a first-level copy-on-write (COW) B+ tree data structure and a plurality of second-level B+ tree data structures have been disclosed. One example method includes obtaining a first first-level mapping associated with a first snapshot from the plurality of snapshots based on a first logical block address, wherein each of the plurality of snapshots corresponds to each of the plurality of second-level B+ tree data structures, identifying a first second-level B+ tree data structure corresponding to one of the plurality of snapshots based on the first first-level mapping, obtaining a first second-level mapping based on the first logical block address in the first second-level B+ tree data structure, obtaining a first physical block address based on the first second-level mapping, and accessing data at the first physical block address.

Abstract: The disclosure herein describes managing write requests in a multi-layer data store using prefetch threads. Prefetch requests received at the multi-layer data store are added to a prefetch request queue. A first prefetch thread obtains a first prefetch request and a second prefetch thread obtains a second prefetch request from the prefetch request queue. A memory cache is updated with first metadata of the first prefetch request by the first prefetch thread and with second metadata of the second prefetch request by the second prefetch thread, in parallel. After the memory cache is updated, a data bank in memory reaches a flush threshold and data in the data bank is written to a second data store, wherein the first metadata and second metadata in the memory cache are used to update a metadata structure with respect to the data written to the second data store.

Abstract: System and method for managing different classes of storage input/output (I/O) requests for a two-phase commit operation in a distributed storage system assigns reserved log sequence values to each of storage I/O requests of a first class, which are added to a two-phase commit queue. The reserved log sequence values of the storage I/O requests of the first class in the two-phase commit queue are assigned to some of the storage I/O requests of the second class, which are added to the two-phase commit queue.

Abstract: The disclosure herein describes performing resynchronization (“resync”) jobs in a distributed storage system based on a parallelism policy. A resync job is obtained from a queue and input/output (I/O) resources that will be used during execution of the resync job are identified. Available bandwidth slots of each I/O resource of the identified I/O resources are determined. The parallelism policy is applied to the identified I/O resources and the available bandwidth slots. Based on the application of the parallelism policy, a bottleneck resource of the I/O resources is determined and a parallel I/O value is calculated based on the available bandwidth slots of the bottleneck resource, wherein the parallel I/O value indicates a quantity of I/O tasks that can be performed in parallel. The resync job is executed using the I/O resources, the execution of the resync job including performance of I/O tasks in parallel based on the parallel I/O value.

Abstract: A method for efficient write-back for journal truncation is provided. A method includes maintaining a journal in a memory of a computing system including a plurality of records. Each record indicates a transaction associated with one or more pages in an ordered data structure and maintaining a dirty list including an entry for each page indicated by a record in the journal. Each entry in the dirty list includes a respective first log sequence number (LSN) associated with a least recent record of the plurality of records that indicates the page and a respective second LSN associated with a most recent record of the plurality of records that indicates the page. The method includes determining to truncate the journal. The method includes identifying one or more records, of the plurality of records, from the journal to write back to a disk, where the identifying is based on the dirty list.

Abstract: A method for efficient journal truncation is provided. A method for journal truncation includes maintaining a journal in a memory of a computing system including a plurality of records. Each record indicates a transaction in an ordered data structure. The method includes maintaining a truncation queue in the memory including one or more entries. Each entry in the truncation queue includes a physical on-disk offset associated with a different record of the plurality of records. The method includes determining to truncate the journal and truncating records, of the plurality of records, from the journal starting from a beginning record in the journal up to the record with the physical on-disk offset associated a least recent entry of the one or more entries in the truncation queue, where the truncating includes removing the records from the memory.

Abstract: A method for modifying key-value pairs of a B+ tree is provided. The method receives a request to modify a particular key-value pair. Each node of the tree has a modification number. The method traverses a path on the tree from the root node toward the particular node. The traversing includes upon reaching a parent node of the path, acquiring a shared lock on both the parent node and a child node one level below the parent node. Upon determining that the child node is the particular node, the method stores the modification number of the particular node, releases the shared lock on the particular node, compares a current modification number of the node with its stored number, and acquires an exclusive lock on the node if the numbers are the same. The method increments the current modification number of the node and modifies it while in the exclusive lock.

Abstract: Components of a distributed data object are synchronized using streamlined tracking metadata. A target component of the distributed data object is detected as it becomes available and stale. A source component that is up-to-date and that mirrors the address space of the detected target component is identified. A set of mapped address ranges and a set of unmapped address ranges of the identified source component are obtained. A mapped address range of the target component that corresponds with an unmapped address range of the source component is identified. The identified mapped address range of the target component is then synchronized with the corresponding unmapped address range of the source component. Thus, unmapped address ranges are synchronized without using tracking metadata of the source component.

Abstract: The disclosure describes growing a data cache using a background hash bucket growth process. A first memory portion is allocated to the data buffer of the data cache and a second memory portion is allocated to the metadata buffer of the data cache based on the cache growth instruction. The quantity of hash buckets in the hash bucket buffer is increased and the background hash bucket growth process is initiated, wherein the process is configured to rehash hash bucket entries of the hash bucket buffer in the increased quantity of hash buckets. A data entry is stored in the data buffer using the allocated first memory portion of the data cache and metadata associated with the data entry is stored using the allocated second memory portion of the metadata buffer, wherein a hash bucket entry associated with the data entry is stored in the increased quantity of hash buckets.

Abstract: Components of a distributed data object are synchronized using streamlined tracking metadata. A target component of the distributed data object is detected as it becomes available and stale. A source component that is up-to-date and that mirrors the address space of the detected target component is identified. A set of mapped address ranges and a set of unmapped address ranges of the identified source component are obtained. A mapped address range of the target component that corresponds with an unmapped address range of the source component is identified. The identified mapped address range of the target component is then synchronized with the corresponding unmapped address range of the source component. Thus, unmapped address ranges are synchronized without using tracking metadata of the source component.

Abstract: A method for efficient distributed object manager (DOM) owner migration in a virtual storage area network (VSAN) is provided. Embodiments include determining to migrate a DOM owner object from a source DOM owner server to a destination DOM owner server. Embodiments include determining that an input or output (I/O) operation affecting the DOM owner object is in progress. Embodiments include, prior to updating the DOM owner object based on the I/O operation, transferring the DOM owner object from the source DOM owner server to the destination DOM owner server. Embodiments include completing execution of the I/O operation on the destination DOM owner server. Embodiments include updating the DOM owner object on the destination DOM owner server based on the I/O operation.

Abstract: A method for dynamic distributed object manager (DOM) owner migration in a virtual storage area network (VSAN) is provided. Embodiments include determining estimated processor load amounts for a plurality of DOM owner servers based on dispatch latency. Embodiments include determining that a first DOM owner server of the plurality of DOM owner servers has a highest estimated processor load amount of the estimated processor load amounts. Embodiments include determining that a second DOM owner server has a lowest estimated processor load amount of the estimated processor load amounts. Embodiments include determining to migrate a DOM owner object from the first DOM owner server to the second DOM owner server based on a dispatch latency value corresponding to the DOM owner object. Embodiments include migrating the DOM owner object from the first DOM owner server to the second DOM owner server.

Abstract: System and method for managing different classes of storage input/output (I/O) requests for a two-phase commit operation in a distributed storage system assigns reserved log sequence values to each of storage I/O requests of a first class, which are added to a two-phase commit queue. The reserved log sequence values of the storage I/O requests of the first class in the two-phase commit queue are assigned to some of the storage I/O requests of the second class, which are added to the two-phase commit queue.