Abstract: An example method of resynchronizing a first replica of an object and a second replica of an object in an object storage system, includes: determining, by storage software in response to the second replica transitioning from failed to available, a stale sequence number for the second replica, the storage software having associated the stale sequence number with the second replica when the second replica failed; querying, by the storage software, block-level metadata for the object using the stale sequence number, the block-level metadata relating logical blocks of the object with sequence numbers for operations on the object; determining, by the software as a result of the querying, a set of the logical blocks each related to a sequence number being the same or after the stale sequence number; and copying, by the storage software, data of the set of logical blocks from the first replica to the second replica.

Abstract: The disclosure herein describes managing write requests in a multi-layer data store using prefetch threads. Prefetch requests received at the multi-layer data store are added to a prefetch request queue. A first prefetch thread obtains a first prefetch request and a second prefetch thread obtains a second prefetch request from the prefetch request queue. A memory cache is updated with first metadata of the first prefetch request by the first prefetch thread and with second metadata of the second prefetch request by the second prefetch thread, in parallel. After the memory cache is updated, a data bank in memory reaches a flush threshold and data in the data bank is written to a second data store, wherein the first metadata and second metadata in the memory cache are used to update a metadata structure with respect to the data written to the second data store.

Abstract: The disclosure herein describes performing resynchronization (“resync”) jobs in a distributed storage system based on a parallelism policy. A resync job is obtained from a queue and input/output (I/O) resources that will be used during execution of the resync job are identified. Available bandwidth slots of each I/O resource of the identified I/O resources are determined. The parallelism policy is applied to the identified I/O resources and the available bandwidth slots. Based on the application of the parallelism policy, a bottleneck resource of the I/O resources is determined and a parallel I/O value is calculated based on the available bandwidth slots of the bottleneck resource, wherein the parallel I/O value indicates a quantity of I/O tasks that can be performed in parallel. The resync job is executed using the I/O resources, the execution of the resync job including performance of I/O tasks in parallel based on the parallel I/O value.

Abstract: The disclosure herein describes enhancing data durability of a base component using a delta component. A delta component is generated based on the base component becoming unavailable. The delta component is configured to include unwritten storage space with an address space matching the base component and a tracking bitmap associated with data blocks of the address space of the delta component. Write operations targeted for the base component are routed to the delta component. Based on the routed write operations, bits associated with data blocks affected by the write operations are changed in the tracking bitmap. Based on the base component becoming available, data blocks affected by routed write operations are identified based on the tracking bitmap and the identified data blocks are synchronized from the delta component to the base component. The delta component is then removed.

Abstract: Systems and methods for inspection and repair of VSAN object metadata. A user-space indirection layer is maintained to map logical addresses of VSAN objects to physical memory addresses of their metadata. Commands may then be sent from the user space to distributed object manager (DOM) clients, with the physical addresses of metadata of objects to be inspected. DOM owners thus may bypass their own indirection layers to retrieve object metadata directly from received user commands. Retrieved information is then used to reconstruct and repair object metadata. Repaired metadata may be written back to the VSAN by transmitting a write request containing the physical address at which the repaired metadata is to be written. DOM owners may be placed in a specified mode in which received I/O instructions are ignored unless they are designated as being for metadata repair purposes, such as by including a physical address.

Abstract: System and method for managing distributed storage objects for host unavailability in a distributed storage system uses at least one of a crash indicator in a specific on-disk block and a paused object indicator for a distributed storage object to determine whether to perform data recovery for the distributed storage object. When the crash indicator is set or the paused object indicator implies that the distributed storage object is a paused object, the distributed storage object is left as a paused object without perform the data recovery for the distributed storage object. When the crash indicator is unset and the paused object indicator implies that the distributed storage object is not a paused object, the data recovery for the distributed storage object is performed.

Abstract: Hybrid synchronization using a shadow component includes detecting a first component of a plurality of mirrored components of a distributed data object becoming unavailable. The mirrored components include a delta component (a special shadow component) and a regular mirror (shadow) component. The delta component indicates a shorter history of changes to data blocks of a log-structured file system (LFS) than is indicated by the regular mirror component. During the unavailability of the first component, at least one write I/O is committed by the delta component. The commit is tracked by the delta component in a first tracking bitmap associated with the delta component. Based at least on detecting the first component becoming available, the first component is synchronized with data from the delta component, based at least on changed data blocks indicated in the first tracking bitmap.

Abstract: The present disclosure is related to methods, systems, and machine-readable media for log-structured file system management operations. An aggregate amount of over-provisioned computing resources assigned to a plurality of log-structured file system (LFS) objects that are allocated for performance of memory management operations associated with a virtual storage area network (vSAN) can be determined. A subset of LFS objects that are candidates for performance of a particular memory management operation based on an amount of over-provisioned computing resources consumed by one or more LFS objects among the plurality of LFS objects exceeding a resource consumption threshold associated with the determined amount of over-provisioned computing resources assigned to the plurality of LFS objects can be selected. The particular memory management operation using one or more of the candidate LFS objects can be performed.

Abstract: Embodiments of the disclosure provide techniques for partitioning a resource object into multiple resource components of a cluster of host computer nodes in a distributed resources system. The distributed resources system translates high-level policy requirements into a resource configuration that the system accommodates. The system determines an allocation based on the policy requirements and identifies resource configurations that are available. Upon selecting a resource configuration, the distributed resources system assigns the allocation and associated values to the selected configuration and publishes the new configuration to other host computer nodes in the cluster.

Abstract: A method for compressing is provided. The method including receiving a block of data to store on at least one physical disk; determining whether to store the data in a data log as uncompressed or compressed data based on a determined size of resulting compressed data. When the method determines to store the data as compressed, compressing the data and storing the compressed data in at least one sector in the data log. Otherwise, the method stores the data, uncompressed, in a plurality of sectors in the data log. The method generates a one or more state bits indicating (i) whether the data is stored as uncompressed or compressed, and (ii) if the data is stored as compressed, a size of the compressed data. The method then stores the one or more state bits in an entry of a logical map table associated with an LBA that corresponds to the data block.

Abstract: Hybrid synchronization using a shadow component includes detecting a first component of a plurality of mirrored components of a distributed data object becoming unavailable. The mirrored components include a delta component (a special shadow component) and a regular mirror (shadow) component. The delta component indicates a shorter history of changes to data blocks of a log-structured file system (LFS) than is indicated by the regular mirror component. During the unavailability of the first component, at least one write I/O is committed by the delta component. The commit is tracked by the delta component in a first tracking bitmap associated with the delta component. Based at least on detecting the first component becoming available, the first component is synchronized with data from the delta component, based at least on changed data blocks indicated in the first tracking bitmap.

Abstract: The disclosure herein describes tracking changes to a stale component using a synchronization bitmap. A first component of a plurality of mirrored components of the distributed data object becomes available from an unavailable state, and a stale log sequence number (LSN) and a last committed LSN are identified. A synchronization bitmap of the first component associated with a range of LSNs (e.g., from the stale LSN to the last committed LSN) is created and configured to track changes to data blocks of the first component. A second component is identified based on the second component including a tracking bitmap associated with an LSN that matches the stale LSN of the first component. The first component is synchronized with data from the second component based on, wherein the synchronizing includes updating the synchronization bitmap to track changes made to data blocks of the first component.

Abstract: The disclosure herein describes performing resynchronization (“resync”) jobs in a distributed storage system based on a parallelism policy. A resync job is obtained from a queue and input/output (I/O) resources that will be used during execution of the resync job are identified. Available bandwidth slots of each I/O resource of the identified I/O resources are determined. The parallelism policy is applied to the identified I/O resources and the available bandwidth slots. Based on the application of the parallelism policy, a bottleneck resource of the I/O resources is determined and a parallel I/O value is calculated based on the available bandwidth slots of the bottleneck resource, wherein the parallel I/O value indicates a quantity of I/O tasks that can be performed in parallel. The resync job is executed using the I/O resources, the execution of the resync job including performance of I/O tasks in parallel based on the parallel I/O value.

Abstract: A method for encrypting data in one or more data blocks is provided. The method receives a first data block to be written to a physical storage that includes one or more physical disks. The method applies a first random tweak to data indicative of the first data block to generate a first encrypted data block, and writes the first encrypted data block and the first random tweak to a first physical block of the physical storage. The method receives a second data block to be written to the physical storage. The method then applies a second random tweak, different than the first random tweak, to data indicative of the second data block to generate a second encrypted data block, and writes the second encrypted data block and the second random tweak to a second physical block of the physical storage.

Abstract: Techniques for efficiently storing client data blocks on a distributed-computing system are provided. The system includes a fast performance tier and a large capacity tier. The capacity tier stores the client data blocks in erasure encoded data stripes. The performance tier stores logical map data including an address map indicating a correspondence between logical addresses associated with a first layer of the system and physical addresses associated with a second layer. A method includes receiving a request to include additional client data blocks in the client blocks. The request indicates logical addresses for additional blocks. Corresponding physical addresses for additional block are determined. Each additional block is stored at the physical address. Additional logical map data is stored in the performance tier. Storing the additional logical map data includes updating the address map to indicate the correspondence between the logical addresses and the physical addresses for the additional blocks.

Abstract: The disclosure herein describes performing resynchronization (“resync”) jobs in a distributed storage system based on a parallelism policy. A resync job is obtained from a queue and input/output (I/O) resources that will be used during execution of the resync job are identified. Available bandwidth slots of each I/O resource of the identified I/O resources are determined. The parallelism policy is applied to the identified I/O resources and the available bandwidth slots. Based on the application of the parallelism policy, a bottleneck resource of the I/O resources is determined and a parallel I/O value is calculated based on the available bandwidth slots of the bottleneck resource, wherein the parallel I/O value indicates a quantity of I/O tasks that can be performed in parallel. The resync job is executed using the I/O resources, the execution of the resync job including performance of I/O tasks in parallel based on the parallel I/O value.

Abstract: Techniques for issuing efficient writes to an erasure coded storage object in a distributed storage system via adaptive logging are provided. In one set of embodiments, a node of the system can receive a write request for updating one or more logical data blocks of the storage object and determine whether a size of the one or more logical data blocks meets or exceeds a threshold size. Upon determining that the size of the one or more logical data blocks meets or exceeds the threshold size, the node can allocate a segment in a capacity object of the storage object, write the one or more logical data blocks via a full stripe write to the segment, and write metadata for the one or more logical data blocks to a log record in a log of a metadata object of the storage object. The metadata written to the log record can include mappings between logical block addresses (LBAs) of the one or more logical data blocks and physical block addresses (PBAs) where the one or more logical data blocks reside in the segment.

Abstract: Techniques for enabling deduplication-aware load balancing in a distributed storage system are provided. In one set of embodiments, a node of the distributed storage system can receive an I/O (Input/Output) request pertaining to a data block of a storage object stored on a local storage component of the node. The node can further determine whether the I/O request requires insertion of a new entry into a deduplication hash table associated with the local storage component or deletion of an existing entry from the deduplication hash table. If the I/O request requires insertion of a new hash table entry, the node can add an identifier of the data block into a probabilistic data structure associated with the local storage component, where the probabilistic data structure is configured to maintain information regarding distinct data blocks that are likely present in the local storage component.

Abstract: A method for compressing is provided. The method including receiving a block of data to store on at least one physical disk; determining whether to store the data in a data log as uncompressed or compressed data based on a determined size of resulting compressed data. When the method determines to store the data as compressed, compressing the data and storing the compressed data in at least one sector in the data log. Otherwise, the method stores the data, uncompressed, in a plurality of sectors in the data log. The method generates a one or more state bits indicating (i) whether the data is stored as uncompressed or compressed, and (ii) if the data is stored as compressed, a size of the compressed data. The method then stores the one or more state bits in an entry of a logical map table associated with an LBA that corresponds to the data block.

Abstract: An efficient scheduling of IOs in a computing system using dynamic bandwidth regulation includes building up a shared regulator to limit the total IOPS scheduling among all IO classes at any given time. Reserved regulators may be used to place limits on the IOPS scheduled for a particular IO class at any given time. An outstanding IO window may also limit the overall number of outstanding IOs, and/or the bytes of outstanding IOs at any particular time. A first stage of IO scheduling may involve enforcing the reserved regulators to limit the IOPS scheduled for particular IO classes. A second stage of IO scheduling may involve enforcing the shared regulator to limit the total IOPS scheduled for all IO classes.