Abstract: A method for modifying key-value pairs of a B+ tree is provided. The method receives a request to modify a particular key-value pair. Each node of the tree has a modification number. The method traverses a path on the tree from the root node toward the particular node. The traversing includes upon reaching a parent node of the path, acquiring a shared lock on both the parent node and a child node one level below the parent node. Upon determining that the child node is the particular node, the method stores the modification number of the particular node, releases the shared lock on the particular node, compares a current modification number of the node with its stored number, and acquires an exclusive lock on the node if the numbers are the same. The method increments the current modification number of the node and modifies it while in the exclusive lock.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for resynchronizing data in a storage system. One of the methods includes determining that a particular primary disk of a capacity object of a storage system has failed, wherein the capacity comprises a plurality of segments, and wherein the each segment comprises: a plurality of primary columns each corresponding to a respective primary disk of the capacity object, and a plurality of parity columns each corresponding to a respective parity disk of the capacity object; and resynchronizing, for each segment of one or more segments of the capacity object, the primary column of the segment corresponding to the particular primary disk using i) the primary columns of the segment corresponding to each other primary disk of the capacity object, ii) one or more parity columns of the segment, and iii) the column summaries of the segment.

Abstract: The disclosure provides for isolation of concurrent read and write transactions on the same file, thereby enabling higher file system throughput relative to serial-only transactions. Race conditions and lock contentions in multi-writer scenarios are avoided in file stat (metadata) updates by the use of an aggregator to merge updates of committed transactions to maintain file stat truth, and an upgrade lock that enforces atomicity of file stat access, even while still permitting multiple processes to concurrently read from and/or write to the file data. The disclosure is applicable to generic file systems, whether native or virtualized, and may be used, for example, to speed access to database files that require prolonged input/output (I/O) transaction time periods.

Abstract: System and method for managing snapshots of storage objects in a storage system use a consolidation operation to reduce read amplification for stored snapshots of a storage object that are stored in log segments in the storage system according to a log-structured file system as storage service objects. The consolidation operation involves identifying target log segments among the log segments that include live blocks that are associated with the latest snapshot of the storage object and determining the number of the live blocks included in each of the target log segments. Based on the number of the live blocks in each of the target log segments, candidate consolidation log segments are determined from the target log segments. The live blocks in the candidate consolidation log segments are then consolidated to new log segments, which are uploaded to the storage system as new storage service objects.

Abstract: The present disclosure is related to methods, systems, and machine-readable media for supporting deduplication in object storage using subset hashes. A plurality of hashes of a plurality of blocks of a plurality of log segments can be received from a software defined data center, wherein each block corresponds to a respective logical address. Each of the plurality of logical addresses can be associated with a respective sequentially-allocated chunk identifier in a logical map. A subset hash comprising a hash of a subset of the plurality of blocks can be determined that corresponds to a contiguous range of the plurality of logical addresses. A search of a hash map for the subset hash can be performed to determine if the subset hash is a duplicate. The subset of the plurality of blocks can be deduplicated responsive to a determination that the subset hash is a duplicate.

Abstract: A method for encrypting data blocks is provided. The method receives a plurality of data blocks and encrypts each data block using an LBA of the data block as a tweak. The method writes the plurality of encrypted data blocks to physical blocks of the plurality of physical disks. The method then performs deduplication on the physical disks by determining that first and second physical blocks in the physical disks are duplicates, decrypting encrypted data in the first physical block using a first LBA associated with the first physical block as the tweak, and re-encrypting decrypted data in the first physical block using a PBA associated with the first physical block as the tweak. When reading the data back, either the LBA or PBA is used as the tweak, depending on whether the data was encrypted using LBA or re-encrypted using PBA during the deduplication process.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for resynchronizing data in a storage system. One of the methods includes receiving, by a first storage subsystem, a plurality of write requests corresponding to respective meta data blocks, wherein the first storage subsystem comprises a meta object; storing, by the first storage subsystem and for each write request, in each disk of the meta object, a version of the corresponding meta data block; determining that a particular disk of the meta object has failed; determining whether one or more valid versions of the meta data block are stored in respective other disks of the meta object; and in response to determining that one or more valid versions of the meta data block are stored in respective other disks of the meta object, resynchronizing the meta data block in the particular disk.

Abstract: The present disclosure provides techniques for deduplicating files. The techniques include creating a data structure that organizes metadata about chunks of files, the organization of the metadata preserving order and locality of the chunks within files. The organization of the metadata within storage blocks of storage devices matches the order of chunks within files. Upon a read or write operation to a metadata, the preservation of locality of metadata results in the likely fetching, from storage into a memory cache, metadata of subsequent and contiguous chunks. The preserved locality results in faster subsequent read and write operations of metadata, because the read and write operations are likely to be executed from memory rather than from storage.

Abstract: System and method for managing snapshots of storage objects in a storage system use a consolidation operation to reduce read amplification for stored snapshots of a storage object that are stored in log segments in the storage system according to a log-structured file system as storage service objects. The consolidation operation involves identifying target log segments among the log segments that include live blocks that are associated with the latest snapshot of the storage object and determining the number of the live blocks included in each of the target log segments. Based on the number of the live blocks in each of the target log segments, candidate consolidation log segments are determined from the target log segments. The live blocks in the candidate consolidation log segments are then consolidated to new log segments, which are uploaded to the storage system as new storage service objects.

Abstract: Example methods are provided for unbalanced storage resource usage configuration for a distributed storage system in a virtualized computing environment. An example method may include obtaining usage data associated with multiple storage resources forming the distributed storage system. The multiple storage resources are supported by the multiple hosts. Based on the usage data, the method may further include determining a higher usage set and a lower usage set of one or more storage resources from the multiple storage resources and configuring the multiple hosts to use the multiple storage resources in an unbalanced manner by using the higher usage set of one or more storage resources at a higher usage level compared to the lower usage set of one or more storage resources.

Abstract: Techniques for efficiently managing a file clone from a filesystem which supports efficient volume snapshots are provided. In some embodiments, a system may receive an instruction to remove the file clone from the filesystem. The file clone may be a point-in-time copy of metadata of an original file. The system may further—for a file map entry in a filesystem tree associated with the file clone, the file map entry indicating a data block—decrement a reference count in a reference count entry associated with the file map entry. The reference count entry may be stored in the filesystem tree according to a key and the key may comprise an identification of the original file. The system may further reclaim the data block in a storage system when the reference count is zero.

Abstract: In one set of embodiments, a computer system can periodically run an unmap service configured to scan a subset of bitmaps maintained by a file system of the computer system. As part of scanning each bitmap in the subset, the unmap service can, for each bit in the bitmap: (1) check whether the bit indicates that a corresponding physical block address (PBA) on the storage backend is currently free; (2) upon determining that the bit indicates the PBA is currently free, identify an extent within the bitmap where the PBA resides; (3) check whether an unmap indicator associated with the extent indicates that at least one free PBA in the extent is not currently unmapped in the storage backend; and (4) upon determining that the unmap indicator indicates at least one free PBA in the extent is not currently unmapped in the storage backend, add the PBA to a list of PBAs to be unmapped.

Abstract: Exemplary methods, apparatuses, and systems include a file system process determining to a flush a node in a first tree. The first node includes a buffer structured as a second tree. The file system process generates an input/output instruction to load the buffer from a first memory to a second memory. The second tree is stored in two more non-contiguous locations in the first memory and the input/output operation includes a read operation corresponding to each of the two or more non-contiguous locations. The file system process causes the input/output instruction to be executed concurrently on the first memory.

Abstract: A method for cleaning an object storage having a plurality of segments is provided. Each segment includes an identifier through which the segment is accessed. The method identifies a first segment in the plurality of segments. The first segment includes a first identifier and a first size. The method determines that a utilization ratio for the first segment is below a threshold. As a result, the method generates a second segment from the first segment, such that the second segment includes a second identifier that is the same as the first identifier and a second size that is smaller than the first size. The method then writes the second segment to the object storage.

Abstract: A method for managing data associated with objects stored in a cloud storage is provided. The method receives, at a first compute node, first data associated with an object stored in the cloud storage, the first compute node being one of a plurality of compute nodes that store data associated with different objects as storage objects in a log-structured merging (LSM) tree data structure. The method then assigns a first unique name to a first storage object associated with the first data, the first unique name comprising a combination of at least an identifier identifying the first compute node and a first incremental local value. The method stores the first storage object in a first level (L0) of the LSM tree data structure.

Abstract: Techniques for supporting large segments when issuing writes to an erasure coded storage object in a distributed storage system are provided. In one set of embodiments, a node of the system can pre-allocate a segment of space in a capacity object of the storage object, receive a write request for updating a logical data block of the storage object, write data/metadata for the block to a record in a data log of a metadata object of the storage object, place the block in an in-memory bank, and determine whether the in-memory bank has become full. If so, the node can compute/fill-in one or more parity blocks for each stripe of the storage object in the in-memory bank and write, based on a next sub-segment pointer pointing to a free sub-segment of the pre-allocated segment, the contents of the in-memory bank via a full stripe write to the free sub-segment.

Abstract: A logical map represents fragments from separate versions of a data object. Migration of data from a first (old) version to the second (new) version happens gradually, where write operations go to the new version of the data object. The logical map initially points to the old data object, but is updated to point to the portions of the new data object as write operations are performed on the new data object. A background migration copies data from the old data object to the new data object.

Abstract: Client data is structured as a set of data blocks. A first subset of data blocks is stored on a current segment of a plurality of disks. A second subset of data blocks is stored on a previous segment. A request to clean client data is received. The request includes a request to update the current segment to include the second subset of data blocks. The second subset of data blocks is accessed and transmitted from a lower layer to a higher layer of the system. Parity data is generated at the higher layer. The parity data is transmitted to the lower layer. The lower layer is employed to generate a local copy of the second subset of data blocks. Each local address that references the local copy of the second subset of data blocks is included in the current segment. The parity data is written in the current segment.

Abstract: A system and method for transferring data between a user space buffer in the address space of a user space process running on a virtual machine and a storage system are described. The user space buffer is represented as a file with a file descriptor. In the method, a file system proxy receives a request for I/O read or write from the user space process without copying data to be transferred. The file system proxy then sends the request to a file system server without copying data to be transferred. The file system server then requests that the storage system perform the requested I/O directly between the storage system and the user space buffer, the only transfer of data being between the storage system and the user space buffer.

Abstract: Techniques for issuing efficient writes to an erasure coded storage object in a distributed storage system are provided. In one set of embodiments, a node of the system can receive a write request for updating a logical data block of the storage object, write data/metadata for the block to a record in a data log of a metadata object of the storage object (where the metadata object is stored on a performance storage tier), place the block data in a free slot of an in-memory bank, and determine whether the in-memory bank has become full. If the in-memory bank is full, the node can further allocate a segment in a capacity object of the storage object for holding contents of the in-memory bank (where the capacity object is stored on a capacity storage tier), and write the in-memory bank contents via a full stripe write to the allocated segment.