Abstract: Systems for accessing client data is described. A request to access a first data block is received. The request indicates a first logical address referencing the first data block. First mapping data is employed to identify a first physical addresses corresponding to the first logical addresses. The first mapping data encodes a first LOM transaction ID and candidate local addresses. The first mapping data is employed to identify the candidate local address and the first LOM transaction ID. A usage table is employed to determine the current status of the first LOM transaction ID. The candidate local address is employed to access the first data block. Second mapping data is employed to identify an updated local address of the set of local addresses. The updated local address currently references the first data block. The updated local address is employed to access the first data block.

Abstract: Client data is structured as a set of data blocks. A first subset of data blocks is stored on a current segment of a plurality of disks. A second subset of data blocks is stored on a previous segment. A request to clean client data is received. The request includes a request to update the current segment to include the second subset of data blocks. The second subset of data blocks is accessed and transmitted from a lower layer to a higher layer of the system. Parity data is generated at the higher layer. The parity data is transmitted to the lower layer. The lower layer is employed to generate a local copy of the second subset of data blocks. Each local address that references the local copy of the second subset of data blocks is included in the current segment. The parity data is written in the current segment.

Abstract: Client data is structured as a set of data blocks. A first subset of data blocks is stored on a current segment of the disks. A second subset of data blocks is stored on a previous segment. A request to clean client data is received, including a request to update the current segment to include the second subset of data blocks. The second subset of data blocks is accessed and transmitted from a lower layer to a higher system layer. Parity data is generated at the higher layer. The parity data is transmitted to the lower layer. The lower layer updates second mapping data. In the updated mapping of the second mapping data, each local address that references a data block of the second subset of data blocks is included in the current segment of the plurality of disks. The lower layer writes the parity data in the current segment.

Abstract: Techniques for efficiently storing client data blocks on a distributed-computing system are provided. The system includes a fast performance tier and a large capacity tier. The capacity tier stores the client data blocks in erasure encoded data stripes. The performance tier stores logical map data including an address map indicating a correspondence between logical addresses associated with a first layer of the system and physical addresses associated with a second layer. A method includes receiving a request to include additional client data blocks in the client blocks. The request indicates logical addresses for additional blocks. Corresponding physical addresses for additional block are determined. Each additional block is stored at the physical address. Additional logical map data is stored in the performance tier. Storing the additional logical map data includes updating the address map to indicate the correspondence between the logical addresses and the physical addresses for the additional blocks.

Abstract: Techniques for using erasure coding across multiple regions to reduce the likelihood of losing objects in a cloud object storage platform are provided. In one set of embodiments, a computer system can upload each of a plurality of data objects to each of a plurality of regions of the cloud object storage platform. The computer system can further compute a parity object based on the plurality of data objects, where the parity object encodes parity information for the plurality of data objects. The computer system can then upload the parity object to another region of the cloud object storage platform different from the plurality of regions.

Abstract: Techniques for using data mirroring across regions to reduce the likelihood of losing objects in a cloud object storage platform are provided. In one set of embodiments, a computer system can upload first and second copies of a data object to first and second regions of the cloud object storage platform respectively, where the first and second copies are identical. The computer system can then attempt to read the first copy of the data object from the first region. If the read attempt fails, the computer system can retrieve the second copy of the data object from the second region.

Abstract: Techniques for using erasure coding in a single region to reduce the likelihood of losing objects in a cloud object storage platform are provided. In one set of embodiments, a computer system can upload a plurality of data objects to a region of a cloud object storage platform, where the plurality of data objects including modifications to a data set. The computer system can further compute a parity object based on the plurality of data objects, where the parity object encodes parity information for the plurality of data objects. The computer system can then upload the parity object to the same region where the plurality of data objects was uploaded.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for resynchronizing data in a storage system. One of the methods includes determining that a particular disk of a capacity object of a storage system is out-of-sync and that a primary disk is unavailable; and for each segment of one or more segments of the capacity object: generating a first version of the column of the segment corresponding to the unavailable primary disk; determining whether the data integrity token in the column summary of the generated first version is valid; and in response to determining that the data integrity token is valid, resynchronizing the column of the segment corresponding to the particular disk using i) the primary columns of the segment corresponding to each available primary disk and ii) the first version of the column of the segment corresponding to the unavailable primary disk.

Abstract: In accordance with the present disclosure, files may be deduplicated in a distributed storage system having a plurality of storage volumes. A uniqueness metric for each file may indicate a degree of deduplication of the respective data files in the given storage volume. The uniqueness metric may be used to identify files for rebalancing in the distributed storage system. The uniqueness metric may be efficiently calculated with enough accuracy using a sampling methodology.

Abstract: The disclosure herein describes synchronizing a data cache and an LSM tree file system on an object storage platform. Instructions to send a cached data set from the data cache to the LSM tree file system are received. An updated metadata catalog is generated. If the LSM tree structure is out of shape, compaction is performed on the LSM tree file system which may be on a different system or server. When an unmerged compacted metadata catalog is identified, a merged metadata catalog is generated, based on the compacted metadata catalog and the cached data set, and associated with the cached data set. The cached data set and the associated metadata catalog are sent to the LSM tree file system, whereby the data cache and the LSM tree file system are synchronized. Synchronization is enabled without the data cache or file system being locked and/or waiting for the other entity.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for resynchronizing data in a storage system. One of the methods includes determining that a particular disk of a capacity object of a storage system was offline for an interval of time, wherein the capacity object comprises a plurality of segments, and wherein the storage system comprises a segment usage table identifying a linked list of particular segments of the capacity object that are in use; determining a time point at which the particular disk went offline; determining one or more first segments of the capacity object that were modified after the time point, wherein determining one or more first segments comprises determining each segment of the segment usage table having a transaction ID that is larger than the time point; and resynchronizing, for each first segment, a portion of the particular disk corresponding to the first segment.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for resynchronizing data in a storage system. One of the methods includes determining that a particular primary disk of a capacity object of a storage system has failed, wherein the capacity comprises a plurality of segments, and wherein the each segment comprises: a plurality of primary columns each corresponding to a respective primary disk of the capacity object, and a plurality of parity columns each corresponding to a respective parity disk of the capacity object; and resynchronizing, for each segment of one or more segments of the capacity object, the primary column of the segment corresponding to the particular primary disk using i) the primary columns of the segment corresponding to each other primary disk of the capacity object, ii) one or more parity columns of the segment, and iii) the column summaries of the segment.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for resynchronizing data in a storage system. One of the methods includes receiving, by a first storage subsystem, a plurality of write requests corresponding to respective meta data blocks, wherein the first storage subsystem comprises a meta object; storing, by the first storage subsystem and for each write request, in each disk of the meta object, a version of the corresponding meta data block; determining that a particular disk of the meta object has failed; determining whether one or more valid versions of the meta data block are stored in respective other disks of the meta object; and in response to determining that one or more valid versions of the meta data block are stored in respective other disks of the meta object, resynchronizing the meta data block in the particular disk.

Abstract: System and method for writing updated versions of a configuration data file for a distributed file system in a storage system uses a directory renaming operation to write a new updated version of the configuration data file using the latest version of the configuration data file and a target directory. After the latest version of the configuration data file is modified by a particular host computer in the storage system, the modified configuration data file is written to a temporary file. The directory naming operation is then initiated on the temporary file to change the directory for the temporary file to the target directory. If the directory renaming operation has failed, a retry is performed by the particular host computer to write the new updated version of the configuration data file using a new latest version and a new target directory.

Abstract: System and method for automatic remediation for a distributed file system uses a file system (FS) remediation module running in a cluster management server and FS remediation agents running in a cluster of host computers. The FS remediation module monitors the cluster of host computers for related events. When a first file system service (FSS)-impacting event is detected, a cluster-level remediation action is executed at the cluster management server by the FS remediation module in response to the detected first FSS-impacting event. When a second FSS-impacting event is detected, a host-level remediation action is executed at one or more of the host computers in the cluster by the FS remediation agents in response to the detected second FSS-impacting event.

Abstract: Supporting distributed and local objects using a multi-writer log-structured file system (LFS) includes, on a node, receiving incoming data from each of a plurality of local objects; coalescing the received data; determining whether the coalesced data comprises a full segment of data; based at least on the coalesced incoming data comprises a full segment, writing at least a first portion of the coalesced data to a first storage of the LFS, wherein the coalesced data comprises the first portion and a remainder portion; writing the remainder portion to a second storage of the LFS; acknowledging the writing to the objects; determining whether at least a full segment of data has accumulated in the second storage; based at least on determining that at least a full segment has accumulated, writing at least a portion of the accumulated data as one or more full segments of data to the first storage.

Abstract: System and method for writing updated versions of a configuration data file for a distributed file system in a storage system uses a directory renaming operation to write a new updated version of the configuration data file using the latest version of the configuration data file and a target directory. After the latest version of the configuration data file is modified by a particular host computer in the storage system, the modified configuration data file is written to a temporary file. The directory naming operation is then initiated on the temporary file to change the directory for the temporary file to the target directory. If the directory renaming operation has failed, a retry is performed by the particular host computer to write the new updated version of the configuration data file using a new latest version and a new target directory.

Abstract: Solutions for supporting storage using a multi-writer log-structured file system (LFS) are disclosed that include receiving incoming data from an object of a plurality of objects that are configured to simultaneously write to the LFS from different nodes; based at least on receiving the incoming data, determining whether sufficient free segments are available in a local segment usage table (SUT) for writing the incoming data; based at least on determining that insufficient free segments are available , requesting allocation of new free segments; writing the incoming data to a log; acknowledging the writing to the object; determining whether the log has accumulated a full segment of data; based at least on determining that the log has accumulated a full segment of data, writing the full segment of data to a first segment of the free segments; and updating the local SUT to mark the first segment as no longer free.

Abstract: Systems, for managing metadata data blocks, include a fast performance tier and a large capacity tier. The capacity tier stores the data blocks in erasure-encoded stripes. The performance tier stores map data indicating correspondences between logical addresses, associated with a first layer of the system, and physical addresses associated with a second layer. The map data is structured as a B-tree with leaf nodes and index nodes. A method includes determining a volume associated with the leaves. In response to the volume being greater than a threshold, migrating a portion of the leaves to the stripes of the capacity tier. A portion of the indexes that include pointers to the migrated portion of the leaves are updated to include updated pointers to physical addresses of the stripes that store the migrated leaves. A portion of the performance tier that stores the migrated leaves is re-allocated to store additional map data.

Abstract: Techniques for issuing efficient writes to an erasure coded storage object in a distributed storage system via adaptive logging are provided. In one set of embodiments, a node of the system can receive a write request for updating one or more logical data blocks of the storage object and determine whether a size of the one or more logical data blocks meets or exceeds a threshold size. Upon determining that the size of the one or more logical data blocks meets or exceeds the threshold size, the node can allocate a segment in a capacity object of the storage object, write the one or more logical data blocks via a full stripe write to the segment, and write metadata for the one or more logical data blocks to a log record in a log of a metadata object of the storage object. The metadata written to the log record can include mappings between logical block addresses (LBAs) of the one or more logical data blocks and physical block addresses (PBAs) where the one or more logical data blocks reside in the segment.