Abstract: The efficiency of segment cleaning for a log-structured file system (LFS) is enhanced at least by storing additional information in a segment usage table (SUT). Live blocks (representing portions of stored objects) in an LFS are determined based at least on the SUT. Chunk identifiers associated with the live blocks are read. The live blocks are coalesced at least by writing at least a portion of the live blocks into at least one new segment. A blind update of at least a portion of the chunk identifiers in a chunk map is performed to indicate the new segment. The blind update includes writing to the chunk map without reading from the chunk map. In some examples, the objects comprise virtual machine disks (VMDKs) and the SUT changes between a list format and a bitmap format, to minimize size.

Abstract: A method for efficient journal truncation is provided. A method for journal truncation includes maintaining a journal in a memory of a computing system including a plurality of records. Each record indicates a transaction in an ordered data structure. The method includes maintaining a truncation queue in the memory including one or more entries. Each entry in the truncation queue includes a physical on-disk offset associated with a different record of the plurality of records. The method includes determining to truncate the journal and truncating records, of the plurality of records, from the journal starting from a beginning record in the journal up to the record with the physical on-disk offset associated a least recent entry of the one or more entries in the truncation queue, where the truncating includes removing the records from the memory.

Abstract: A method for efficient write-back for journal truncation is provided. A method includes maintaining a journal in a memory of a computing system including a plurality of records. Each record indicates a transaction associated with one or more pages in an ordered data structure and maintaining a dirty list including an entry for each page indicated by a record in the journal. Each entry in the dirty list includes a respective first log sequence number (LSN) associated with a least recent record of the plurality of records that indicates the page and a respective second LSN associated with a most recent record of the plurality of records that indicates the page. The method includes determining to truncate the journal. The method includes identifying one or more records, of the plurality of records, from the journal to write back to a disk, where the identifying is based on the dirty list.

Abstract: The disclosure describes growing a data cache using a background hash bucket growth process. A first memory portion is allocated to the data buffer of the data cache and a second memory portion is allocated to the metadata buffer of the data cache based on the cache growth instruction. The quantity of hash buckets in the hash bucket buffer is increased and the background hash bucket growth process is initiated, wherein the process is configured to rehash hash bucket entries of the hash bucket buffer in the increased quantity of hash buckets. A data entry is stored in the data buffer using the allocated first memory portion of the data cache and metadata associated with the data entry is stored using the allocated second memory portion of the metadata buffer, wherein a hash bucket entry associated with the data entry is stored in the increased quantity of hash buckets.

Abstract: A method for resumeable snapshot deletion is provided. A method for deletion of nodes maintained in an ordered data structure for a first snapshot includes processing the nodes maintained in the ordered data structure according to a defined order, setting a node path cursor with a pointer to a node and an indication of the deletion of the node; storing the node path cursor in a persistent storage; and during processing of the nodes: detecting a failure; after the failure, checking the pointer of the node path cursor; and resuming processing of the nodes starting from the first node indicated by the pointer.

Abstract: Example methods and systems associated with dynamic configuration of a statistics endpoint in a virtualized computing environment have been disclosed. One example method includes in response to receiving a first request, by a host in the virtualized computing environment, accepting a configuration file specified in the first request; in response to receiving a second request, by the host, parsing a rule based on the configuration file and collecting statistics based on the rule; processing, by the host, the statistics collected based on the rule; and sending, by the host, the processed statistics to a monitoring terminal.

Abstract: System and method for managing distributed storage objects for host unavailability in a distributed storage system uses at least one of a crash indicator in a specific on-disk block and a paused object indicator for a distributed storage object to determine whether to perform data recovery for the distributed storage object. When the crash indicator is set or the paused object indicator implies that the distributed storage object is a paused object, the distributed storage object is left as a paused object without perform the data recovery for the distributed storage object. When the crash indicator is unset and the paused object indicator implies that the distributed storage object is not a paused object, the data recovery for the distributed storage object is performed.

Abstract: The disclosure provides techniques for deduplicating files. The techniques include, upon creating or modifying a file, placing a logical timestamp of the current logical time, within a queue associated with the directory of the file. The techniques further include placing the logical timestamp within a queue of each parent directory of the directory of the file. To determine a set of files for deduplication, the techniques disclosed herein identify files that have been modified within a logical time range. The set of files modified within a logical time is identified by traversing directories of a storage system, the directories being organized within a tree structure. If a directory's queue does not contain a timestamp that is within the logical time range, then all child directories can be skipped over for further processing, such that no files within the child directories end up being within the set of files for deduplication.

Abstract: Techniques for using data mirroring across regions to reduce the likelihood of losing objects in a cloud object storage platform are provided. In one set of embodiments, a computer system can upload first and second copies of a data object to first and second regions of the cloud object storage platform respectively, where the first and second copies are identical. The computer system can then attempt to read the first copy of the data object from the first region. If the read attempt fails, the computer system can retrieve the second copy of the data object from the second region.

Abstract: Techniques for recovering metadata associated with data backed up in cloud object storage are provided. In one set of embodiments, a computer system can create a snapshot of a data set, where the snapshot includes a plurality of data blocks of the data set that have been modified since the creation of a prior snapshot of the data set. The computer system can further upload the snapshot to a cloud object storage platform of a cloud infrastructure, where the snapshot is uploaded as a plurality of log segments conforming to an object format of the cloud object storage platform, and where each log segment includes one or more data blocks in the plurality of data blocks, and a set of metadata comprising, for each of the one or more data blocks, an identifier of the data set, an identifier of the snapshot, and a logical block address (LBA) of the data block.

Abstract: A method for resumeable snapshot deletion is provided. A method for deletion of nodes maintained in an ordered data structure for a first snapshot includes processing the nodes maintained in the ordered data structure according to a defined order, setting a node path cursor with a pointer to a node and an indication of the deletion of the node; storing the node path cursor in a persistent storage; and during processing of the nodes: detecting a failure; after the failure, checking the pointer of the node path cursor; and resuming processing of the nodes starting from the first node indicated by the pointer.

Abstract: A method for managing data associated with objects stored in a cloud storage is provided. The method receives, at a first compute node, first data associated with an object stored in the cloud storage, the first compute node being one of a plurality of compute nodes that store data associated with different objects as storage objects in a log-structured merging (LSM) tree data structure. The method then assigns a first unique name to a first storage object associated with the first data, the first unique name comprising a combination of at least an identifier identifying the first compute node and a first incremental local value. The method stores the first storage object in a first level (L0) of the LSM tree data structure.

Abstract: A method for copy on write (COW) operations generally includes receiving a write request to a first node in an ordered data structure and updating a write ahead log record associated with COW operation with, instead of the content of the first node, a physical disk address of a second node owned by the run point in the ordered data structure that is a parent node of the first node, a pointer to the first node in the second node, a physical disk address of the first node, and a physical disk address of the third node. A metadata table record for a snapshot that owns the first node may be updated with a log sequence number (LSN) of the COW operation. A method for deleting a snapshot includes determining whether the COW operation recorded in the WAL record for the LSN is completed before deleting the snapshot.

Abstract: Disclosed techniques include deduplication. Techniques include determining whether a file is unique, and depending on whether the file is unique, deduplicating only part of the file or the entire file. The techniques include processing the first chunk of a file to determine whether the hash of the chunk hash is already within a chunk hash table, and if not, then a percentage of chunks of the file is similarly processed. If any of the hashes of chunks are already in the chunk hash table, then at least some of file has been previously deduplicated, and file is not unique the storage system. If none of the processed chunks have a hash that is already in the chunk hash table, then the file is considered to be unique within chunk store and only a partial percentage of the file's chunks are deduplicated. Not all of a unique file's chunks are deduplicated.

Abstract: In one set of embodiments, a computer system can create a snapshot of a data set, where the snapshot includes a plurality of data blocks of the data set. The computer system can upload the snapshot to a cloud object storage platform of a cloud infrastructure, where the snapshot is uploaded as a plurality of log segments conforming to an object format of the cloud object storage platform, and where each log segment includes one or more data blocks in the plurality of data blocks, and a set of metadata comprising, for each of the one or more data blocks, an identifier of the data set, an identifier of the snapshot, and a logical block address (LBA) of the data block. The computer system can then communicate the set of metadata to a server component running in a cloud compute and block storage platform of the cloud infrastructure.

Abstract: The disclosure herein describes writing data to a log-structured merge (LSM) tree file system on an object storage platform. Write data instructions indicating data for writing to the LSM tree file system are received. Based on the received instructions, the data is written to the first data cache. Based on an instruction to transfer data in the live data cache to the LSM tree file system, the first data cache is converted to a stable cache. A second data cache configured as a live data cache is then generated based on cloning the first data cache. The data in the first data cache is then written to the LSM tree file system. Use of a stable cache and a cloned live data cache enables parallel writing data to the file system by the stable cache and handling write data instructions by the live data cache.

Abstract: Scalable segment cleaning for log-structured file systems (LFSs) includes determining counts of segment cleaners and virtual nodes, with each virtual node being associated with a plurality of objects. Each virtual node is assigned to a selected segment cleaner. Based at least on the assignments, performing, for each virtual node, segment cleaning of the objects by the assigned segment cleaner. A portion, less than all, of the virtual nodes are reassigned to a newly selected segment cleaner based on a change of the count of the segment cleaners and/or a change of the count of the virtual nodes. Based at least on the reassignments, segment cleaning of the objects is performed, for each reassigned virtual node, by the reassigned segment cleaner. In some examples, the objects comprise virtual machine disks (VMDKs) and the segment cleaning uses a segment usage table (SUT) to track segment usage and identify segment cleaning candidates.

Abstract: Techniques for using data mirroring across regions to reduce the likelihood of losing objects in a cloud object storage platform are provided. In one set of embodiments, a computer system can upload first and second copies of a data object to first and second regions of the cloud object storage platform respectively, where the first and second copies are identical. The computer system can then attempt to read the first copy of the data object from the first region. If the read attempt fails, the computer system can retrieve the second copy of the data object from the second region.

Abstract: Techniques for using erasure coding in a single region to reduce the likelihood of losing objects in a cloud object storage platform are provided. In one set of embodiments, a computer system can upload a plurality of data objects to a region of a cloud object storage platform, where the plurality of data objects including modifications to a data set. The computer system can further compute a parity object based on the plurality of data objects, where the parity object encodes parity information for the plurality of data objects. The computer system can then upload the parity object to the same region where the plurality of data objects was uploaded.

Abstract: Techniques for using erasure coding across multiple regions to reduce the likelihood of losing objects in a cloud object storage platform are provided. In one set of embodiments, a computer system can upload each of a plurality of data objects to each of a plurality of regions of the cloud object storage platform. The computer system can further compute a parity object based on the plurality of data objects, where the parity object encodes parity information for the plurality of data objects. The computer system can then upload the parity object to another region of the cloud object storage platform different from the plurality of regions.