Abstract: Techniques for recovering metadata associated with data backed up in cloud object storage are provided. In one set of embodiments, a computer system can create a snapshot of a data set, where the snapshot includes a plurality of data blocks of the data set that have been modified since the creation of a prior snapshot of the data set. The computer system can further upload the snapshot to a cloud object storage platform of a cloud infrastructure, where the snapshot is uploaded as a plurality of log segments conforming to an object format of the cloud object storage platform, and where each log segment includes one or more data blocks in the plurality of data blocks, and a set of metadata comprising, for each of the one or more data blocks, an identifier of the data set, an identifier of the snapshot, and a logical block address (LBA) of the data block.

Abstract: In one set of embodiments, a computer system can create a snapshot of a data set, where the snapshot includes a plurality of data blocks of the data set. The computer system can upload the snapshot to a cloud object storage platform of a cloud infrastructure, where the snapshot is uploaded as a plurality of log segments conforming to an object format of the cloud object storage platform, and where each log segment includes one or more data blocks in the plurality of data blocks, and a set of metadata comprising, for each of the one or more data blocks, an identifier of the data set, an identifier of the snapshot, and a logical block address (LBA) of the data block. The computer system can then communicate the set of metadata to a server component running in a cloud compute and block storage platform of the cloud infrastructure.

Abstract: The present disclosure is related to methods, systems, and machine-readable media for supporting deduplication in file storage using file chunk hashes. A hash of a chunk of a log segment can be received from a software defined data center. A chunk identifier can be associated with the hash in a hash map that stores associations between sequentially-allocated chunk identifiers and hashes. The chunk identifier can be associated with a logical address corresponding to the chunk of the log segment in a logical map that stores associations between the sequentially-allocated chunk identifiers and logical addresses. A search of the hash map can be performed to determine if the chunk is a duplicate, and the chunk can be deduplicated responsive to a determination that the chunk is a duplicate.

Abstract: The present disclosure is related to methods, systems, and machine-readable media for supporting deduplication in object storage using subset hashes. A plurality of hashes of a plurality of blocks of a plurality of log segments can be received from a software defined data center, wherein each block corresponds to a respective logical address. Each of the plurality of logical addresses can be associated with a respective sequentially-allocated chunk identifier in a logical map. A subset hash comprising a hash of a subset of the plurality of blocks can be determined that corresponds to a contiguous range of the plurality of logical addresses. A search of a hash map for the subset hash can be performed to determine if the subset hash is a duplicate. The subset of the plurality of blocks can be deduplicated responsive to a determination that the subset hash is a duplicate.

Abstract: System and method for managing space in storage object structures stored in a storage system uses an object B tree structure to manage a storage object in a storage object structure in the storage system, which provides virtual physical block addresses for data of the storage object and for metadata of nodes of the object B tree structure. The metadata of the nodes of the object B tree structure is written in the storage object structure using first real physical block addresses, which are translated from the virtual block addresses for the metadata of the nodes of the object B tree structure using a metadata system file B tree structure. The data of the storage object is written in the storage object structure using second real physical block addresses, which are translated from the virtual block addresses for the data of the storage object using a data system file B tree structure.

Abstract: System and method for creating group snapshots of multiple storage objects uses storage structures to store the storage objects in a storage system. In order to create a group snapshot of storage objects, a prepare snapshot process is executed at each of multiple host computers managing the storage objects as storage structures to block input and output operations on the storage objects stored in the storage structures. A commit snapshot process is then executed at each of the multiple host computers to create snapshots of the storage object in the respective storage structures. In addition, the snapshots of the storage object are recorded in the respective storage structures as snapshot entries in the respective storage structures.

Abstract: Techniques for the increased efficiency of storing data objects storage in the object storage of a software designed data center (SDDC) are provided. The techniques include the efficient storage of data, while enabling snapshots of each updating of the data. The snapshots of the data may be efficiently recovered via the techniques. Difference-level mappings for each snapshot are encoded in compact self-balancing data trees included in the object's metadata. The metadata mappings include mappings between various address spaces employed by the SDDC, as well as the address spaces employed by data stores that store the data on physical medium. Because the metadata is efficiently structured, the metadata for an object may be cached for quick lookups during data access and/or snapshot recovery. The techniques also provide low-latency recovery and/or system rollback in the event of any failure in the SDDC.

Abstract: Techniques for the increased efficiency of storing data objects storage in the object storage of a software designed data center (SDDC) are provided. The techniques include the efficient storage of data, while enabling snapshots of each updating of the data. The snapshots of the data may be efficiently recovered via the techniques. Difference-level mappings for each snapshot are encoded in compact self-balancing data trees included in the object's metadata. The metadata mappings include mappings between various address spaces employed by the SDDC, as well as the address spaces employed by data stores that store the data on physical medium. Because the metadata is efficiently structured, the metadata for an object may be cached for quick lookups during data access and/or snapshot recovery. The techniques also provide low-latency recovery and/or system rollback in the event of any failure in the SDDC.

Abstract: Techniques for tiering data to a cold storage tier of a cloud object storage platform are provided. In one set of embodiments, a computer system can identify one or more old snapshots of a data set that reside in a first storage tier of the cloud object storage platform, where the one or more old snapshots are snapshots that are unlikely to be deleted from the cloud object storage platform within a period of N days. The computer system can further, for each snapshot in the one or more old snapshots: identify one or more data blocks in the snapshot that are superseded by a more recent snapshot in the one or more old snapshots; write the one or more data blocks to a second (i.e., cold) storage tier of the cloud object storage platform that has a lower storage cost than the first storage tier; and cause the one or more data blocks to be deleted from the first storage tier.

Abstract: The present disclosure is related to methods, systems, and machine-readable media for supporting deduplication in object storage using subset hashes. A plurality of hashes of a plurality of blocks of a plurality of log segments can be received from a software defined data center, wherein each block corresponds to a respective logical address. Each of the plurality of logical addresses can be associated with a respective sequentially-allocated chunk identifier in a logical map. A subset hash comprising a hash of a subset of the plurality of blocks can be determined that corresponds to a contiguous range of the plurality of logical addresses. A search of a hash map for the subset hash can be performed to determine if the subset hash is a duplicate. The subset of the plurality of blocks can be deduplicated responsive to a determination that the subset hash is a duplicate.

Abstract: The present disclosure is related to methods, systems, and machine-readable media for supporting deduplication in file storage using file chunk hashes. A hash of a chunk of a log segment can be received from a software defined data center. A chunk identifier can be associated with the hash in a hash map that stores associations between sequentially-allocated chunk identifiers and hashes. The chunk identifier can be associated with a logical address corresponding to the chunk of the log segment in a logical map that stores associations between the sequentially-allocated chunk identifiers and logical addresses. A search of the hash map can be performed to determine if the chunk is a duplicate, and the chunk can be deduplicated responsive to a determination that the chunk is a duplicate.

Abstract: Techniques for recovering metadata associated with data backed up in cloud object storage are provided. In one set of embodiments, a computer system can create a snapshot of a data set, where the snapshot includes a plurality of data blocks of the data set that have been modified since the creation of a prior snapshot of the data set. The computer system can further upload the snapshot to a cloud object storage platform of a cloud infrastructure, where the snapshot is uploaded as a plurality of log segments conforming to an object format of the cloud object storage platform, and where each log segment includes one or more data blocks in the plurality of data blocks, and a set of metadata comprising, for each of the one or more data blocks, an identifier of the data set, an identifier of the snapshot, and a logical block address (LBA) of the data block.

Abstract: Techniques for tiering data to a cold storage tier of a cloud object storage platform are provided. In one set of embodiments, a computer system can identify one or more old snapshots of a data set that reside in a first storage tier of the cloud object storage platform, where the one or more old snapshots are snapshots that are unlikely to be deleted from the cloud object storage platform within a period of N days. The computer system can further, for each snapshot in the one or more old snapshots: identify one or more data blocks in the snapshot that are superseded by a more recent snapshot in the one or more old snapshots; write the one or more data blocks to a second (i.e., cold) storage tier of the cloud object storage platform that has a lower storage cost than the first storage tier; and cause the one or more data blocks to be deleted from the first storage tier.

Abstract: A method for managing metadata for data stored in a cloud storage is provided. The method receives, at a first of a plurality of metadata servers, information associated with an object stored in the cloud storage, the information comprising a plurality of LBAs for where the object is stored. Each metadata server allocates contiguous chunk IDs for a group of objects. The method generates a new chunk ID for the object, which is a combination of a unique fixed value and a monotonically incrementing local value associated with each LBA, such that a first LBA is mapped to a first chunk ID having a first local value and a next LBA is mapped to a second chunk ID having the first local value incremented as a second local value. The method stores the new chunk ID and other metadata in one or more tables stored in a metadata storage.

Abstract: System and method for creating linked clones of storage objects stored in a storage uses a target storage structure that includes a linked clone being created from a selected snapshot of a storage object and surface snapshots. The linked clone is created in the target storage structure with references to the selected snapshot of the storage object in a source storage structure without completely copying the selected snapshot to the target storage structure. The surface snapshots, which correspond to other snapshots of the storage object in the source storage structure, are created in the target storage structure with references to the other snapshots without completely copying the other snapshots to the target storage structure to produce the linked clone with similar snapshots as the storage object at a point in time when the selected snapshot was captured.

Abstract: System and method for managing different types of snapshots of storage objects stored in a storage system uses a storage structure that includes both a storage object and any snapshots of the storage object. When a request to delete a snapshot of the storage object is received, a determination is made whether the snapshot is a first snapshot type. After determining that the snapshot is not the first snapshot type, the snapshot is deleted from the storage structure. After determining that the snapshot is the first snapshot type, the snapshot is marked as delete designated, and a deletion of the snapshot from the storage structure is prevented when there is a second snapshot type snapshot remaining in the storage structure that was created after the snapshot had been created so that the snapshot is available when the second snapshot type snapshot is subsequently used.

Abstract: System and method for managing snapshots of storage objects stored in a storage system uses a storage structure that includes both a storage object and any snapshots of the storage object. When a request to delete the storage object is received, the storage structure that includes the storage object is checked to determine if there is at least one snapshot in the storage structure. After determining that there is at least one snapshot in the storage structure, the storage object is marked as being delete designated without deleting the storage structure. After determining that there is no snapshot in the storage structure, the storage structure is deleted.

Abstract: System and method for creating group snapshots of multiple storage objects uses storage structures to store the storage objects in a storage system. In order to create a group snapshot of storage objects, a prepare snapshot process is executed at each of multiple host computers managing the storage objects as storage structures to block input and output operations on the storage objects stored in the storage structures. A commit snapshot process is then executed at each of the multiple host computers to create snapshots of the storage object in the respective storage structures. In addition, the snapshots of the storage object are recorded in the respective storage structures as snapshot entries in the respective storage structures.

Abstract: System and method for managing snapshots of storage objects stored in a storage system uses a storage structure that includes both a storage object and any snapshots of the storage object. When a request to delete the storage object is received, the storage structure that includes the storage object is checked to determine if there is at least one snapshot in the storage structure. After determining that there is at least one snapshot in the storage structure, the storage structure is marked as being delete designated without deleting the storage structure. After determining that there is no snapshot in the storage structure, the storage structure is deleted.

Abstract: System and method for managing space in storage object structures stored in a storage system uses an object B tree structure to manage a storage object in a storage object structure in the storage system, which provides virtual physical block addresses for data of the storage object and for metadata of nodes of the object B tree structure. The metadata of the nodes of the object B tree structure is written in the storage object structure using first real physical block addresses, which are translated from the virtual block addresses for the metadata of the nodes of the object B tree structure using a metadata system file B tree structure. The data of the storage object is written in the storage object structure using second real physical block addresses, which are translated from the virtual block addresses for the data of the storage object using a data system file B tree structure.