Abstract: In various embodiments, methods and systems for implementing a distributed metadata management system in distributed storage systems are provided. A distributed storage system operates based on data storage resources (e.g., extents and streams). The distributed metadata management system is implemented for extent and stream metadata to facilitate the scalability of metadata processing. The distributed storage system implements extent managers and stream managers that independently manage extent and stream metadata, respectively. The extent managers are associated with an extent table that stores extent metadata. The stream managers are associated with streams that store associations with extents. The distributed metadata management system can also utilize a bootstrap layer that leverages components of a legacy distributed storage system to facilitate distributed management of extent and stream metadata.

Abstract: A system including a data plane including a first virtual machine designated as a primary node and a second virtual machine designated as a secondary node, and a lock service configured to control changes in designation of the primary node and the secondary node. The first virtual machine repeatedly issues requests to the second virtual machine or lock service extending designation as the primary node for a first duration. The request is repeatedly issued within a duration shorter than the first duration. The second virtual machine receives an indication of the designation of the first virtual machine as the primary node being extended, issues a request to the lock service to change the designation of the primary node after passage the first duration of time without further indication of first virtual machine being extended as primary node, and receives confirmation from the lock service of the change in designation.

Abstract: In various embodiments, methods and systems for implementing a distributed metadata management system in distributed storage systems are provided. A distributed storage system operates based on data storage resources (e.g., extents and streams). The distributed metadata management system is implemented for extent and stream metadata to facilitate the scalability of metadata processing. The distributed storage system implements extent managers and stream managers that independently manage extent and stream metadata, respectively. The extent managers are associated with an extent table that stores extent metadata. The stream managers are associated with streams that store associations with extents. The distributed metadata management system can also utilize a bootstrap layer that leverages components of a legacy distributed storage system to facilitate distributed management of extent and stream metadata.

Abstract: In various embodiments, methods and systems for implementing garbage collection in distributed storage systems are provided. The distributed storage system operates based on independent management of metadata of extent and stream data storage resources. A hybrid garbage collection system based on reference counting garbage collection operations and mark-and-sweep garbage collection operations is implemented. An extent lifetime table that tracks reference weights and mark sequences for extents is initialized and updated based on indications from extent managers and stream managers, respectively. Upon determining that an extent is to be handed-off from weighted reference counting garbage collection operations to mark-and-sweep garbage collection operations, a reference weight field for the extent is voided and a mark sequence field of the extent is updated. The mark sequence field is updated with a latest global sequence number.

Abstract: In various embodiments, methods and systems for implementing a distributed metadata management system in distributed storage systems are provided. A distributed storage system operates based on data storage resources (e.g., extents and streams). The distributed metadata management system is implemented for extent and stream metadata to facilitate the scalability of metadata processing. The distributed storage system implements extent managers and stream managers that independently manage extent and stream metadata, respectively. The extent managers are associated with an extent table that stores extent metadata. The stream managers are associated with streams that store associations with extents. The distributed metadata management system can also utilize a bootstrap layer that leverages components of a legacy distributed storage system to facilitate distributed management of extent and stream metadata.

Abstract: In various embodiments, methods and systems for erasure coding with enhanced local protection groups are provided. An erasure coding scheme can be defined based on a Vertical Local Reconstruction Code (VLRC) that achieves high storage efficiency by combining the Local Reconstruction Code and conventional erasure coding, where the local reconstruction code (LRC) is carefully laid out across zones. Thus, when a zone is down, remaining fragments form an appropriate LRC. Further, an inter-zone erasure coding scheme—Zone Local Reconstruction Code (ZZG-2 code)—is provided having both local reconstruction within every zone and a-of-b recovery property across zones. An inter-zone adaptive erasure coding (uber code) scheme is provided, the uber code is configurable to produce near optimal performance in different environments characterized by intra and inter-zone bandwidth and machine failure rates.

Abstract: In various embodiments, methods and systems for implementing garbage collection in distributed storage systems are provided. The distributed storage system operates based on independent management of metadata of extent and stream data storage resources. A hybrid garbage collection system based on reference counting garbage collection operations and mark-and-sweep garbage collection operations is implemented. An extent lifetime table that tracks reference weights and mark sequences for extents is initialized and updated based on indications from extent managers and stream managers, respectively. Upon determining that an extent is to be handed-off from weighted reference counting garbage collection operations to mark-and-sweep garbage collection operations, a reference weight field for the extent is voided and a mark sequence field of the extent is updated. The mark sequence field is updated with a latest global sequence number.

Abstract: In various embodiments, methods and systems for implementing a distributed metadata management system in distributed storage systems are provided. A distributed storage system operates based on data storage resources (e.g., extents and streams). The distributed metadata management system is implemented for extent and stream metadata to facilitate the scalability of metadata processing. The distributed storage system implements extent managers and stream managers that independently manage extent and stream metadata, respectively. The extent managers are associated with an extent table that stores extent metadata. The stream managers are associated with streams that store associations with extents. The distributed metadata management system can also utilize a bootstrap layer that leverages components of a legacy distributed storage system to facilitate distributed management of extent and stream metadata.

Abstract: In various embodiments, methods and systems for erasure coding with enhanced local protection groups are provided. An erasure coding scheme can be defined based on a Vertical Local Reconstruction Code (VLRC) that achieves high storage efficiency by combining the Local Reconstruction Code and conventional erasure coding, where the local reconstruction code (LRC) is carefully laid out across zones. Thus, when a zone is down, remaining fragments form an appropriate LRC. Further, an inter-zone erasure coding scheme—Zone Local Reconstruction Code (ZZG-2 code)—is provided having both local reconstruction within every zone and a-of-b recovery property across zones. An inter-zone adaptive erasure coding (uber code) scheme is provided, the uber code is configurable to produce near optimal performance in different environments characterized by intra and inter-zone bandwidth and machine failure rates.

Abstract: In various embodiments, methods and systems for erasure coding data across multiple storage zones are provided. This may be accomplished by dividing a data chunk into a plurality of sub-fragments. Each of the plurality of sub-fragments is associated with a zone. Zones comprise buildings, data centers, and geographic regions providing a storage service. A plurality of reconstruction parities is computed. Each of the plurality of reconstruction parities computed using at least one sub-fragment from the plurality of sub-fragments. The plurality of reconstruction parities comprises at least one cross-zone parity. The at least one cross-zone parity is assigned to a parity zone. The cross-zone parity provides cross-zone reconstruction of a portion of the data chunk.

Abstract: In various embodiments, methods and systems for erasure coding data across multiple storage zones are provided. This may be accomplished by dividing a data chunk into a plurality of sub-fragments. Each of the plurality of sub-fragments is associated with a zone. Zones comprise buildings, data centers, and geographic regions providing a storage service. A plurality of reconstruction parities is computed. Each of the plurality of reconstruction parities computed using at least one sub-fragment from the plurality of sub-fragments. The plurality of reconstruction parities comprises at least one cross-zone parity. The at least one cross-zone parity is assigned to a parity zone. The cross-zone parity provides cross-zone reconstruction of a portion of the data chunk.

Abstract: Cloud data storage systems, methods, and techniques partition system data symbols into predefined-sized groups and then encode each group to form corresponding parity symbols, encode all data symbols into global redundant symbols, and store each symbol (data, parity, and redundant) in different failure domains in a manner that ensures independence of failures. In several implementations, the resultant cloud-encoded data features both data locality and ability to recover up to a predefined threshold tolerance of simultaneous erasures (unavailable data symbols) without any information loss. In addition, certain implementations also feature the placement of cloud-encoded data in domains (nodes or node groups) to provide similar locality and redundancy features simultaneous with the recovery of an entire domain of data that is unavailable due to software or hardware upgrades or failures.

Abstract: In various embodiments, methods and systems for erasure coding data across multiple storage zones are provided. This may be accomplished by dividing a data chunk into a plurality of sub-fragments. Each of the plurality of sub-fragments is associated with a zone. Zones comprise buildings, data centers, and geographic regions providing a storage service. A plurality of reconstruction parities is computed. Each of the plurality of reconstruction parities computed using at least one sub-fragment from the plurality of sub-fragments. The plurality of reconstruction parities comprises at least one cross-zone parity. The at least one cross-zone parity is assigned to a parity zone. The cross-zone parity provides cross-zone reconstruction of a portion of the data chunk.

Abstract: In various embodiments, methods and systems for erasure coding data across multiple storage zones are provided. This may be accomplished by dividing a data chunk into a plurality of sub-fragments. Each of the plurality of sub-fragments is associated with a zone. Zones comprise buildings, data centers, and geographic regions providing a storage service. A plurality of reconstruction parities is computed. Each of the plurality of reconstruction parities computed using at least one sub-fragment from the plurality of sub-fragments. The plurality of reconstruction parities comprises at least one cross-zone parity. The at least one cross-zone parity is assigned to a parity zone. The cross-zone parity provides cross-zone reconstruction of a portion of the data chunk.

Abstract: Embodiments of the present invention relate to systems, methods and computer storage media for erasure coding data in a distributed computing environment. A sealed extent is identified that is comprised of two or more data blocks and two or more index blocks. The sealed extent is optimized for erasure coding by grouping the two or more data blocks within the optimized sealed extent together and grouping the two or more index blocks within the optimized sealed extent together. The optimized extent may also be erasure coded, which includes creating data fragments and coding fragments. The data fragments and the coding fragments may also be stored in the distributed computing environment. Additional embodiments include monitoring statistical information to determine if replication, erasure coding or a hybrid storage plan should be utilized.

Abstract: Cloud data storage systems, methods, and techniques partition system data symbols into predefined-sized groups and then encode each group to form corresponding parity symbols, encode all data symbols into global redundant symbols, and store each symbol (data, parity, and redundant) in different failure domains in a manner that ensures independence of failures. In several implementations, the resultant cloud-encoded data features both data locality and ability to recover up to a predefined threshold tolerance of simultaneous erasures (unavailable data symbols) without any information loss. In addition, certain implementations also feature the placement of cloud-encoded data in domains (nodes or node groups) to provide similar locality and redundancy features simultaneous with the recovery of an entire domain of data that is unavailable due to software or hardware upgrades or failures.

Abstract: Embodiments of the present invention relate to systems, methods and computer storage media for erasure coding data in a distributed computing environment. A sealed extent is identified that is comprised of two or more data blocks and two or more index blocks. The sealed extent is optimized for erasure coding by grouping the two or more data blocks within the optimized sealed extent together and grouping the two or more index blocks within the optimized sealed extent together. The optimized extent may also be erasure coded, which includes creating data fragments and coding fragments. The data fragments and the coding fragments may also be stored in the distributed computing environment. Additional embodiments include monitoring statistical information to determine if replication, erasure coding or a hybrid storage plan should be utilized.

Abstract: Systems, methods, and computer storage media for load balancing at a client device in a distributed computing environment based on a dynamic validity duration and utilization metrics of one or more data stores are provided. The client device selects a data store from a plurality of data stores that each maintains a copy of a datum. The selection of data stores by a client is based on utilization metrics of the data stores. The utilization metrics are stored at the client. A data store determines if a request for a particular datum may be fulfilled within a validity period associated with a request from the client. The validity period is dynamically changed by the client across multiple requests to aid in the load balancing process. Additionally, a gossip protocol may be used to propagate utilization metrics of a plurality of data stores across the system.

Abstract: Systems, methods, and computer storage media for load balancing at a client device in a distributed computing environment based on a dynamic validity duration and utilization metrics of one or more data stores are provided. The client device selects a data store from a plurality of data stores that each maintains a copy of a datum. The selection of data stores by a client is based on utilization metrics of the data stores. The utilization metrics are stored at the client. A data store determines if a request for a particular datum may be fulfilled within a validity period associated with a request from the client. The validity period is dynamically changed by the client across multiple requests to aid in the load balancing process. Additionally, a gossip protocol may be used to propagate utilization metrics of a plurality of data stores across the system.