Abstract: In various embodiments, methods and systems for erasure coding with enhanced local protection groups are provided. An erasure coding scheme can be defined based on a Vertical Local Reconstruction Code (VLRC) that achieves high storage efficiency by combining the Local Reconstruction Code and conventional erasure coding, where the local reconstruction code (LRC) is carefully laid out across zones. Thus, when a zone is down, remaining fragments form an appropriate LRC. Further, an inter-zone erasure coding scheme—Zone Local Reconstruction Code (ZZG-2 code)—is provided having both local reconstruction within every zone and a-of-b recovery property across zones. An inter-zone adaptive erasure coding (uber code) scheme is provided, the uber code is configurable to produce near optimal performance in different environments characterized by intra and inter-zone bandwidth and machine failure rates.

Abstract: In various embodiments, methods and systems for implementing multiple transaction logs in a distributed storage system are provided. A log stream component detects performance metrics of a plurality of log streams. The performance metrics are associated with requests from partitions in the distributed storage system. A transaction component receives a request to execute a transaction using a log stream. The request is received from a partition of the distributed storage system. The performance metrics of the plurality of log streams can be referenced, where the performance metrics indicate a performance capacity of a selected log stream to process the request. A log stream for executing the transaction is determined based on the performance capacity. The log stream selected can also factor request attributes of the request. The transaction component communicates the request to be executed, using the log stream to perform the transaction.

Abstract: In various embodiments, methods and systems for transferring data using a storage medium are provided. A storage medium may be shipped by a customer to a datacenter such that the data on the storage medium is copied to a storage associated with the datacenter or data in the storage is copied to the storage medium. The datacenter may support a cloud computing infrastructure that provides a storage account to the customer that is associated with the data copied from or copied to the storage medium. The storage medium further corresponds to a data transfer manifest that includes at least in part data mapping between storage service infrastructure and data in the storage medium. It is contemplated that embodiments of the present invention may further be implemented with data transfer service components that support a client component, storage service component, and a data transfer management component.

Abstract: In various embodiments, methods and systems for erasure coding with enhanced local protection groups are provided. An erasure coding scheme can be defined based on a Vertical Local Reconstruction Code (VLRC) that achieves high storage efficiency by combining the Local Reconstruction Code and conventional erasure coding, where the local reconstruction code (LRC) is carefully laid out across zones. Thus, when a zone is down, remaining fragments form an appropriate LRC. Further, an inter-zone erasure coding scheme—Zone Local Reconstruction Code (ZZG-2 code)—is provided having both local reconstruction within every zone and a-of-b recovery property across zones. An inter-zone adaptive erasure coding (uber code) scheme is provided, the uber code is configurable to produce near optimal performance in different environments characterized by intra and inter-zone bandwidth and machine failure rates.

Abstract: In various embodiments, methods and systems for erasure coding data across multiple storage zones are provided. This may be accomplished by dividing a data chunk into a plurality of sub-fragments. Each of the plurality of sub-fragments is associated with a zone. Zones comprise buildings, data centers, and geographic regions providing a storage service. A plurality of reconstruction parities is computed. Each of the plurality of reconstruction parities computed using at least one sub-fragment from the plurality of sub-fragments. The plurality of reconstruction parities comprises at least one cross-zone parity. The at least one cross-zone parity is assigned to a parity zone. The cross-zone parity provides cross-zone reconstruction of a portion of the data chunk.

Abstract: In various embodiments, methods and systems for implementing multiple transaction logs in a distributed storage system are provided. A log stream component detects performance metrics of a plurality of log streams. The performance metrics are associated with requests from partitions in the distributed storage system. A transaction component receives a request to execute a transaction using a log stream. The request is received from a partition of the distributed storage system. The performance metrics of the plurality of log streams can be referenced, where the performance metrics indicate a performance capacity of a selected log stream to process the request. A log stream for executing the transaction is determined based on the performance capacity. The log stream selected can also factor request attributes of the request. The transaction component communicates the request to be executed, using the log stream to perform the transaction.

Abstract: In various embodiments, methods and systems for erasure coding data across multiple storage zones are provided. This may be accomplished by dividing a data chunk into a plurality of sub-fragments. Each of the plurality of sub-fragments is associated with a zone. Zones comprise buildings, data centers, and geographic regions providing a storage service. A plurality of reconstruction parities is computed. Each of the plurality of reconstruction parities computed using at least one sub-fragment from the plurality of sub-fragments. The plurality of reconstruction parities comprises at least one cross-zone parity. The at least one cross-zone parity is assigned to a parity zone. The cross-zone parity provides cross-zone reconstruction of a portion of the data chunk.

Abstract: In various embodiments, methods and systems for transferring data using a storage medium are provided. A storage medium may be shipped by a customer to a datacenter such that the data on the storage medium is copied to a storage associated with the datacenter or data in the storage is copied to the storage medium. The datacenter may support a cloud computing infrastructure that provides a storage account to the customer that is associated with the data copied from or copied to the storage medium. The storage medium further corresponds to a data transfer manifest that includes at least in part data mapping between storage service infrastructure and data in the storage medium. It is contemplated that embodiments of the present invention may further be implemented with data transfer service components that support a client component, storage service component, and a data transfer management component.

Abstract: In various embodiments, methods and systems for erasure coding data across multiple storage zones are provided. This may be accomplished by dividing a data chunk into a plurality of sub-fragments. Each of the plurality of sub-fragments is associated with a zone. Zones comprise buildings, data centers, and geographic regions providing a storage service. A plurality of reconstruction parities is computed. Each of the plurality of reconstruction parities computed using at least one sub-fragment from the plurality of sub-fragments. The plurality of reconstruction parities comprises at least one cross-zone parity. The at least one cross-zone parity is assigned to a parity zone. The cross-zone parity provides cross-zone reconstruction of a portion of the data chunk.

Abstract: In various embodiments, methods and systems for erasure coding data across multiple storage zones are provided. This may be accomplished by dividing a data chunk into a plurality of sub-fragments. Each of the plurality of sub-fragments is associated with a zone. Zones comprise buildings, data centers, and geographic regions providing a storage service. A plurality of reconstruction parities is computed. Each of the plurality of reconstruction parities computed using at least one sub-fragment from the plurality of sub-fragments. The plurality of reconstruction parities comprises at least one cross-zone parity. The at least one cross-zone parity is assigned to a parity zone. The cross-zone parity provides cross-zone reconstruction of a portion of the data chunk.