Abstract: An append-only streams capability may be implemented that allows the host (e.g., the file system) to determine an optimal stream size based on the data to be stored in that stream. The storage device may expose to the host one or more characteristics of the available streams on the device, including but not limited to the maximum number of inactive and active streams on the device, the erase block size, the maximum number of erase blocks that can be written in parallel, and an optimal write size of the data. Using this information, the host may determine which particular stream offered by the device is best suited for the data to be stored.

Abstract: A system includes determination of a plurality of meta-copysets of a plurality of storage devices, each of the meta-copysets including a first number of storage devices, determination of a first copyset of a second number of storage devices from a first one of the meta-copysets, where the second number is less than the first number, storage of first data in a fault-tolerant format using the first copyset, determination to migrate the first data from a first storage device of the first copyset, and, in response to the determination to migrate the first data, determine a second copyset of the second number of storage devices including a storage device from the first meta-copyset which is not in the first copyset, and the storage devices of the first copyset except for the first storage device of the first copyset and storage of the first data in a fault-tolerant format using the second copyset.

Abstract: A first set of replicated state machines includes a first state machine that compares a clock value included in a state update message incremented by a first amount, a clock value for the first state machine incremented by a second amount, and a current local wall clock value for the first state machine to determine a maximum value and assigns the maximum value as the clock value for the first state machine. Additionally, in response to a passage of an amount of time, the first state machine advances the clock value for the first state machine to its current local wall clock value and propagates this clock value to the other state machines in the first set of replicated state machines. The advancement of the clock value for all state machines even in the absence of state updates improves their ability to respond to distributed read requests.

Abstract: A system includes determination of a plurality of meta-copysets of a plurality of storage devices, each of the meta-copysets including a first number of storage devices, determination of a first copyset of a second number of storage devices from a first one of the meta-copysets, where the second number is less than the first number, storage of first data in a fault-tolerant format using the first copyset, determination to migrate the first data from a first storage device of the first copyset, and, in response to the determination to migrate the first data, determine a second copyset of the second number of storage devices including a storage device from the first meta-copyset which is not in the first copyset, and the storage devices of the first copyset except for the first storage device of the first copyset and storage of the first data in a fault-tolerant format using the second copyset.

Abstract: A first set of replicated state machines includes a first state machine that compares a clock value included in a state update message incremented by a first amount, a clock value for the first state machine incremented by a second amount, and a current local wall clock value for the first state machine to determine a maximum value and assigns the maximum value as the clock value for the first state machine. Additionally, in response to a passage of an amount of time, the first state machine advances the clock value for the first state machine to its current local wall clock value and propagates this clock value to the other state machines in the first set of replicated state machines. The advancement of the clock value for all state machines even in the absence of state updates improves their ability to respond to distributed read requests.

Abstract: In various embodiments, methods and systems for erasure coding with enhanced local protection groups are provided. An erasure coding scheme can be defined based on a Vertical Local Reconstruction Code (VLRC) that achieves high storage efficiency by combining the Local Reconstruction Code and conventional erasure coding, where the local reconstruction code (LRC) is carefully laid out across zones. Thus, when a zone is down, remaining fragments form an appropriate LRC. Further, an inter-zone erasure coding scheme—Zone Local Reconstruction Code (ZZG-2 code)—is provided having both local reconstruction within every zone and a-of-b recovery property across zones. An inter-zone adaptive erasure coding (uber code) scheme is provided, the uber code is configurable to produce near optimal performance in different environments characterized by intra and inter-zone bandwidth and machine failure rates.

Abstract: An append-only streams capability may be implemented that allows the host (e.g., the file system) to determine an optimal stream size based on the data to be stored in that stream. The storage device may expose to the host one or more characteristics of the available streams on the device, including but not limited to the maximum number of inactive and active streams on the device, the erase block size, the maximum number of erase blocks that can be written in parallel, and an optimal write size of the data. Using this information, the host may determine which particular stream offered by the device is best suited for the data to be stored.

Abstract: Streaming functionality may be utilized for optimizing the storage of data on a solid state device. In one embodiment, an append-only streaming method may comprise determining a size of one or more related groups of data, determining a size of one or more erase blocks in a file system, requesting from the file system one or more stream identifiers based on the size of the one or more related groups of data and the size of the one or more erase blocks, requesting from a solid state device and using the one or more stream identifiers an optimal writable space on the solid state device, and writing data to the optimal writable space on the solid state device.

Abstract: In various embodiments, methods and systems for implementing multiple transaction logs in a distributed storage system are provided. A log stream component detects performance metrics of a plurality of log streams. The performance metrics are associated with requests from partitions in the distributed storage system. A transaction component receives a request to execute a transaction using a log stream. The request is received from a partition of the distributed storage system. The performance metrics of the plurality of log streams can be referenced, where the performance metrics indicate a performance capacity of a selected log stream to process the request. A log stream for executing the transaction is determined based on the performance capacity. The log stream selected can also factor request attributes of the request. The transaction component communicates the request to be executed, using the log stream to perform the transaction.

Abstract: In various embodiments, methods and systems for erasure coding with enhanced local protection groups are provided. An erasure coding scheme can be defined based on a Vertical Local Reconstruction Code (VLRC) that achieves high storage efficiency by combining the Local Reconstruction Code and conventional erasure coding, where the local reconstruction code (LRC) is carefully laid out across zones. Thus, when a zone is down, remaining fragments form an appropriate LRC. Further, an inter-zone erasure coding scheme—Zone Local Reconstruction Code (ZZG-2 code)—is provided having both local reconstruction within every zone and a-of-b recovery property across zones. An inter-zone adaptive erasure coding (uber code) scheme is provided, the uber code is configurable to produce near optimal performance in different environments characterized by intra and inter-zone bandwidth and machine failure rates.

Abstract: Systems, methods, and computer storage media for upgrading a domain in a distributed computing environment are provided. Upgrading of the domain includes preparing for the upgrade, upgrading, and finalizing the upgrade. The preparation of the domain includes ensuring predefined quantities of role instances are available in domains other than the upgrade domain. The preparation also includes ensuring that a predefined number of extent replicas are available in domains other than the upgrade domain. The preparation may also include checkpointing partitions within the upgrade domain to facilitate faster loading once transferred to a domain other than the upgrade domain. The finalization may include allowing nodes within the upgrade domain to resume functionality that was suspended during the upgrade.

Abstract: In various embodiments, methods and systems for erasure coding data across multiple storage zones are provided. This may be accomplished by dividing a data chunk into a plurality of sub-fragments. Each of the plurality of sub-fragments is associated with a zone. Zones comprise buildings, data centers, and geographic regions providing a storage service. A plurality of reconstruction parities is computed. Each of the plurality of reconstruction parities computed using at least one sub-fragment from the plurality of sub-fragments. The plurality of reconstruction parities comprises at least one cross-zone parity. The at least one cross-zone parity is assigned to a parity zone. The cross-zone parity provides cross-zone reconstruction of a portion of the data chunk.

Abstract: In various embodiments, methods and systems for implementing multiple transaction logs in a distributed storage system are provided. A log stream component detects performance metrics of a plurality of log streams. The performance metrics are associated with requests from partitions in the distributed storage system. A transaction component receives a request to execute a transaction using a log stream. The request is received from a partition of the distributed storage system. The performance metrics of the plurality of log streams can be referenced, where the performance metrics indicate a performance capacity of a selected log stream to process the request. A log stream for executing the transaction is determined based on the performance capacity. The log stream selected can also factor request attributes of the request. The transaction component communicates the request to be executed, using the log stream to perform the transaction.

Abstract: In various embodiments, methods and systems for erasure coding data across multiple storage zones are provided. This may be accomplished by dividing a data chunk into a plurality of sub-fragments. Each of the plurality of sub-fragments is associated with a zone. Zones comprise buildings, data centers, and geographic regions providing a storage service. A plurality of reconstruction parities is computed. Each of the plurality of reconstruction parities computed using at least one sub-fragment from the plurality of sub-fragments. The plurality of reconstruction parities comprises at least one cross-zone parity. The at least one cross-zone parity is assigned to a parity zone. The cross-zone parity provides cross-zone reconstruction of a portion of the data chunk.

Abstract: In various embodiments, methods and systems for erasure coding data across multiple storage zones are provided. This may be accomplished by dividing a data chunk into a plurality of sub-fragments. Each of the plurality of sub-fragments is associated with a zone. Zones comprise buildings, data centers, and geographic regions providing a storage service. A plurality of reconstruction parities is computed. Each of the plurality of reconstruction parities computed using at least one sub-fragment from the plurality of sub-fragments. The plurality of reconstruction parities comprises at least one cross-zone parity. The at least one cross-zone parity is assigned to a parity zone. The cross-zone parity provides cross-zone reconstruction of a portion of the data chunk.

Abstract: In various embodiments, methods and systems for erasure coding data across multiple storage zones are provided. This may be accomplished by dividing a data chunk into a plurality of sub-fragments. Each of the plurality of sub-fragments is associated with a zone. Zones comprise buildings, data centers, and geographic regions providing a storage service. A plurality of reconstruction parities is computed. Each of the plurality of reconstruction parities computed using at least one sub-fragment from the plurality of sub-fragments. The plurality of reconstruction parities comprises at least one cross-zone parity. The at least one cross-zone parity is assigned to a parity zone. The cross-zone parity provides cross-zone reconstruction of a portion of the data chunk.

Abstract: Systems, methods, and computer storage media for upgrading a domain in a distributed computing environment are provided. Upgrading of the domain includes preparing for the upgrade, upgrading, and finalizing the upgrade. The preparation of the domain includes ensuring predefined quantities of role instances are available in domains other than the upgrade domain. The preparation also includes ensuring that a predefined number of extent replicas are available in domains other than the upgrade domain. The preparation may also include checkpointing partitions within the upgrade domain to facilitate faster loading once transferred to a domain other than the upgrade domain. The finalization may include allowing nodes within the upgrade domain to resume functionality that was suspended during the upgrade.

Abstract: Systems, methods, and computer storage media for upgrading a domain in a distributed computing environment are provided. Upgrading of the domain includes preparing for the upgrade, upgrading, and finalizing the upgrade. The preparation of the domain includes ensuring predefined quantities of role instances are available in domains other than the upgrade domain. The preparation also includes ensuring that a predefined number of extent replicas are available in domains other than the upgrade domain. The preparation may also include checkpointing partitions within the upgrade domain to facilitate faster loading once transferred to a domain other than the upgrade domain. The finalization may include allowing nodes within the upgrade domain to resume functionality that was suspended during the upgrade.

Abstract: Systems, methods, and computer storage media for storing and retrieving data from a data store in a distributed computing environment are provided. An embodiment includes receiving data at a data store comprising a sequential journal store, RAM, and a non-sequential target store. When RAM utilization is below a threshold, received data is stored to the RAM as a write cache for the target store and the journal store. But, when the utilization is above the threshold, the data is stored to the journal store without write-caching to the RAM for the target store. When the RAM utilization falls below a threshold, data committed to the journal store, but not write-cached to the RAM for the target store, is later read from the journal store and write-cached to the RAM for a target store.

Abstract: Systems, methods, and computer storage media for load balancing at a client device in a distributed computing environment based on a dynamic validity duration and utilization metrics of one or more data stores are provided. The client device selects a data store from a plurality of data stores that each maintains a copy of a datum. The selection of data stores by a client is based on utilization metrics of the data stores. The utilization metrics are stored at the client. A data store determines if a request for a particular datum may be fulfilled within a validity period associated with a request from the client. The validity period is dynamically changed by the client across multiple requests to aid in the load balancing process. Additionally, a gossip protocol may be used to propagate utilization metrics of a plurality of data stores across the system.