Abstract: A data storage system includes multiple head nodes and data storage sleds. Volume data is replicated between a primary and one or more secondary head nodes for a volume partition and is further flushed to a set of mass storage devices of the data storage sleds. Volume metadata is maintained in a primary and one or more secondary head nodes for a volume partition and is updated in response to volume data being flushed to the data storage sleds. Also, the primary and secondary head nodes store check-points of volume metadata to the data storage sleds, wherein in response to a failure of a primary or secondary head node for a volume partition, a replacement secondary head node for the volume partition recreates a secondary replica for the volume partition based, at least in part, on a stored volume metadata checkpoint.

Abstract: Techniques for transferring data between a host system and a remote system over a network are disclosed. A large command with associated data to be transmitted is divided into segments. Each segment is encapsulated in a network transport unit that includes a corresponding segment identifier for the segment. The network transport units for the segments are then submitted for transmission between the host system and the remote system using multiple network paths. A system receiving the network transport units for the segments reassembles the command with associated data based on the corresponding segment identifiers in the network transport units.

Abstract: A slave storage is provisioned using metadata of a master B-tree and updates to references (e.g., offsets) pertaining to data operations of the master B-tree. Master-slave pairs can be used to provide data redundancy, and a master copy can include the master B-tree with references to corresponding data. When provisioning a slave copy, the master sends a B-tree copy to the slave, which stores the slave B-tree copy, allocates the necessary space on local storage, and updates respective offsets of the slave B-tree copy to point to the local storage. Data from the master can then be transferred to the slave and stored according to a note and commit process that ensures operational sequence of the data. Operations received to the master during the process can be committed to the slave copy until the slave is consistent with the master and able to take over as master in the event of a failure.

Abstract: Systems and methods for provisioning a slave copy for redundant data storage and for writing data to persistent storage in a block-based storage system using sequential operation numbers are provided. In one embodiment, the method includes maintaining a master copy and a slave copy of a data volume, the master copy including data generated by a plurality of operations having respective sequential operation numbers, receiving a write instruction for second data to be added to the master copy, and recording the second data as a note that is not readable. The method may further include sending a copy of the note from the master copy to the slave copy, committing the note to the master copy with a sequential operation number, and committing the copy of the note to the slave copy based in part on the sequential operation number. A B-tree may be created based at least in part on an offset for a write instruction associated with the second data, a length, and an operation number included in the note.

Abstract: A data storage system stores information indicating a determined sequence for performing operations on a data store. A lock is acquired on a portion of the data store. It is determined that performing the operations comprises performing at least one additional operation on the data store. Uncommitted changes implied by the operations are stored in a transaction buffer according to the determined sequence. Changes implied by the additional operation are determined based on a reentrant call to a data store interface. The logged sequence of changes is applied to the data store and the lock is released.

Abstract: Data replication groups may be used to store data in a distributed computing environment. A data replication group may include a set of nodes executing a consensus protocol to maintain data durably. In order to increase efficiency and performance of the data replication, a particular node of the data replication group may be assigned the role of master node. The role of master node may be lease in accordance with a consensus protocol. If the lease is not renewed within an interval of time election/selection of a new master node may be commenced.

Abstract: A data storage system includes a head node and mass storage devices. The head node is configured to flush data stored in a storage of the head node, based at least in part on one or more triggers being met, from the storage of the head node to a set of the mass storage devices of the data storage system. The flushed data is written to a segment of free storage space across the set of the mass storage devices allocated for the given data flush operation. In some embodiments, a head node may flush both current version data and point-in-time version data to the set of mass storage devices. Also, the data storage system maintains an index that indicates storage locations of data for particular portions of a volume before and after the data is flushed to the set of mass storage devices.

Abstract: A method for processing packet data in a service provider environment includes, by a network-enabled data processing device within a server computer of the service provider environment, receive packet data including header information and payload information. The packet header information can be separated from the payload information. The separated header information can be forwarded to a processor of the server computer for processing, without forwarding at least a portion of the payload information. Transforming instructions and at least one address of one or more storage locations can be received from the processor, based at least in part on the header information. The payload information can be transformed based on the transforming instructions. The transformed payload information can be stored in the one or more storage locations based on the at least one address.

Abstract: Burst throttling methods may be used to manage computing resources of a data storage service. Tokens may represent I/O operations executed by a customer of the data storage service. A first token bucket may contain a set of tokens representing the overall I/O operation capacity of the data storage service. Additionally, a second token bucket may contain a set of tokens for a given logical volume maintained by the data storage service. When I/O requests are received tokens may be charged the first token bucket and the second token bucket. Furthermore, if there is sufficient capacity, the data storage service may charge a reduced number of tokens to the third token bucket.

Abstract: A data storage system includes multiple head nodes and data storage sleds. A control plane of the data storage system designates, for a volume partition, one of the head nodes to function as a primary head node storing a primary replica of the volume partition and designates two or more other head nodes to function as reserve head nodes storing reserve replicas of the volume partition. Additionally, the primary head node causes volume data for the volume partition to be erasure encoded and stored on multiple mass storage devices in different ones of the data storage sleds.

Abstract: Master-slave pairs can be used to provide data redundancy in an electronic data environment. A master peer can include a B-tree with references to the corresponding data. When provisioning a slave, the master can send a point-in-time copy of the B-tree to the slave, which can allocate the necessary space on local storage and update the references of the B-tree to point to the local storage for the slave. If the master and slave become disconnected, one of the peers can function as a solo master until the peers are again connected, at which point the old peer can be brought current or a new slave provisioned. A log peer can also be provisioned by a solo master, which can store data for operations received during the disconnect for use in catching up a slave peer, which could be the old slave, the log peer, or a new peer.

Abstract: A method for processing packet data in a service provider environment includes, by a network-enabled data processing device within a server computer of the service provider environment, receiving packet data comprising header information and payload information. The header information is separated from the payload information. The separated header information is forwarded to a processor of the server computer for processing, without forwarding at least a portion of the payload information. At least one address of a storage location is received from the processor. The at least one address is associated with a logical-to-physical address mapping based on the header information. The payload information is stored in the storage location based on the at least one address.

Abstract: Computing resource services have computing resources allocated among regions and data zones, where regions can be completely separate deployments of an infrastructure to provide the highest degree of independence. The more regions that are utilized by a customer, the higher the level of fault tolerance may exist. Current networks consist of heavily distributed infrastructures or mainly centralized infrastructures, both of which can cause complete control plane failures when even a subset of a data zone or region loses network connectivity. The placement of control plane and data plane resources provides for optimal availability and durability during infrastructure failures.

Abstract: Burst throttling methods may be used to manage computing resources of a data storage service. Tokens may represent I/O operations executed by a customer of the data storage service. A first token bucket may contain a set of tokens representing the overall I/O operation capacity of the data storage service. Additionally, a second token bucket may contain a set of tokens for a given logical volume maintained by the data storage service. When I/O requests are received tokens may be charged the first token bucket and the second token bucket. Furthermore, if there is sufficient capacity, the data storage service may charge a reduced number of tokens to the third token bucket.

Abstract: Data replication groups may be used to store data in a distributed computing environment. A data replication group may include a set of nodes executing a consensus protocol to maintain data durably. In order to increase efficiency and performance of the data replication, a particular node of the data replication group may be assigned the role of master node. The role of master node may be lease in accordance with a consensus protocol. If the lease is not renewed within an interval of time election/selection of a new master node may be commenced.

Abstract: The present disclosure provides persistent storage for a master copy using operation numbers. A master copy can include a B-tree with references to corresponding data. When provisioning a slave copy, the master copy sends a point-in-time copy of the B-tree to the slave copy, which stores a copy of the B-tree, allocates the necessary space, and updates the references of the B-tree to point to a local storage before the data is transferred. When writing the data to persistent storage, a snapshot created on the master copy is an operation that is replicated to the slave copy. The snapshot is generated using a volume view that includes changes to chunks of data of the master copy since a previous snapshot, as determined using the operation number for the previous snapshot. Data (and metadata) for the snapshot is written to persistent storage while new I/O operations are processed.

Abstract: Persistent storage for a master copy is provided using operation numbers. A master copy can include a persistent key-value store such as a B-tree with references to corresponding data. When provisioning a slave copy, the master copy sends a point-in-time copy of the B-tree to the slave copy, which stores a copy of the B-tree, allocates the necessary space, and updates the references of the B-tree to point to a local storage before the data is transferred. When writing the data to persistent storage, a snapshot created on the master copy is an operation that is replicated to the slave copy. The snapshot is generated using a volume view that includes changes to chunks of data of the master copy since a previous snapshot, as determined using the operation number for the previous snapshot. Data (and metadata) for the snapshot is written to persistent storage while new input/output operations are processed.

Abstract: Server computers may include one or more input/output (I/O) adapter devices for communicating with a network and/or direct-attached device. The I/O adapter device may implement processes to manage write requests in a general and flexible manner. The I/O adapter device may also implement processes to manage write requests in a fast an efficient—that is, low latency—manner. Low latency write requests processes may include determining that a write packet for a write request can be processed without additional assistance from a processor, once a processor has initiated a memory access request to fetch write data and also generated protocol information for transmitting the write packet. The I/O adapter device may then process and transmit the write packet through an offload pipeline, without interrupting a processor.

Abstract: A data storage system includes a head node and mass storage devices. The head node is configured to flush data stored in a storage of the head node, based at least in part on one or more triggers being met, from the storage of the head node to a set of the mass storage devices of the data storage system. The flushed data is written to a segment of free storage space across the set of the mass storage devices allocated for the given data flush operation. In some embodiments, a head node may flush both current version data and point-in-time version data to the set of mass storage devices. Also, the data storage system maintains an index that indicates storage locations of data for particular portions of a volume before and after the data is flushed to the set of mass storage devices.

Abstract: A data storage system includes a head node and mass storage devices. The head node is configured to flush data stored in a storage of the head node to a set of the mass storage devices of the data storage system. A head node may flush both current version data and point-in-time version data to the set of mass storage devices. Also, the data storage system maintains an index that indicates storage locations of data for particular portions of a volume before and after the data is flushed to the set of mass storage devices. In some embodiments an index includes a current version reference for a volume or portion of a volume and one or more point-in-time snapshot references.