Abstract: Disclosed are various embodiments for distributing data items within a plurality of nodes. A data item that is subject to a data item update request is updated from a master node to a plurality of slave notes. The update of the data item is determined to be locality-based durable based at least in part on acknowledgements received from the slave nodes. Upon detection that the master node has failed, a new master candidate is determined via an election among the plurality of slave nodes.

Abstract: A stateful cluster may implement scaling of the stateful cluster while maintaining access to the state of the stateful cluster. A scaling event for a stateful cluster may be detected, and in response the stateful cluster may be adjusted to include a different number of nodes. The state of the cluster may then be logically distributed among the different number of nodes according to a monotone distribution scheme. The adjusted node may then service access requests according to the monotone distribution scheme. Prior to making the adjusted storage cluster available for servicing access requests, the nodes from the original cluster may still service access requests for state.

Abstract: Disclosed are various embodiments for distributing data items within a plurality of nodes. A data item that is subject to a data item update request is updated from a master node to a plurality of slave notes. The update of the data item is determined to be locality-based durable based at least in part on acknowledgements received from the slave nodes. Upon detection that the master node has failed, a new master candidate is determined via an election among the plurality of slave nodes.

Abstract: A system that implements a data storage service may store data on behalf of storage service clients. The system may maintain data in multiple replicas of various partitions that are stored on respective computing nodes in the system. The system may employ a single master failover protocol, usable when a replica attempts to become the master replica for a replica group of which it is a member. Attempting to become the master replica may include acquiring a lock associated with the replica group, and gathering state information from the other replicas in the group. The state information may indicate whether another replica supports the attempt (in which case it is included in a failover quorum) or stores more recent data or metadata than the replica attempting to become the master (in which case synchronization may be required). If the failover quorum includes enough replicas, the replica may become the master.

Abstract: Incremental, out-of-place updates may be implemented for index structures maintained for data stores. Portions of the index structure may be selected for updating, and an updated version of the portion of the index structure generated in another storage location different than a current storage location for the index structure such that the index structure may be searched in order to perform access requests. Updating the portion of the index structure may include compacting the portion of the index structure and/or merging the portion of the index structure with a sub-index structure generated from a portion of a log of index updates that may be maintained. The current portion of the index structure may then be replaced with the updated version of the current portion so that the updated version may be evaluated when searches of the index structure are performed.

Abstract: A balanced distribution of sort order values may be implemented for a multi-column sort order of a database table. Columns of the database table to be included in the multi-column sort order may be identified. Some columns containing string data values may be converted to equally-sized integer data values. The data values of columns may be evaluated to determine buckets representing the ranges of data values within the columns for depth-balanced histograms of the columns. Multi-column sort order values may be generated for individual entries in the database table according to bucket values assigned to the buckets that include the columns values of the individual entries. The entries of the database table may then be stored according to a sorted ordering of multi-column sort order values for the entries.

Abstract: A data store may implement incremental out-of-place updates to a dataset. A dataset may maintain data across different storage locations linked together according to an ordering schema for servicing queries. As updates to the dataset are received, the updates may be persisted but not maintained in-place. In order to update the data store and maintain the ordering schema, incremental updates to the dataset may be performed without blocking queries directed toward the dataset. The dataset may be divided into multiple data chunks that correspond to different storage locations and an updated version of the data chunk may be generated in new storage locations. The new storage locations may then replace the storage locations of the prior version of the data chunk in order to link the new storage locations to the other linked storage locations in the dataset for servicing queries.

Abstract: Incremental, out-of-place updates may be implemented for index structures maintained for data stores. Portions of the index structure may be selected for updating, and an updated version of the portion of the index structure generated in another storage location different than a current storage location for the index structure such that the index structure may be searched in order to perform access requests. Updating the portion of the index structure may include compacting the portion of the index structure and/or merging the portion of the index structure with a sub-index structure generated from a portion of a log of index updates that may be maintained. The current portion of the index structure may then be replaced with the updated version of the current portion so that the updated version may be evaluated when searches of the index structure are performed.

Abstract: A system that implements a data storage service may store data on behalf of storage service clients. The system may maintain data in multiple replicas of various partitions that are stored on respective computing nodes in the system. The system may employ a single master failover protocol, usable when a replica attempts to become the master replica for a replica group of which it is a member. Attempting to become the master replica may include acquiring a lock associated with the replica group, and gathering state information from the other replicas in the group. The state information may indicate whether another replica supports the attempt (in which case it is included in a failover quorum) or stores more recent data or metadata than the replica attempting to become the master (in which case synchronization may be required). If the failover quorum includes enough replicas, the replica may become the master.

Abstract: A system that implements a data storage service may store data on behalf of storage service clients. The system may maintain data in multiple replicas of partitions that are stored on respective computing nodes in the system. A master replica for a replica group may increment a membership version indicator for the group, and may propagate metadata (including the membership version indicator) indicating a membership change for the group to other members of the group. Propagating the metadata may include sending a log record containing the metadata to the other replicas to be appended to their respective logs. Once the membership change becomes durable, it may be committed. A replica attempting to become the master of a replica group may determine that another replica in the group has observed a more recent membership version, in which case logs may be synchronized or snipped, or the attempt may be abandoned.

Abstract: A distributed data warehouse system maintains data blocks on behalf of clients, and stores primary and secondary copies of data blocks on different disks or nodes in a cluster. The data warehouse system may back up data blocks in a key-value backup storage system. In response to a query targeting a data block previously stored in the cluster, the data warehouse system may determine whether a consistent, uncorrupted copy of the data block is available in the cluster (e.g., by applying a consistency check). If not (e.g., if a disk or node failed), the data warehouse system may automatically initiate an operation to restore the data block from the backup storage system, using a unique identifier of the data block to access a backup copy. The target data may be returned in a query response prior to restoring primary and secondary copies of the data block in the cluster.

Abstract: A distributed data warehouse system maintains data blocks on behalf of clients, and stores primary and secondary copies of data blocks on different disks or nodes in a cluster. The data warehouse system may back up data blocks in a key-value backup storage system. In response to a query targeting a data block previously stored in the cluster, the data warehouse system may determine whether a consistent, uncorrupted copy of the data block is available in the cluster (e.g., by applying a consistency check). If not (e.g., if a disk or node failed), the data warehouse system may automatically initiate an operation to restore the data block from the backup storage system, using a unique identifier of the data block to access a backup copy. The target data may be returned in a query response prior to restoring primary and secondary copies of the data block in the cluster.

Abstract: A system that implements a data storage service may store data on behalf of storage service clients. The system may maintain data in multiple replicas of partitions that are stored on respective computing nodes in the system. A master replica for a replica group may increment a membership version indicator for the group, and may propagate metadata (including the membership version indicator) indicating a membership change for the group to other members of the group. Propagating the metadata may include sending a log record containing the metadata to the other replicas to be appended to their respective logs. Once the membership change becomes durable, it may be committed. A replica attempting to become the master of a replica group may determine that another replica in the group has observed a more recent membership version, in which case logs may be synchronized or snipped, or the attempt may be abandoned.

Abstract: A system that implements a scalable data storage service may maintain tables in a non-relational data store on behalf of clients. The system may provide a Web services interface through which service requests are received, and an API usable to request that a table be created, deleted, or described; that an item be stored, retrieved, deleted, or its attributes modified; or that a table be queried (or scanned) with filtered items and/or their attributes returned. An asynchronous workflow may be invoked to create or delete a table. Items stored in tables may be partitioned and indexed using a simple or composite primary key. The system may not impose pre-defined limits on table size, and may employ a flexible schema. The service may provide a best-effort or committed throughput model. The system may automatically scale and/or re-partition tables in response to detecting workload changes, node failures, or other conditions or anomalies.

Abstract: Disclosed are various embodiments for distributing data items. A plurality of nodes forms a distributed data store. A new master candidate is determined through an election among the plurality of nodes. Before performing a failover from a failed master to the new master candidate, a consensus is reached among a locality-based failover quorum of the nodes. The quorum excludes any of the nodes that are in a failover quorum ineligibility mode.

Abstract: Techniques for producing a gentle reduction in throughput in a distributed service when a node of the service encounters a very large backlog of requests and/or when a previously offline node of the service is brought back online. These techniques may utilize multiple different algorithms to determine an amount of work that the distributed service is able to accept at any given time, rather than a single algorithm.

Abstract: A system that implements a scalable data storage service may maintain tables in a non-relational data store on behalf of clients. The system may provide a Web services interface through which service requests are received, and an API usable to request that a table be created, deleted, or described; that an item be stored, retrieved, deleted, or its attributes modified; or that a table be queried (or scanned) with filtered items and/or their attributes returned. An asynchronous workflow may be invoked to create or delete a table. Items stored in tables may be partitioned and indexed using a simple or composite primary key. The system may not impose pre-defined limits on table size, and may employ a flexible schema. The service may provide a best-effort or committed throughput model. The system may automatically scale and/or re-partition tables in response to detecting workload changes, node failures, or other conditions or anomalies.

Abstract: Disclosed are various embodiments for distributing data items. A plurality of nodes forms a distributed data store. A new master candidate is determined through an election among the plurality of nodes. Before performing a failover from a failed master to the new master candidate, a consensus is reached among a locality-based failover quorum of the nodes. The quorum excludes any of the nodes that are in a failover quorum ineligibility mode.

Abstract: A distributed database system may perform a single phase commit for transactions involving updates to multiple databases of the distributed database system. A client request may be received that involves updates to multiple database of the distributed database system. The updates may be performed at a front-end database and a back-end database. Log records indicating the updates to the front-end database may be sent to the back-end database. The log records and the updates performed at the back-end database may be committed together as a single phase commit at the back-end database. In the event of a system failure of the front-end database, log records may be requested and received from the back-end database. A restoration of the front-end database may be performed based, at least in part, on the received log records.

Abstract: Disclosed are various embodiments for distributing data items within a plurality of nodes. A data item that is subject to a data item update request is updated from a master node to a plurality of slave notes. The update of the data item is determined to be locality-based durable based at least in part on acknowledgements received from the slave nodes. Upon detection that the master node has failed, a new master candidate is determined via an election among the plurality of slave nodes.