Abstract: A distributed data store may implement passive distribution encryption keys to enable access to encrypted data stored in the distributed data store. Keys to encrypt a data volume stored in the distributed data store may be encrypted according to a distribution key and provided to a client of the distributed data store. Storage nodes that maintain portions of the data volume may receive the encrypted key from a client to enable access to the data volume. The storage nodes may decrypt the key according to the distribution key and enable access to the data volume at the storage nodes. In to some embodiments, a key hierarchy may be implemented to encrypt the keys that provide access to the encrypted data. The key hierarchy may include a user key.

Abstract: A database management system may utilize a log-structured storage system instead of a conventional storage system. The log-structured storage system may store a baseline of an object and a series of log entries describing changes to the object. At various positions in the log, a combined record may be written to contain a description of a change to the object and a copy of the corresponding version of the object. Combined records may be used to employ alternative strategies for constructing or transmitting a version of an object. Embodiments may employ these alternative strategies for replication and data caches.

Abstract: A distributed database system may implement dynamic quorum group membership changes. In various embodiments, a quorum set may maintain a replica of a data object among group members according to a protection group policy for the data object. A group member may be identified as to be replaced. In response, a new quorum set may be created from the remaining group members and a new group member. The protection group policy may be updated to include the new group members such that subsequently received updates are maintained at both the previous to quorum set and the new quorum set. Previously received updates may be replicated on the new group member. Upon completion of replicating the previously received updates, the protection group policy for the data object may be revised such that subsequently received updates are maintained at the new quorum set.

Abstract: A distributed database system may implement log-structured distributed storage using a single log sequence number space. A log for a data volume may be maintained in a log-structured distributed storage system. The log may be segmented across multiple protection groups according to a partitioning of user data for the data volume. Updates to the log may be assigned a log sequence number from a log sequence number space for the data volume. A protection group may be determined for an update according to which partition of user data space the update pertains. Metadata to be included with the log record may indicate a previous log sequence number of a log record maintained at the protection group. The log record may be sent to the protection group and identified as committed based on acknowledgments received from storage nodes implementing the protection group.

Abstract: A log-structured distributed storage system may implement individual write quorums. Log records may be sent to different storage nodes of a quorum set storing data for a storage client sufficient to satisfy a write quorum requirement. For each log record, acknowledgments from storage nodes are received, and a determination is made whether the write quorum requirement is satisfied for the log record. Different log records may be maintained at different storage nodes, and still satisfy the write quorum requirement such that in some embodiments no one storage node may maintain all of the log records sent to storage nodes in the quorum set.

Abstract: A distributed database system may implement dynamic quorum group membership changes. In various embodiments, a quorum set may maintain a replica of a data object among group members according to a protection group policy for the data object. A group member may be identified as to be replaced. In response, a new quorum set may be created from the remaining group members and a new group member. The protection group policy may be updated to include the new group members such that subsequently received updates are maintained at both the previous quorum set and the new quorum set. Previously received updates may be replicated on the new group member. Upon completion of replicating the previously received updates, the protection group policy for the data object may be revised such that subsequently received updates are maintained at the new quorum set.

Abstract: Application program data stored in system memory may be selectively persisted. An indication may be provided to an application program that an application data object or a range of application data stored in system memory may be treated as persistent. Data backup may be enabled for the application data object or range of application data in the event of a system failure, copying the application data object or range of application data from system memory to non-volatile data storage. Upon recovery from a system failure, further data backup for the application data object or the range of application data may be disabled. In some embodiments, at least some of the application data object or range of application data may be recovered for the application program to access. Data backup for the application data object or the range of application data may also be re-enabled.

Abstract: A data store maintaining data may implement reducing input/output (I/O) operations for on-demand data page generation. Log records may be maintained for data pages of data describing changes to the data pages. A coalesce operation may be performed when log records for a data page exceed a coalesce threshold for the data page, applying the log records for the data page to a version of the data page and creating a new version that includes the changes indicated by the log records. An indication may be received to increase the coalesce threshold for a particular data page, delaying to a coalesce operation for the data page according to the increased coalesce threshold. The indication may be received from a storage engine that identifies a delay for the particular data page.

Abstract: A data store maintaining data may implement memory-based on-demand data page generation. Log records may be maintained in system memory according to a version threshold for data pages of data stored in persistent storage. When a request for a version of a particular data page is received that is within the version threshold, the data page may be obtained from persistent storage and log records associated with the data page may be applied to the data page to generate the requested version of the data page. The version threshold for the data pages may change, and log records in system memory may be identified for coalesce operations. The coalesce operations may be efficiently performed (e.g., in the background), updating the data pages in persistent storage to include the changes of the identified log records and space in system memory for the identified log records may be reclaimed.

Abstract: A log-structured data store implementing data backup may implement variable data replication. Write requests may be received at different storage nodes maintaining respective replicas of a portion of a log for data maintained in the log-structured data store. Log records indicating the write requests may be stored in the respective replicas of the log portions at the different storage nodes. The log records may be sent to a backup data store to be durability persisted as part of an archived version of the log. At some of the storage nodes, in response to determining that the log records have been durably persisted in the backup data store, storage space for the log records may be reclaimed. In other remaining storage nodes, the log records may be retained and made accessible for servicing read requests.

Abstract: A data store maintaining data may implement reducing input/output (I/O) operations for on-demand data page generation. Log records may be maintained for data pages of data describing changes to the data pages. A coalesce operation may be performed when log records for a data page exceed a coalesce threshold for the data page, applying the log records for the data page to a version of the data page and creating a new version that includes the changes indicated by the log records. An indication may be received to increase the coalesce threshold for a particular data page, delaying a coalesce operation for the data page according to the increased coalesce threshold. The indication may be received from a storage engine that identifies a delay for the particular data page.

Abstract: Hybrid quorum policies may be implemented for access requests to state maintained in a distributed system. An update to state maintained in a distributed system may be sent to multiple nodes in the distributed system maintain the state in order to satisfy a hybrid quorum policy for updating the state in the distributed system. The hybrid quorum policy may include individual quorum policies for multiple quorum sets including nodes of the distributed system. Acknowledgments received from nodes in the distributed system indicating that the update is performed may be evaluated according to the individual quorum policies to determine whether the hybrid quorum policy is satisfied. If the hybrid quorum policy is satisfied, then the update to the state may be considered durable.

Abstract: Self-describing data blocks of a minimum atomic write size may be stored for a data store. Data may be received for storage in a data block at a persistent storage device. Metadata may be generated for the data that includes an error detection code which is generated for the data and the metadata together. An individual atomic write operation may write together the data and the metadata in the data block. When accessed, the error detection code is applicable to detect errors and the metadata may also be applicable to determine whether the data is stored for a currently assigned purpose or a previously assigned purpose of the data block.

Abstract: A data store maintaining data may implement memory-based on-demand data page generation. Log records may be maintained in system memory according to a version threshold for data pages of data stored in persistent storage. When a request for a version of a particular data page is received that is within the version threshold, the data page may be obtained from persistent storage and log records associated with the data page may be applied to the data page to generate the requested version of the data page. The version threshold for the data pages may change, and log records in system memory may be identified for coalesce operations. The coalesce operations may be efficiently performed (e.g., in the background), updating the data pages in persistent storage to include the changes of the identified log records and space in system memory for the identified log records may be reclaimed.

Abstract: A log-structured data store implementing data backup may implement variable data replication. Write requests may be received at different storage nodes maintaining respective replicas of a portion of a log for data maintained in the log-structured data store. Log records indicating the write requests may be stored in the respective replicas of the log portions at the different storage nodes. The log records may be sent to a backup data store to be durability persisted as part of an archived version of the log. At some of the storage nodes, in response to determining that the log records have been durably persisted in the backup data store, storage space for the log records may be reclaimed. In other remaining storage nodes, the log records may be retained and made accessible for servicing read requests.

Abstract: Hybrid quorum policies may be implemented for access requests to state maintained in a distributed system. An update to state maintained in a distributed system may be sent to multiple nodes in the distributed system maintain the state in order to satisfy a hybrid quorum policy for updating the state in the distributed system. The hybrid quorum policy may include individual quorum policies for multiple quorum sets including nodes of the distributed system. Acknowledgments received from nodes in the distributed system indicating that the update is performed may be evaluated according to the individual quorum policies to determine whether the hybrid quorum policy is satisfied. If the hybrid quorum policy is satisfied, then the update to the state may be considered durable.

Abstract: A storage client and a quorum-based distributed storage system may implement dynamic utilization of bandwidth for a quorum-based distributed storage system. An update at a storage client may be received, and storage nodes of a protection group may be sent a write request indicating the update. In some embodiment, storage nodes that receive the write request may determine whether other storage nodes have not received the update and send the write request to be completed at those other storage nodes. In some embodiments, if a latency threshold is exceeded other storage nodes in the protection group not previously sent the write request may be identified and sent the write request. Based on acknowledgements received from storage nodes in the distributed storage system, it may be determined whether a write quorum requirement is met for a write request.

Abstract: A distributed database system may implement log-structured distributed storage using a single log sequence number space. A log for a data volume may be maintained in a log-structured distributed storage system. The log may be segmented across multiple protection groups according to a partitioning of user data for the data volume. Updates to the log may be assigned a log sequence number from a log sequence number space for the data volume. A protection group may be determined for an update according to which partition of user data space the update pertains. Metadata to be included with the log record may indicate a previous log sequence number of a log record maintained at the protection group. The log record may be sent to the protection group and identified as committed based on acknowledgments received from storage nodes implementing the protection group.

Abstract: Self-describing data blocks of a minimum atomic write size may be stored for a data store. Data may be received for storage in a data block of a plurality of data blocks at a persistent storage device that are equivalent to a minimum atomic write size for the persistent storage device. Metadata may be generated for the data that includes an error detection code which is generated for the data and the metadata together. The data and the metadata are sent to the persistent storage to device to store together in the data block. An individual atomic write operation may write together the data and the metadata in the data block. When accessed, the error detection code is applicable to detect errors. The metadata may also be applicable to determine whether the data is stored for a currently assigned purpose or a previously assigned purpose of the data block.

Abstract: A log-structured data store may implement efficient garbage collection. Log records may be maintained in data blocks according to a log record sequence. Based, at least in part, on a log reclamation point, the log records may be evaluated to identify data blocks to reclaim that have log records in the log sequence prior to the log reclamation point. New versions of data pages updated by log records in the identified data blocks may be generated and stored in base page storage for the log structured data store. The identified data blocks may then be reclaimed for storing new data.