Abstract: A database system may include a database service and a separate distributed storage service. The database service (or a database engine head node thereof) may be responsible for query parsing, optimization, and execution, transactionality, and consistency, while the storage service may be responsible for generating data pages from redo log records and for durability of those data pages. For example, in response to a write request directed to a particular data page, the database engine head node may generate a redo log record and send it, but not the data page, to a storage service node. The storage service node may store the redo log record and return a write acknowledgement to the database service prior to applying the redo log record. The server node may apply the redo log record and other redo log records to a previously stored version of the data page to create a current version.

Abstract: A database system may maintain a plurality of log records at a distributed storage system. Each of the plurality of log records may be associated with a respective change to a data page. The plurality of log records may be transformed (e.g., cropped, prune, reduce, fused, deleted, merged, added, etc.).

Abstract: Self-describing data blocks of a minimum atomic write size may be stored for a data store. Data may be received for storage in a data block of a plurality of data blocks at a persistent storage device that are equivalent to a minimum atomic write size for the persistent storage device. Metadata may be generated for the data that includes an error detection code which is generated for the data and the metadata together. The data and the metadata are sent to the persistent storage to device to store together in the data block. An individual atomic write operation may write together the data and the metadata in the data block. When accessed, the error detection code is applicable to detect errors. The metadata may also be applicable to determine whether the data is stored for a currently assigned purpose or a previously assigned purpose of the data block.

Abstract: A log-structured data store may implement efficient garbage collection. Log records may be maintained in data blocks according to a log record sequence. Based, at least in part, on a log reclamation point, the log records may be evaluated to identify data blocks to reclaim that have log records in the log sequence prior to the log reclamation point. New versions of data pages updated by log records in the identified data blocks may be generated and stored in base page storage for the log structured data store. The identified data blocks may then be reclaimed for storing new data.

Abstract: A distributed database system may implement log-structured distributed storage using a single log sequence number space. A log for a data volume may be maintained in a log-structured distributed storage system. The log may be segmented across multiple protection groups according to a partitioning of user data for the data volume. Updates to the log may be assigned a log sequence number from a log sequence number space for the data volume. A protection group may be determined for an update according to which partition of user data space the update pertains. Metadata to be included with the log record may indicate a previous log sequence number of a log record maintained at the protection group. The log record may be sent to the protection group and identified as committed based on acknowledgments received from storage nodes implementing the protection group.

Abstract: Data temporarily stored in volatile memory (e.g., RAM) on a host machine can be protected using a component such as an NV-DIMM, which includes components such as an ASIC, non-volatile memory, and a battery. If power is lost to the host, the battery provides the ASIC with the power needed to determine data in the volatile memory that is protected. This protected data then can be transferred to the non-volatile memory on the NV-DIMM. When power is restored, an application or other entity can contact the NV-DIMM to recover the data, which can be transferred over a sideband channel to be restored as appropriate for a prior operation. In at least some embodiments, the NV-DIMM can receive a key over the sideband channel that can be used to encrypt and decrypt the data for further security.

Abstract: A network-based services provider may reserve and provision primary resource instance capacity for a given service (e.g., enough compute instances, storage instances, or other virtual resource instances to implement the service) in one or more availability zones, and may designate contingency resource instance capacity for the service in another availability zone (without provisioning or reserving the contingency instances for the exclusive use of the service). For example, the service provider may provision resource instance(s) for a database engine head node in one availability zone and designate resource instance capacity for another database engine head node in another availability zone without instantiating the other database engine head node. While the service operates as expected using the primary resource instance capacity, the contingency resource capacity may be leased to other entities on a spot market. Leases for contingency instance capacity may be revoked when needed for the given service (e.g.

Abstract: A database system may include a database service and a separate distributed storage service. The database service (or a database engine head node thereof) may be responsible for query parsing, optimization, and execution, transactionality, and consistency, while the storage service may be responsible for generating data pages from redo log records and for durability of those data pages. For example, in response to a write request directed to a particular data page, the database engine head node may generate a redo log record and send it, but not the data page, to a storage service node. The storage service node may store the redo log record and return a write acknowledgement to the database service prior to applying the redo log record. The server node may apply the redo log record and other redo log records to a previously stored version of the data page to create a current version.

Abstract: A distributed database system may efficiently replicate distributed storage system changes on one or more read-only nodes. Storage metadata that indicates an access scheme for obtaining data for a particular view of the database from a distributed storage system may be maintained at a read-only node. An update to the storage metadata may be received that indicates a change to the distributed storage system that affects the validity of read requests for a different particular view of the database. In some embodiments, the change may indicate which storage nodes may be storing data according to the different particular view of the database, such as a change in membership of a protection group for the data at the distributed storage system. The storage metadata at the read-only node may be updated according to the received update such that a subsequent read request for the different particular view of the database is sent to the distributed storage system according to the updated access scheme.

Abstract: A database system may maintain a plurality of log records at a distributed storage system. Each of the plurality of log records may be associated with a respective change to a data page. The plurality of log records may be transformed (e.g., cropped, prune, reduce, fused, deleted, merged, added, etc.).

Abstract: Systems, methods, and computer storage media for upgrading a domain in a distributed computing environment are provided. Upgrading of the domain includes preparing for the upgrade, upgrading, and finalizing the upgrade. The preparation of the domain includes ensuring predefined quantities of role instances are available in domains other than the upgrade domain. The preparation also includes ensuring that a predefined number of extent replicas are available in domains other than the upgrade domain. The preparation may also include checkpointing partitions within the upgrade domain to facilitate faster loading once transferred to a domain other than the upgrade domain. The finalization may include allowing nodes within the upgrade domain to resume functionality that was suspended during the upgrade.

Abstract: Self-describing data blocks of a minimum atomic write size may be stored for a data store. Data may be received for storage in a data block of a plurality of data blocks at a persistent storage device that are equivalent to a minimum atomic write size for the persistent storage device. Metadata may be generated for the data that includes an error detection code which is generated for the data and the metadata together. The data and the metadata are sent to the persistent storage device to store together in the data block. An individual atomic write operation may write together the data and the metadata in the data block. When accessed, the error detection code is applicable to detect errors. The metadata may also be applicable to determine whether the data is stored for a currently assigned purpose or a previously assigned purpose of the data block.

Abstract: A log-structured data store may implement efficient garbage collection. Log records may be maintained in data blocks according to a log record sequence. Based, at least in part, on a log reclamation point, the log records may be evaluated to identify data blocks to reclaim that have log records in the log sequence prior to the log reclamation point. New versions of data pages updated by log records in the identified data blocks may be generated and stored in base page storage for the log structured data store. The identified data blocks may then be reclaimed for storing new data.

Abstract: Approaches for automatically backing up data from volatile memory to persistent storage in the event of a power outage, blackout or other such failure are described. The approaches can be implemented on a computing device that includes a motherboard, central processing unit (CPU) a main power source, volatile memory (e.g., random access memory (RAM)), an alternate power source and circuitry (e.g., a specialized application-specific integrated circuit (ASIC)) for performing the backup of volatile memory to a persistent storage device. In the event of a power failure of the main power source, the alternate power source is configured to supply power to the specialized ASIC for backing up the data in the volatile memory. For example, when power failure is detected, the ASIC can read the data from the DIMM socket using power supplied from the alternate power source and write that data to a persistent storage device.

Abstract: A distributed data store may provide volume recovery access to multiple recovery agents. A data volume may be maintained for a storage client at the distributed data store. Write access to the data volume may be granted according to a single writer consistency scheme. When a recovery event is detected for the data volume, the data volume may be made available to multiple recovery agents that may perform respective recovery operations. Upon first completion of a recovery operation for the data volume, granting access to the data volume according to the single writer consistency scheme may be resumed. In some embodiments, the distributed data store may be a log-structured data store.

Abstract: Approaches for automatically backing up data from volatile memory to persistent storage in the event of a power outage, blackout or other such failure are described. The approaches can be implemented on a computing device that includes a motherboard, central processing unit (CPU) a main power source, volatile memory (e.g., random access memory (RAM)), an alternate power source and circuitry (e.g., a specialized application-specific integrated circuit (ASIC)) for performing the backup of volatile memory to a persistent storage device. In the event of a power failure of the main power source, the alternate power source is configured to supply power to the specialized ASIC for backing up the data in the volatile memory. For example, when power failure is detected, the ASIC can read the data from the DIMM socket using power supplied from the alternate power source and write that data to a persistent storage device.

Abstract: A log-structured data store may implement optimized log storage for asynchronous log updates. In some embodiments, log records may be received indicating updates to data stored for a storage client and indicating positions in a log record sequence. The log records themselves may not be guaranteed to be received according to the log record sequence. Received log records may be stored in a hot log portion of a block-based storage device according to an order in which they are received. Log records in the hot log portion may then be identified to be moved to a cold log portion of the block-based storage device in order to complete a next portion of the log record sequence. Log records may be modified, such as compressed, or coalesced, before being stored together in a data block of the cold log portion according to the log record sequence.

Abstract: High-speed processing of packets to, and from, a virtualization environment can be provided while utilizing hardware-based segmentation offload and other such functionality. A hardware vendor such as a network interface card (NIC) manufacturer can enable the hardware to support open and proprietary stateless tunneling in conjunction with a protocol such as single root I/O virtualization (SR-IOV) in order to implement a virtualized overlay network. The hardware can utilize various rules, for example, that can be used by the NIC to perform certain actions, such as to encapsulate egress packets and decapsulate packets.

Abstract: Variably-sized data objects may be received for storage at a data store. The data store may have a minimum write size. In various embodiments, received data objects may be divided into one or more equally-sized portions that equal the minimum write size of the data store and a remainder of the data object. The one or more equally-sized portions of the data object may be stored in data blocks that are equivalent to the minimum write size of the data store in a fixed-size data storage area of the data store. The remainder of the data object may be stored in a variably-sized data storage area of the data store along with one or more other data portions in a same data block. The remainder of the data object may, in some embodiments, be linked to the one or more equally-sized portions of the data object.

Abstract: A network-based services provider may reserve and provision primary resource instance capacity for a given service (e.g., enough compute instances, storage instances, or other virtual resource instances to implement the service) in one or more availability zones, and may designate contingency resource instance capacity for the service in another availability zone (without provisioning or reserving the contingency instances for the exclusive use of the service). For example, the service provider may provision resource instance(s) for a database engine head node in one availability zone and designate resource instance capacity for another database engine head node in another availability zone without instantiating the other database engine head node. While the service operates as expected using the primary resource instance capacity, the contingency resource capacity may be leased to other entities on a spot market. Leases for contingency instance capacity may be revoked when needed for the given service (e.g.