Abstract: A probabilistic data structure is generated for efficient query processing using a histogram for unsorted data in a column of a columnar database. A bucket range size is determined for multiples buckets of a histogram of a column in a columnar database table. In at least some embodiments, the histogram may be a height-balanced histogram. A probabilistic data structure is generated to indicate for which particular buckets in the histogram there is a data value stored in the data block. When an indication of a query directed to the column for select data is received, the probabilistic data structure for each of the data blocks storing data for the column may be examined to determine particular ones of the data blocks which do not need to be read in order to service the query for the select data.

Abstract: A database system may maintain a plurality of log records at a distributed storage system. Each of the plurality of log records may be associated with a respective change to a data page. Upon detection of a coalesce event for a particular data page, log records linked to the particular data page may be applied to generate the particular data page in its current state. Detecting the coalesce event may be a determination that the number of log records linked to the particular data page exceeds a threshold.

Abstract: A stateful cluster may implement scaling of the stateful cluster while maintaining access to the state of the stateful cluster. A scaling event for a stateful cluster may be detected, and in response the stateful cluster may be adjusted to include a different number of nodes. The state of the cluster may then be logically distributed among the different number of nodes according to a monotone distribution scheme. The adjusted node may then service access requests according to the monotone distribution scheme. Prior to making the adjusted storage cluster available for servicing access requests, the nodes from the original cluster may still service access requests for state.

Abstract: A distributed database system may implement dynamic quorum group membership changes. In various embodiments, a quorum set may maintain a replica of a data object among group members according to a protection group policy for the data object. A group member may be identified as to be replaced. In response, a new quorum set may be created from the remaining group members and a new group member. The protection group policy may be updated to include the new group members such that subsequently received updates are maintained at both the previous to quorum set and the new quorum set. Previously received updates may be replicated on the new group member. Upon completion of replicating the previously received updates, the protection group policy for the data object may be revised such that subsequently received updates are maintained at the new quorum set.

Abstract: A query optimizer may receive a query (e.g., from a source that generated the query). Input that specifies both a query hint string and a hint may be received to a hint specification interface. The hint may be applied to the query, from outside the query, to optimize a query execution plan. Applying the hint may be based, at least in part, on a query hint string. For example, which query block is associated with the query hint string may be determined. Upon such a determination, the hint may be applied to the determined query block.

Abstract: A distributed database system may implement reduced transaction I/O. In various embodiments, a plurality of log records may be stored in persistent storage for a plurality of pages of a log-structured storage system. A coalesce operation may be performed for at least some of the plurality of data pages for the log-structured storage system. The coalesce operation may, in some embodiments, coalesce two or more log records for one of the plurality of pages to generate an instance of the page and store the instance of the page to a new location in the persistent storage. Log records for one or more of the plurality of pages may be excluded from the coalesce operation. These excluded log records may not be dependent on a prior version of the page or portion of the page corresponding to the log record. Excluded log records may include one or more of redo records, undo records, or transaction table records.

Abstract: Queries that involve multiple tables in a data warehouse system may be specified through structured query language (“SQL”) statements that include union, order by, and limit clauses. When the tables are partitioned, a query of this type may be processed by forming a query plan in which intermediate results are formed by applying sort and limit operators at the partition level, combining the intermediate results from the partitions, and again applying sort and limit operations. Additional clauses, such as a unique clause, may also be pushed down to the partition level and reapplied at the level of the result set.

Abstract: A distributed database system may implement log-structured distributed storage using a single log sequence number space. A log for a data volume may be maintained in a log-structured distributed storage system. The log may be segmented across multiple protection groups according to a partitioning of user data for the data volume. Updates to the log may be assigned a log sequence number from a log sequence number space for the data volume. A protection group may be determined for an update according to which partition of user data space the update pertains. Metadata to be included with the log record may indicate a previous log sequence number of a log record maintained at the protection group. The log record may be sent to the protection group and identified as committed based on acknowledgments received from storage nodes implementing the protection group.

Abstract: A probabilistic counting structure such as a hyperloglog may be formed during a table scan for each of a selected set of columns. The columns may be selected based on an initial estimate of relatedness, which may be based on data types of the respective columns. An estimated cardinality of an intersection or union of columns may be formed based on an intersection of the probabilistic data structures. A join path may be determined based on the estimated cardinality of an intersection or union of the columns.

Abstract: A log-structured distributed storage system may implement individual write quorums. Log records may be sent to different storage nodes of a quorum set storing data for a storage client sufficient to satisfy a write quorum requirement. For each log record, acknowledgments from storage nodes are received, and a determination is made whether the write quorum requirement is satisfied for the log record. Different log records may be maintained at different storage nodes, and still satisfy the write quorum requirement such that in some embodiments no one storage node may maintain all of the log records sent to storage nodes in the quorum set.

Abstract: A distributed database system may implement dynamic quorum group membership changes. In various embodiments, a quorum set may maintain a replica of a data object among group members according to a protection group policy for the data object. A group member may be identified as to be replaced. In response, a new quorum set may be created from the remaining group members and a new group member. The protection group policy may be updated to include the new group members such that subsequently received updates are maintained at both the previous quorum set and the new quorum set. Previously received updates may be replicated on the new group member. Upon completion of replicating the previously received updates, the protection group policy for the data object may be revised such that subsequently received updates are maintained at the new quorum set.

Abstract: A stateful cluster may implement scaling of the stateful cluster while maintaining access to the state of the stateful cluster. A scaling event for a stateful cluster may be detected, and in response the stateful cluster may be adjusted to include a different number of nodes. The state of the cluster may then be logically distributed among the different number of nodes according to a monotone distribution scheme. The adjusted node may then service access requests according to the monotone distribution scheme. Prior to making the adjusted storage cluster available for servicing access requests, the nodes from the original cluster may still service access requests for state.

Abstract: Data temporarily stored in volatile memory (e.g., RAM) on a host machine can be protected using a component such as an NV-DIMM, which includes components such as an ASIC, non-volatile memory, and a battery. If power is lost to the host, the battery provides the ASIC with the power needed to determine data in the volatile memory that is protected. This protected data then can be transferred to the non-volatile memory on the NV-DIMM. When power is restored, an application or other entity can contact the NV-DIMM to recover the data, which can be transferred over a sideband channel to be restored as appropriate for a prior operation. In at least some embodiments, the NV-DIMM can receive a key over the sideband channel that can be used to encrypt and decrypt the data for further security.

Abstract: A separate distributed buffer cache system may be implemented for a storage client of a distributed storage system. Storage I/O requests may be sent from a storage client to one or more buffer cache nodes in a distributed buffer cache system that maintain portions of an in-memory buffer cache to which the requests pertain. The distributed buffer cache system may send the write requests on to the distributed storage system to be completed, and in response to receiving acknowledgements from the storage system, sending a completion acknowledgement back to the storage client. Buffer cache nodes may update buffer cache entries for received requests such that they are not available for reads until complete at the distributed storage system. For read requests where the buffer cache entries at the buffer cache node are invalid, valid data may be obtained from the distributed storage system and sent to the storage client.

Abstract: A database system may maintain a plurality of log records at a distributed storage system. Each of the plurality of log records may be associated with a respective change to a data page. A snapshot may be generated that is usable to read the data as of a state corresponding to the snapshot. Generating the snapshot may include generating metadata that is indicative of a particular log identifier of a particular one of the log records. Generating the snapshot may be performed without additional reading, copying, or writing of the data.

Abstract: A probabilistic counting structure such as a hyperloglog may be formed during a table scan for each of a selected set of columns. The columns may be selected based on an initial estimate of relatedness, which may be based on data types of the respective columns. An estimated cardinality of an intersection or union of columns may be formed based on an intersection of the probabilistic data structures. A join path may be determined based on the estimated cardinality of an intersection or union of the columns.

Abstract: Data to be stored in a data block for a columnar database table may be compressed according to a multi-level compression scheme. Data to be stored in the data block may be received. The data may be compressed according a column-specific compression technique to produce compressed data. The compressed data may then be compressed according to a second compression technique different than the column-specific compression technique to produce multi-level compressed data. The multi-level compressed data may be stored in the data block. When reading from the data block, multi-level compressed data may be decompressed according to the column-specific compression technique and the default compression technique applied to the data.

Abstract: Incremental, out-of-place updates may be implemented for index structures maintained for data stores. Portions of the index structure may be selected for updating, and an updated version of the portion of the index structure generated in another storage location different than a current storage location for the index structure such that the index structure may be searched in order to perform access requests. Updating the portion of the index structure may include compacting the portion of the index structure and/or merging the portion of the index structure with a sub-index structure generated from a portion of a log of index updates that may be maintained. The current portion of the index structure may then be replaced with the updated version of the current portion so that the updated version may be evaluated when searches of the index structure are performed.

Abstract: A balanced distribution of sort order values may be implemented for a multi-column sort order of a database table. Columns of the database table to be included in the multi-column sort order may be identified. Some columns containing string data values may be converted to equally-sized integer data values. The data values of columns may be evaluated to determine buckets representing the ranges of data values within the columns for depth-balanced histograms of the columns. Multi-column sort order values may be generated for individual entries in the database table according to bucket values assigned to the buckets that include the columns values of the individual entries. The entries of the database table may then be stored according to a sorted ordering of multi-column sort order values for the entries.

Abstract: A data store may implement dynamically assign storage locations to logical partitions according to query predicate indexes. Query predicate indexes may be maintained for individual storage locations in a data store that indicate whether storage locations store data values that do not satisfy a query predicate. The query predicate indexes may be evaluated to assign the storage locations to one or multiple logical partitions. The logical partitions may then be utilized to provide access to data stored in the data store. In some embodiments, a logical partition may be a remote storage partition that identifies data in storage locations to copy to a remote data store.