Abstract: A distributed data warehouse system maintains data blocks on behalf of clients, and stores primary and secondary copies of data blocks on different disks or nodes in a cluster. The data warehouse system may back up data blocks in a key-value backup storage system. In response to a query targeting a data block previously stored in the cluster, the data warehouse system may determine whether a consistent, uncorrupted copy of the data block is available in the cluster (e.g., by applying a consistency check). If not (e.g., if a disk or node failed), the data warehouse system may automatically initiate an operation to restore the data block from the backup storage system, using a unique identifier of the data block to access a backup copy. The target data may be returned in a query response prior to restoring primary and secondary copies of the data block in the cluster.

Abstract: Code may be dynamically routed to computing resources for execution. Code may be received for execution on behalf of a client. Execution criteria for the code may be determined and computing resources that satisfy the execution criteria may be identified. The identified computing resources may then be procured for executing the code and then the code may be routed to the procured computing resources for execution. Permissions or authorization to execute the code may be shared to ensure that computing resources executing the code have the same permissions or authorization when executing the code.

Abstract: Tiered storage may be implemented for processing data. Data processors may maintain some of a data set, including user data and metadata describing the user data, locally. The data set is also maintained a data store remote to the data processor. When processing requests are received, a determination is made as to whether the local portions of the data set can execute the processing request or one or more additional portions of the data set are needed from the remote data store. If additional portions of the data set are needed, then a request may be sent to the data store for the additional portions. Once received, the data processor may execute the processing request utilizing the additional portions. Portions of the data set maintained locally at the data processor may be selected and flushed from local storage to the remote data store.

Abstract: A stateful cluster may implement scaling of the stateful cluster while maintaining access to the state of the stateful cluster. A scaling event for a stateful cluster may be detected, and in response the stateful cluster may be adjusted to include a different number of nodes. The state of the cluster may then be logically distributed among the different number of nodes according to a monotone distribution scheme. The adjusted node may then service access requests according to the monotone distribution scheme. Prior to making the adjusted storage cluster available for servicing access requests, the nodes from the original cluster may still service access requests for state.

Abstract: History for data objects may be maintained to detect data events. An indication of an Extract, Transform, Load (ETL) process applied to one or more source data objects to generate one or more transformed data objects may be received. History for the source data objects may be updated to include the transformed data objects and the ETL process that generated the transformed data objects. An evaluation of the update may be performed to determine whether an event associated with the data lineage is triggered. If the event is triggered, a notification of the event may be sent to one or more subscribers for the event.

Abstract: Specified performance attributes may be used to configure machine learning transformations for ETL jobs. Performance attributes for a machine learning pipeline that applies a model to as part of a transformation for an ETL job may be used to configure a parameter in a stage of the machine learning pipeline. The configured stage may then be used when training the model. The trained machine learning pipeline may then be applied as part of a transformation operation included in an ETL job performed by the ETL system.

Abstract: Proxy-based scaling may be performed for databases. A proxy may be implemented for a database that can establish a connection between the proxy and a database engine to perform a database queries received from a client at the proxy. A scaling event may be detected for the database responsive to which the proxy may establish a connection with a new database engine which may, in some embodiments, have different capabilities or resources that address the features or criteria that triggered the scaling event. Session state may be copied from the database engine to the new database engine so that the new database engine may be able to provide access to the database on behalf of requests received from the client through the proxy.

Abstract: A multi-column index is generated based on an interleaving of data bits for selectivity for efficient processing of data in a relational database system. Two or more columns may be identified for inclusion in the multi-column index for a relational database table. Based, at least in part, on the interleaving of data bits for selectivity from the identified columns, a multi-column index is generated for the relational database table that provides a respective index value for each entry in the relational database table. The entries of the relational database table may then be stored according to the index values of the multi-column index.

Abstract: Online restore operations for a database engine can be performed. A request to restore a database to a previous state can be received. Previously stored content of the database, such as snapshot stored prior to the previous state, can be identified along with log records describing changes to be made to the content prior to the previous state. State information in a query engine can be updated based on the previously stored content and log records so that queries can be processed based on the state information at the database restore to the previous state.

Abstract: Data transformation workflows may be generated to transform data objects. A source data schema for a data object and a target data format or target data schema for a data object may be identified. A comparison of the source data schema and the target data format or schema may be made to determine what transformations can be performed to transform the data object into the target data format or schema. Code to execute the transformation operations may then be generated. The code may be stored for subsequent modification or execution.

Abstract: Code may be dynamically routed to computing resources for execution. Code may be received for execution on behalf of a client. Execution criteria for the code may be determined and computing resources that satisfy the execution criteria may be identified. The identified computing resources may then be procured for executing the code and then the code may be routed to the procured computing resources for execution. Permissions or authorization to execute the code may be shared to ensure that computing resources executing the code have the same permissions or authorization when executing the code.

Abstract: A database system may implement efficient database journaling using non-volatile system memory. An update to a database may be received, and log records indicating the update may be generated. A non-volatile portion of the system memory may be directly accessed to store the generated log records such that in the event of a failure of the database, the log records may be still maintained. Log records stored to the non-volatile portion of the system memory may be identified as durably persisted, in some embodiments. Log records may be identified and sent to block-based storage, in some embodiments. Recovery from a database system failure, in some embodiments, by obtaining log records from the non-volatile portion of system memory.

Abstract: Specified performance attributes may be used to configure machine learning transformations for ETL jobs. Performance attributes for a machine learning pipeline that applies a model to as part of a transformation for an ETL job may be used to configure a parameter in a stage of the machine learning pipeline. The configured stage may then be used when training the model. The trained machine learning pipeline may then be applied as part of a transformation operation included in an ETL job performed by the ETL system.

Abstract: Online restore operations for a database engine can be performed. A request to restore a database to a previous state can be received. Previously stored content of the database, such as snapshot stored prior to the previous state, can be identified along with log records describing changes to be made to the content prior to the previous state. State information in a query engine can be updated based on the previously stored content and log records so that queries can be processed based on the state information at the database restore to the previous state.

Abstract: Data transformation workflows may be generated to transform data objects. A source data schema for a data object and a target data format or target data schema for a data object may be identified. A comparison of the source data schema and the target data format or schema may be made to determine what transformations can be performed to transform the data object into the target data format or schema. Code to execute the transformation operations may then be generated. The code may be stored for subsequent modification or execution.

Abstract: A multi-column index is generated based on an interleaving of data bits for selectivity for efficient processing of data in a relational database system. Two or more columns may be identified for inclusion in the multi-column index for a relational database table. Based, at least in part, on the interleaving of data bits for selectivity from the identified columns, a multi-column index is generated for the relational database table that provides a respective index value for each entry in the relational database table. The entries of the relational database table may then be stored according to the index values of the multi-column index.

Abstract: A distributed database system may implement dynamic quorum group membership changes. In various embodiments, a quorum set may maintain a replica of a data object among group members according to a protection group policy for the data object. A group member may be identified as to be replaced. In response, a new quorum set may be created from the remaining group members and a new group member. The protection group policy may be updated to include the new group members such that subsequently received updates are maintained at both the previous to quorum set and the new quorum set. Previously received updates may be replicated on the new group member. Upon completion of replicating the previously received updates, the protection group policy for the data object may be revised such that subsequently received updates are maintained at the new quorum set.

Abstract: Incremental, out-of-place updates may be implemented for index structures maintained for data stores. Portions of the index structure may be selected for updating, and an updated version of the portion of the index structure generated in another storage location different than a current storage location for the index structure such that the index structure may be searched in order to perform access requests. Updating the portion of the index structure may include compacting the portion of the index structure and/or merging the portion of the index structure with a sub-index structure generated from a portion of a log of index updates that may be maintained. The current portion of the index structure may then be replaced with the updated version of the current portion so that the updated version may be evaluated when searches of the index structure are performed.

Abstract: A storage engine may implement dynamic splitting of contentious data pages. Data pages may store data for a table of a data store as part of an indexing structure for the table. Access to the table may be provided by locating the corresponding data pages via the indexing structure. Access contention for different data pages may be monitored. Data pages may be identified for splitting based on the monitoring. A split operation for an identified data page may be formed to store the data on the identified data page on two different data pages so that subsequent access requests for the data are divided between the two data pages. Monitoring of access contention may also be performed to identify data pages for merging in order to consolidate access requests to a single data page.

Abstract: Data processing engines implement format independent data processing for distributed data in local and remote data stores. Requests execute at least a portion of an access request are received from a data processing engine that does not support a data format for data stored in a data store. The data format for the data is identified and the data accessed according to the identified data format. The portion of the access request is applied to the accessed data in order to generate results. The generated results for the portion of the access request are then sent to the data processing engine.