Abstract: Embodiments of the invention relate to processing data records, and for a multi-phase partitioned data reduction. The first phase relates to processing data records and partitioning the records into a first partition of records having a common characteristic and a second partition of records that are not members of the first partition. The data records in each partition are subject to intra-partition data reduction responsive to a resource constraint. The data records in each partition are also subject to an inter-partition data reduction, also referred to as an aggregation to reduce a footprint for storing the records. Partitions and/or individual records are logically aggregated and a data reduction operation for the logical aggregation of records takes place in response to available resources.

Abstract: Join operations may be selectively assigned from local data processing to remote data processing. Aggregation queries directed to distributed data sets may have query plans generated that include join operations. Some join operations may be executed locally while others may be assigned for remote execution at a remote data processing engine. Join operations may be identified and evaluated for remote execution according to a processing constraint for the remote data processing engine. For those join operations that satisfy the remote data processing engine, the query plan may be modified to assign the join operation for remote execution. A result for the aggregation query may be generated according to the modified query plan and the result sent in response to the aggregation query.

Abstract: Techniques for rapid secure authentication and communications through multitenant components in a provider network are described. A main database cluster can request a burst cluster from a burst service, which can provide the main cluster with a shared secret associated with only a selected burst cluster. The main cluster can use the shared secret to encrypt a value that can be passed, via a proxy, to the selected burst cluster in a connection request. The selected burst cluster can validate that the connection request was truly originated by the main cluster by validating the encrypted value using the shared secret.

Abstract: Data processing engines implement format independent data processing for distributed data in local and remote data stores. Requests execute at least a portion of an access request are received from a data processing engine that does not support a data format for data stored in a data store. The data format for the data is identified and the data accessed according to the identified data format. The portion of the access request is applied to the accessed data in order to generate results. The generated results for the portion of the access request are then sent to the data processing engine.

Abstract: A method creating a compact hash table includes a thread executing using a processor for building a linear probe hash table based on allocating space for an array of keys. Payloads are inserted into the linear probe hash table. Keys from the array of keys are inserted into the linear probe hash table with linear probing. A linear probing bitmap is generated based on occupied locations in the linear probe hash table. A compact hash table is created based on the linear probe hash table and the linear probing bitmap.

Abstract: A computer system comprises memory to store computer-executable instructions. The computer system may, as a result of execution of the instructions by one or more processors, cause the system to load a first subset of a set of data elements into a first cache, load a second subset of the set of data elements into a second cache, and as a result of elements of the first subset being processed, issue commands to place elements of the second subset into the first cache to enable processing the second subset to be processed from the first cache.

Abstract: Queries may be dynamically assigned to secondary query processing resources. A primary query engine may receive a query and determine whether the query is eligible for performance at a secondary query engine. If eligible, the primary query engine may evaluate the availability of the first query engine to perform the query. The first query engine may determine whether to assign the query to the primary query engine or to the secondary query according to availability evaluation. For queries assigned to the secondary query engine, the primary query engine may send a request to the secondary query engine to being processing of the query.

Abstract: Relocating data sharing operations for query processing may be implemented when generating plans to perform a query. A query operation that causes a node to share data obtained from a different set of nodes with other nodes in a same set of nodes may be identified. The identified query operation may be replaced with another operation that causes the different set of nodes to share the data directly with the other nodes in the same set of nodes.

Abstract: Queries received at a query engine may be executed for structured data and not-structured data. A query execution plan may be generated for the query that includes stateless operations to apply the query to the not-structured data at remote query processing engines. The remote query processing engines may perform the stateless operations and return results to the query engine. The query engine may generate a result for the query based on the results received from the remote query engine as well as results determined as part of applying the query to structured data. The result to the query may be returned.

Abstract: A method of orchestrated shuffling of data in a non-uniform memory access device including a plurality of processing nodes that connected by interconnects. The method includes running an application on a plurality of threads executing on the plurality of processing nodes. Data to be shuffled is identified from source threads running on source processing nodes among the processing nodes to target threads executing on target processing nodes among the processing nodes. The method further includes generating a plan for orchestrating the shuffling of the data among the all of the memory devices associated with the threads and for simultaneously transmitting data over different interconnects to a plurality of different target processing nodes from a plurality of different source processing nodes. The data is shuffled among all of the memory devices based on the plan and each processing node is capable of accessing data from first and second local memory devices.

Abstract: A method includes logically organizing, by an object hierarchy processor, data objects in a first hierarchy. A portion of the data objects in the first hierarchy logically includes groupings of other data objects. The object hierarchy processor physically organizes the data objects across two or more types of memory in a second hierarchy. Another portion of the data objects in the second hierarchy physically includes groupings of other data objects. Groupings of the data objects in the second hierarchy are dynamically moved across the two or more types of memory. Levels of access of the data objects are tracked using a data structure that maps groupings of the data objects in the first hierarchy onto metadata information including combined access frequencies of the data objects, and current number of accessors to the data objects, in each grouping of the data objects.

Abstract: Embodiments include a method, system, and computer program product for encoding data while it is being processed as part of a query is provided. The method includes receiving a query request and determining a set of values associated with data to be encoded for completing the query request. The method also includes encoding those values such that any subsequent processing operations can be performed on the encoded values to complete the requested query. After performing the subsequent processing operations to complete the requested query, each value is decoded back to its original value.

Abstract: Burst performance of a database query may be determined according to a size of the database query. A query to a database may be received. A size may be determined for the query. If the size is less than a size threshold assigned to a first query engine, then the query may be performed at the first query engine. If the size is greater than or equal to the size threshold assigned to the first query engine, then the query may be performed at a second query engine.

Abstract: A method of orchestrated shuffling of data in a non-uniform memory access device that includes a plurality of processing nodes that are connected by interconnects. The method includes running an application on a plurality of threads executing on the plurality of processing nodes. Data to be shuffled is identified from source threads running on source processing nodes among the processing nodes to target threads executing on target processing nodes among the processing nodes. The method further includes generating a plan for orchestrating the shuffling of the data among the all of the memory devices associated with the threads and for simultaneously transmitting data over different interconnects to a plurality of different target processing nodes from a plurality of different source processing nodes. The data is shuffled among all of the memory devices based on the plan. The data includes operand data and operational state data of the source threads.

Abstract: Data processing engines implement tiered data processing for distributed data in local and remote data stores. Requests to access distributed data including a data object in a remote data store are received at a data processing engine. A query plan is generated to service the access request. Different operations in the query plan are identified and assigned to one or more remote query processing engines that may access the remote data object. Requests to perform the different operations are sent to the one or more remote query processing engines. A final result is generated for the request based on the results received for the different operations from the remote query processing engine and results from operations performed with respect to locally stored data.

Abstract: A method of orchestrated shuffling of data in a non-uniform memory access device that includes a plurality of processing nodes connected by interconnects. The method includes running an application on a plurality of threads executing on the plurality of processing nodes. Data to be shuffled is identified from source threads running on source processing nodes among the processing nodes to target threads executing on target processing nodes among the processing nodes. The method further includes generating a plan for orchestrating the shuffling of the data among the all of the memory devices associated with the threads and for simultaneously transmitting data over different interconnects to a plurality of different target processing nodes from a plurality of different source processing nodes. The data is shuffled among all of the memory devices based on the plan. Shifting the data-shifting table includes rotating a first ring with respect to a second ring.

Abstract: A computer system comprises memory to store computer-executable instructions. The computer system may, as a result of execution of the instructions by one or more processors, cause the system to load a first subset of a set of data elements into a first cache, load a second subset of the set of data elements into a second cache, and as a result of elements of the first subset being processed, issue commands to place elements of the second subset into the first cache to enable processing the second subset to be processed from the first cache.

Abstract: A method creating a compact hash table includes a thread executing using a processor for building a linear probe hash table based on allocating space for an array of keys. Payloads are inserted into the linear probe hash table. Keys from the array of keys are inserted into the linear probe hash table with linear probing. A linear probing bitmap is generated based on occupied locations in the linear probe hash table. A compact hash table is created based on the linear probe hash table and the linear probing bitmap.

Abstract: A method for creating a compact hash table includes a thread executing using a processor for scanning a subset of keys and values, and for each key, performing a hash operation and inserting hashed keys into a bitmap structure. Cumulative population counts of keys and values are determined within the bitmap. The method repeats scanning the subset of the keys and values. The keys and values are inserted into a compacted array using the cumulative population counts.

Abstract: A method for building a hash table over a subset of data in a data set includes partitioning a subset of keys and values in the data set into multiple partitions. A hash table is formed with space reserved for each partition based on cumulative counts for a number of keys and values in each partition of the multiple partitions. Each thread selects one or more partitions and inserts keys and values belonging to the selected one or more partitions into the hash table in the reserved space for those partitions. A compact hash table is created that includes a bitmap and a compacted army.