Abstract: A method of orchestrated shuffling of data in a non-uniform memory access device that includes a plurality of processing nodes that are connected by interconnects. The method includes running an application on a plurality of threads executing on the plurality of processing nodes. Data to be shuffled is identified from source threads running on source processing nodes among the processing nodes to target threads executing on target processing nodes among the processing nodes. The method further includes generating a plan for orchestrating the shuffling of the data among the all of the memory devices associated with the threads and for simultaneously transmitting data over different interconnects to a plurality of different target processing nodes from a plurality of different source processing nodes. The data is shuffled among all of the memory devices based on the plan. The data includes operand data and operational state data of the source threads.

Abstract: A method of orchestrated shuffling of data in a non-uniform memory access device including a plurality of processing nodes that connected by interconnects. The method includes running an application on a plurality of threads executing on the plurality of processing nodes. Data to be shuffled is identified from source threads running on source processing nodes among the processing nodes to target threads executing on target processing nodes among the processing nodes. The method further includes generating a plan for orchestrating the shuffling of the data among the all of the memory devices associated with the threads and for simultaneously transmitting data over different interconnects to a plurality of different target processing nodes from a plurality of different source processing nodes. The data is shuffled among all of the memory devices based on the plan and each processing node is capable of accessing data from first and second local memory devices.

Abstract: A method of orchestrated shuffling of data in a non-uniform memory access device that includes a plurality of processing nodes connected by interconnects. The method includes running an application on a plurality of threads executing on the plurality of processing nodes. Data to be shuffled is identified from source threads running on source processing nodes among the processing nodes to target threads executing on target processing nodes among the processing nodes. The method further includes generating a plan for orchestrating the shuffling of the data among the all of the memory devices associated with the threads and for simultaneously transmitting data over different interconnects to a plurality of different target processing nodes from a plurality of different source processing nodes. The data is shuffled among all of the memory devices based on the plan. Shifting the data-shifting table includes rotating a first ring with respect to a second ring.

Abstract: Embodiments relate to hierarchical database compression. An aspect includes applying a first level of a first type of compression to a first partition of a column of a database. Another aspect includes applying a second level of the first type of compression to a subset of the first partition, wherein the first level of the first type of compression comprises a first first-level dictionary and the second level of the first type of compression comprises a first second-level dictionary, and wherein a code size of the first first-level dictionary is larger than a code size of the first second-level dictionary.

Abstract: A method for building a hash table over a subset of data in a data set includes mapping keys in the data set to values in the data set using multiple parallel computation threads. Each thread scans a subset of the keys and values and partitioning the subset of the keys and values into multiple partitions. A cumulative count for keys and values in each partition is determined. A hash table with space reserved for each partition is formed based on the determined cumulative counts. Each thread selects one or more partitions and inserts keys and values belonging to the selected one or more partitions into the hash table in the reserved space for those partitions.

Abstract: A first request may be received to update a first set of values. The first set of values may be stored at a first location within a first data page of a database. The first location may be read-only. In response to the receiving of the first request, a first set of records may be inserted into a second data page. The first set of records may include the update of the first set of values. In response to the inserting, a forward pointer may be stored in the first data page that points to the first set of records on the second data page. One or more committed values may be identified on the second data page. In response to the identifying, the one or more committed values may be merged from the second data page to a third data page.

Abstract: Embodiments of the invention relate to processing queries. A query operation to be performed on a table of data is translated into a series of bit level logical operations using expansion and/or saturation operations. A mask is created from the series of bit level logical operations. This mask is then simultaneously applied to multiple rows from the table of data.

Abstract: A first request may be received to update a first set of values. The first set of values may be stored at a first location within a first data page of a database. The first location may be read-only. In response to the receiving of the first request, a first set of records may be inserted into a second data page. The first set of records may include the update of the first set of values. In response to the inserting, a forward pointer may be stored in the first data page that points to the first set of records on the second data page. One or more committed values may be identified on the second data page. In response to the identifying, the one or more committed values may be merged from the second data page to a third data page.

Abstract: Embodiments of the present invention provide a database processing system for efficient partitioning of a database table with column-major layout for executing one or more join operations. One embodiment comprises a method for partitioning a database table with column-major layout, partitioning only the join-columns by limiting the partitions by size and number, executing one or more join operations for joining the partitioned columns, and optionally de-partitioning the join result to the original order by sequentially writing and randomly reading table values using P cursors.

Abstract: Embodiments relate to an immediate join spilling scheme for a database. An aspect includes receiving a command to perform a join of an inner table and an outer table in a database. Another aspect includes spilling a plurality of hash tables to a disk memory, each hash table corresponding to a respective inner table partition. Another aspect includes registering, by a first agent of a plurality of agents, a set of inner table partitions that require processing by the first agent to complete the join for a first stride of rows of the outer table. Another aspect includes, based on determining that a hash table corresponding to a registered inner table partition of the first agent has been loaded into the local memory by the second agent of the plurality of agents, performing the join of the loaded hash table and the first stride of rows of the outer table by the first agent.

Abstract: A method, a system, and a computer program product are provided. A filter is created for each portion of a data set. The filter indicates which one or more characteristics are present among each of the portions. Each of the one or more characteristics comprises one or more groups defined by the data grouping operation. The filters for the portions of the data sets are transferred to one or more filter processors and combined within the one or more filter processors to indicate characteristics of data residing across multiple processing elements to produce a result for a data grouping operation, utilizing transfers based on a combined filter result. In various embodiments, the filter may be a Bloom filter.

Abstract: A computer program product provides efficient multiple aggregation distinct processing. The computer program product including a computer readable storage medium having program instructions embodied therewith. The program instructions are executable by a processor to cause the processor to convert a query with multiple aggregation distincts into a two-stage group-by process using a nest operator. The two-stage group-by process including further program instructions to cause the processor to: perform a first stage group-by process including the nest operator taking a single input row, and concatenating a grouping key with a measure value for each aggregation distinct that forms multiple inputs to a single group-by table, and perform a second stage group-by process including the nest operator bringing together entries for each original group.

Abstract: A computer-implemented method includes receiving a query plan. The method includes identifying a plurality of qualified relevant rows and one or more encoding candidate payload columns. The method includes analyzing the relevant rows in the encoding candidate payload columns to yield a count of distinct contents and a payload column width. The method includes estimating a cost and determining whether the cost is larger than an amount of available memory for on-the-fly encoding all of the plurality of encoding candidate payload columns. The method is responsive to the estimated cost being less than the amount of available memory, by on-the-fly encoding the encoding candidates and responsive to the estimated cost being greater than the amount of available memory by on-the-fly encoding fewer than all of the encoding candidates so as not to exceed the available memory, and leaving alone one or more remaining encoding candidate payload columns unencoded.

Abstract: System, method and computer program products for probing a hash table by receiving a compressed input key, computing a hash value for the compressed input key and probing one or more buckets in a hash table for a match. Each bucket includes multiple chunks. For a bucket in the hash table, chunks are searched in that bucket by comparing in parallel the hash value with multiple slots in each chunk, such that if a value in a chunk equals the hash value of the compressed input key, then a match is declared and a vector is returned with a significant bit of a matching slot in the bucket set to a value. If a value stored in a chunk corresponds to an empty slot, then a mismatch is declared, and the vector is returned as the result with the significant bit of a matching empty slot set to the value.

Abstract: A system for sorting in a column organized relational database by sorting only key columns in the column organized relational database to reduce memory and CPU usage. The sort may be performed using an in memory sort. After sorting the key columns, the system orders a payload column according to an order indicated by the key columns. The payload column is associated with the key columns in the column organized relational database. The system assesses memory capacity prior to ordering the payload column. Prior to ordering a plurality of payload columns, the system refrains from automatically concatenating the plurality of payload columns, and refrains from automatically de-concatenating any concatenated payload columns after the ordering the plurality of payload columns. The system determines an amount of required memory necessary to order the payload column, and detects an amount of available memory accessible to order the payload column.

Abstract: A system for sorting in a column organized relational database by sorting only key columns in the column organized relational database to reduce memory and CPU usage. The sort may be performed using an in memory sort. After sorting the key columns, the system orders a payload column according to an order indicated by the key columns. The payload column is associated with the key columns in the column organized relational database. The system assesses memory capacity prior to ordering the payload column. Prior to ordering a plurality of payload columns, the system refrains from automatically concatenating the plurality of payload columns, and refrains from automatically de-concatenating any concatenated payload columns after the ordering the plurality of payload columns. The system determines an amount of required memory necessary to order the payload column, and detects an amount of available memory accessible to order the payload column.

Abstract: A method includes scanning multiple incoming database transaction requests. Each transaction includes one or more operations. Operations are clustered into a set of combined operations based on type of operation constraints. Log records are prepared and written for re-performing operations upon system failures, and for undoing operations upon an operation or a transaction failing to be processed fully. Each set of combined operations are performed within a thread. Each update operation is marked for a transaction within which the update operation belongs. Recoverable update operations belonging to a plurality of transactions are performed within a single logical thread of execution.

Abstract: A method implemented by at least one processing device, a processing device, and a computer program product are provided for adjusting hash partitions for a hash join operation. An amount of rows within each of an initial number of hash partitions is determined during assignment of respective rows to respective hash partitions. The initial number of hash partitions is adjusted to a final number of hash partitions based on the amount of rows within each of the initial number of hash partitions and an amount of available memory. The adjusting further includes determining the final number of hash partitions based on the amount of rows within each of the initial number of hash partitions and the amount of available memory, and assigning the rows to the final number of hash partitions. The hash join operation is then performed.

Abstract: A method implemented by at least one processing device, a processing device, and a computer program product are provided for adjusting hash partitions for a hash join operation. An amount of rows within each of an initial number of hash partitions is determined during assignment of respective rows to respective hash partitions. The initial number of hash partitions is adjusted to a final number of hash partitions based on the amount of rows within each of the initial number of hash partitions and an amount of available memory. The adjusting further includes determining the final number of hash partitions based on the amount of rows within each of the initial number of hash partitions and the amount of available memory, and assigning the rows to the final number of hash partitions. The hash join operation is then performed.

Abstract: A computer-implemented method includes receiving a query plan. The method includes identifying a plurality of qualified relevant rows and one or more encoding candidate payload columns. The method includes analyzing the relevant rows in the encoding candidate payload columns to yield a count of distinct contents and a payload column width. The method includes estimating a cost and determining whether the cost is larger than an amount of available memory for on-the-fly encoding all of the plurality of encoding candidate payload columns. The method is responsive to the estimated cost being less than the amount of available memory, by on-the-fly encoding the encoding candidates and responsive to the estimated cost being greater than the amount of available memory by on-the-fly encoding fewer than all of the encoding candidates so as not to exceed the available memory, and leaving alone one or more remaining encoding candidate payload columns unencoded.