Abstract: Embodiments of the invention relate to processing data records, and for a multi-phase partitioned data reduction. The first phase relates to processing data records and partitioning the records into a first partition of records having a common characteristic and a second partition of records that are not members of the first partition. The data records in each partition are subject to intra-partition data reduction responsive to a resource constraint. The data records in each partition are also subject to an inter-partition data reduction, also referred to as an aggregation to reduce a footprint for storing the records. Partitions and/or individual records are logically aggregated and a data reduction operation for the logical aggregation of records takes place in response to available resources.

Abstract: A database system, computer program product, and a method for evaluating aggregates in database systems includes hashing of aggregation keys on a per bucket basis, and depending on a number of hashed tuples per bucket, sorting said tuples. Additionally, depending on the number of hashed tuples per bucket, the bucket is kept without change. Moreover, depending on the number of hashed tuples per bucket, maintaining a secondary hash table for a particular bucket, map tuples to it, aggregate as you map.

Abstract: Provided herein are techniques for processing a database query aggregating data. Data tuples of a database object each including a grouping element and a data element are analyzed to determine a length of the data element for each data tuple. A plurality of tables each accommodate a successively greater length for the data element. A corresponding table of the plurality of tables to store each data tuple is determined based on the length of the data element of that data tuple relative to the accommodated lengths of the plurality of tables. Each data tuple in the determined corresponding table is stored to group the data tuples within each of the plurality of tables based on the grouping element, and an indication of corresponding tables containing members for each group is provided. The groups are combined across the plurality of tables to aggregate the data tuples for a database query.

Abstract: A computer program product provides efficient multiple aggregation distinct processing. The computer program product including a computer readable storage medium having program instructions embodied therewith. The program instructions are executable by a processor to cause the processor to convert a query with multiple aggregation distincts into a two-stage group-by process using a nest operator. The two-stage group-by process including further program instructions to cause the processor to: perform a first stage group-by process including the nest operator taking a single input row, and concatenating a grouping key with a measure value for each aggregation distinct that forms multiple inputs to a single group-by table, and perform a second stage group-by process including the nest operator bringing together entries for each original group.

Abstract: Embodiments include a method, system, and computer program product for encoding data while it is being processed as part of a query is provided. The method includes receiving a query request and determining a set of values associated with data to be encoded for completing the query request. The method also includes encoding those values such that any subsequent processing operations can be performed on the encoded values to complete the requested query. After performing the subsequent processing operations to complete the requested query, each value is decoded back to its original value.

Abstract: A method, a system, and a computer program product are provided. A filter is created for each portion of a data set. The filter indicates which one or more characteristics are present among each of the portions. Each of the one or more characteristics comprises one or more groups defined by the data grouping operation. The filters for the portions of the data sets are transferred to one or more filter processors and combined within the one or more filter processors to indicate characteristics of data residing across multiple processing elements to produce a result for a data grouping operation, utilizing transfers based on a combined filter result. In various embodiments, the filter may be a Bloom filter.

Abstract: A method, a system, and a computer program product are provided. A filter is created for each portion of a data set. The filter indicates which one or more characteristics are present among each of the portions. Each of the one or more characteristics comprises one or more groups defined by the data grouping operation. The filters for the portions of the data sets are transferred to one or more filter processors and combined within the one or more filter processors to indicate characteristics of data residing across multiple processing elements to produce a result for a data grouping operation, utilizing transfers based on a combined filter result. In various embodiments, the filter may be a Bloom filter.

Abstract: A hardware accelerator includes a bitmap processor that processes a bitmap structure for multiple list population. A population count processor processes population counts for data and aggregates the population counts. The bitmap data structure includes bitmap bit fields interleaved with aggregated population count fields.

Abstract: Provided herein are techniques for processing a database query aggregating data. Data tuples of a database object each including a grouping element and a data element are analyzed to determine a length of the data element for each data tuple. A plurality of tables each accommodate a successively greater length for the data element. A corresponding table of the plurality of tables to store each data tuple is determined based on the length of the data element of that data tuple relative to the accommodated lengths of the plurality of tables. Each data tuple in the determined corresponding table is stored to group the data tuples within each of the plurality of tables based on the grouping element, and an indication of corresponding tables containing members for each group is provided. The groups are combined across the plurality of tables to aggregate the data tuples for a database query.

Abstract: A method, a system, and a computer program product are provided. A filter is created for each portion of a data set. The filter indicates which one or more characteristics are present among each of the portions. Each of the one or more characteristics comprises one or more groups defined by the data grouping operation. The filters for the portions of the data sets are transferred to one or more filter processors and combined within the one or more filter processors to indicate characteristics of data residing across multiple processing elements to produce a result for a data grouping operation, utilizing transfers based on a combined filter result. In various embodiments, the filter may be a Bloom filter.

Abstract: Systems and methods for sorting a data set stored on an external device. A plurality of smaller radix sizes are determined, based on a first radix size and performance characteristics of an external data storage device, whose sizes add up to a first radix size for an in-place radix sort. The smaller radix sizes reduce a total time to perform the in-place radix sort. Each level of a multiple level in-place radix sort is performed with the smaller radix sizes. Each level of the sort includes dividing the data set into N buckets; dividing the buffer into N buckets; and iteratively loading a respective segment in each bucket of the data set into a respective bucket of the buffer, performing an in-place radix sort on the data in the buffer, and returning sorted buffer data to the data set on the external storage device.

Abstract: Methods for sorting a data set. Data items each having a first portion and a second portion is stored. The first and second portions are stored separately and each has a separate set of keys. The first portion has a pointer indicating the second portion. At least some of the first set of keys for each data item is stored in a local memory of a first processor. At least one data stripe set is defined with one stripe within each bucket. An in-place partial bucket radix sort is performed on data items within one data stripe set with a first processor using an initial key. Incorrectly sorted data items are grouped into respective incorrect data item groups within each bucket. A radix sort is then performed using the initial radix on the incorrect data item groups. A first level sorted output is produced.

Abstract: A method, a system, and a computer program product are provided. A filter is created for each portion of a data set. The filter indicates which one or more characteristics are present among each of the portions. Each of the one or more characteristics comprises one or more groups defined by the data grouping operation. The filters for the portions of the data sets are transferred to one or more filter processors and combined within the one or more filter processors to indicate characteristics of data residing across multiple processing elements to produce a result for a data grouping operation, utilizing transfers based on a combined filter result. In various embodiments, the filter may be a Bloom filter.

Abstract: Systems and methods for sorting a data set. Data items each having a first portion and a second portion is stored. The first and second portions are stored separately and each has a separate set of keys. The first portion has a pointer indicating the second portion. At least some of the first set of keys for each data item is stored in a local memory of a first processor. At least one data stripe set is defined with one stripe within each bucket. An in-place partial bucket radix sort is performed on data items within one data stripe set with a first processor using an initial key. Incorrectly sorted data items are grouped into respective incorrect data item groups within each bucket. A radix sort is then performed using the initial radix on the incorrect data item groups. A first level sorted output is produced.

Abstract: A computer program product provides efficient multiple aggregation distinct processing. The computer program product including a computer readable storage medium having program instructions embodied therewith. The program instructions are executable by a processor to cause the processor to convert a query with multiple aggregation distincts into a two-stage group-by process using a nest operator. The two-stage group-by process including further program instructions to cause the processor to: perform a first stage group-by process including the nest operator taking a single input row, and concatenating a grouping key with a measure value for each aggregation distinct that forms multiple inputs to a single group-by table, and perform a second stage group-by process including the nest operator bringing together entries for each original group.

Abstract: A database system, computer program product, and a method for evaluating aggregates in database systems includes hashing of aggregation keys on a per bucket basis, and depending on a number of hashed tuples per bucket, sorting said tuples. Additionally, depending on the number of hashed tuples per bucket, the bucket is kept without change. Moreover, depending on the number of hashed tuples per bucket, maintaining a secondary hash table for a particular bucket, map tuples to it, aggregate as you map.

Abstract: Systems and methods for sorting a data set stored on an external device. A plurality of smaller radix sizes are determined, based on a first radix size and performance characteristics of an external data storage device, whose sizes add up to a first radix size for an in-place radix sort. The smaller radix sizes reduce a total time to perform the in-place radix sort. Each level of a multiple level in-place radix sort is performed with the smaller radix sizes. Each level of the sort includes dividing the data set into N buckets; dividing the buffer into N buckets; and iteratively loading a respective segment in each bucket of the data set into a respective bucket of the buffer, performing an in-place radix sort on the data in the buffer, and returning sorted buffer data to the data set on the external storage device.

Abstract: Embodiments include a method and computer program product for encoding data while it is being processed as part of a query is provided. The method includes receiving a query request and determining a set of values associated with data to be encoded for completing the query request. The method also includes encoding those values such that any subsequent processing operations can be performed on the encoded values to complete the requested query. After performing the subsequent processing operations to complete the requested query, each value is decoded back to its original value.

Abstract: Embodiments include a method, system, and computer program product for encoding data while it is being processed as part of a query is provided. The method includes receiving a query request and determining a set of values associated with data to be encoded for completing the query request. The method also includes encoding those values such that any subsequent processing operations can be performed on the encoded values to complete the requested query. After performing the subsequent processing operations to complete the requested query, each value is decoded back to its original value.

Abstract: Embodiments include a system for encoding data while it is being processed. The system includes a processor, an encoder and a decoder. The processor is configured to process a query request by determining a set of values. The encoder is configured for encoding the set of values, such that a subsequent processing operation can be performed on the encoded values. The processor performs the subsequent processing operations. The decoder is configured for decoding each value back to its value prior to being encoded upon completion of the processor completing the requested query.