Abstract: Techniques are described for executing a query with a top-N clause to select a first N-number of rows in a data source arranged at least according to a first key and a second key of the data source using a first sort order respectively specified for the first key and a second sort order respectively specified for the second key by the query. The data source may include one or more tiles that include at least a portion of the first key and the second key. To execute the query, in an embodiment, a DBMS determines, in a first vector of first key values that are in a first tile, row identifiers identifying entries of the first vector that contain values equal to a tail value that follows a particular top number of the first key values. The DBMS may select, from a second vector of values of the second key in the first tile, second key values identified based on the determined row identifiers of the first vector.

Abstract: Techniques provide for hardware accelerated data movement between main memory and an on-chip data movement system that comprises multiple core processors that operate on the tabular data. The tabular data is moved to or from the scratch pad memories of the core processors. While the data is in-flight, the data may be manipulated by data manipulation operations. The data movement system includes multiple data movement engines, each dedicated to moving and transforming tabular data from main memory data to a subset of the core processors. Each data movement engine is coupled to an internal memory that stores data (e.g. a bit vector) that dictates how data manipulation operations are performed on tabular data moved from a main memory to the memories of a core processor, or to and from other memories. The internal memory of each data movement engine is private to the data movement engine.

Abstract: Embodiments comprise a distributed join processing technique that reduces the data exchanged over the network. Embodiments first evaluate the join using a partitioned parallel join based on join tuples that represent the rows that are to be joined to produce join result tuples that represent matches between rows for the join result. Embodiments fetch, over the network, projected columns from the appropriate partitions of the tables among the nodes of the system using the record identifiers from the join result tuples. To further conserve network bandwidth, embodiments perform an additional record-identifier shuffling phase based on the respective sizes of the projected columns from the relations involved in the join operation. Specifically, the result tuples are shuffled such that transmitting projected columns from the join relation with the larger payload is avoided and the system need only exchange, over the network, projected columns from the join relation with the smaller payload.

Abstract: Techniques are described for executing a query with a top-N clause to select a first N-number of rows in a data source arranged at least according to a first key and a second key of the data source using a first sort order respectively specified for the first key and a second sort order respectively specified for the second key by the query. The data source may include one or more tiles that include at least a portion of the first key and the second key. To execute the query, in an embodiment, a DBMS determines, in a first vector of first key values that are in a first tile, row identifiers identifying entries of the first vector that contain values equal to a tail value that follows a particular top number of the first key values. The DBMS may select, from a second vector of values of the second key in the first tile, second key values identified based on the determined row identifiers of the first vector.

Abstract: Techniques are provided herein for optimal initialization of value ranges of machine learning algorithm hyperparameters and other predictions based on dataset meta-features. In an embodiment for each particular hyperparameter of a machine learning algorithm, a computer invokes, based on an inference dataset, a distinct trained metamodel for the particular hyperparameter to detect an improved subrange of possible values for the particular hyperparameter. The machine learning algorithm is configured based on the improved subranges of possible values for the hyperparameters. The machine learning algorithm is invoked to obtain a result. In an embodiment, a gradient-based search space reduction (GSSR) finds an optimal value within the improved subrange of values for the particular hyperparameter. In an embodiment, the metamodel is trained based on performance data from exploratory sampling of configuration hyperspace, such as by GSSR.

Abstract: Techniques are provided for redundant execution by a better processor for intensive dynamic profiling after initial execution by a constrained processor. In an embodiment, a system of computer(s) receives a request to profile particular runtime aspects of an original binary executable. Based on the particular runtime aspects and without accessing source logic, the system statically rewrites the original binary executable into a rewritten binary executable that invokes telemetry instrumentation that makes observations of the particular runtime aspects and emits traces of those observations. A first processing core having low power (capacity) performs a first execution of the rewritten binary executable to make first observations and emit first traces of the first observations. Afterwards, a second processing core performs a second (redundant) execution of the original binary executable based on the first traces.

Abstract: Techniques are described herein for grouping of operations in local memory of a processing unit. The techniques involve adding a first operation for a first leaf operator of a query execution plan to a first pipelined group. The query execution plan includes a set of leaf operators and a set of non-leaf operators. Each leaf operator of the set of one or more leaf operators has a respective parent non-leaf operator and each non-leaf operator has one or more child operators from among the set of leaf operators or others of the set of non-leaf operators. The techniques further involve determining a memory requirement of executing the first operation for the first leaf operator and executing a second operation for the respective parent non-leaf operator of the first leaf operator. The output of the first operation is input to the second operation. The techniques further involve determining whether the memory requirement is satisfied by an amount of local memory.

Abstract: Techniques related to distributed relational dictionaries are disclosed. In some embodiments, one or more non-transitory storage media store a sequence of instructions which, when executed by one or more computing devices, cause performance of a method. The method involves generating, by a query optimizer at a distributed database system (DDS), a query execution plan (QEP) for generating a code dictionary and a column of encoded database data. The QEP specifies a sequence of operations for generating the code dictionary. The code dictionary is a database table. The method further involves receiving, at the DDS, a column of unencoded database data from a data source that is external to the DDS. The DDS generates the code dictionary according to the QEP. Furthermore, based on joining the column of unencoded database data with the code dictionary, the DDS generates the column of encoded database data according to the QEP.

Abstract: A system and method for processing a group and aggregate query on a relation are disclosed. A database system determines whether assistance of a heterogeneous system (HS) of compute nodes is beneficial in performing the query. Assuming that the relation has been partitioned and loaded into the HS, the database system determines, in a compile phase, whether the HS has the functional capabilities to assist, and whether the cost and benefit favor performing the operation with the assistance of the HS. If the cost and benefit favor using the assistance of the HS, then the system enters the execution phase. The database system starts, in the execution phase, an optimal number of parallel processes to produce and consume the results from the compute nodes of the HS. After any needed transaction consistency checks, the results of the query are returned by the database system.

Abstract: Techniques herein map between key spaces to generate a balanced adaptive resolution histogram for dataset partitioning. In embodiments, a computer (C) creates a mapping that associates sparse keys (SKs) with distinct dense keys. C constructs a trie by processing each item of a dataset as follows. Based on the item, C obtains an SK. C navigates from a root NT (node of the trie) to a particular NT based on a sequence of dense digits (SDD). Each dense digit of the SDD is based on the mapping. Each NT identifies a dense prefix comprising dense digits. C assigns the item to a target node based on a threshold and count of items assigned to a subtree rooted at the particular node. C determines a range of SKs for each partition of the dataset, based on: an item count for a node or subtree, dense prefixes of NTs, and the mapping.

Abstract: Techniques are described herein for introducing transcode operators into a generated operator tree during query processing. Setting up the transcode operators with correct encoding type at runtime is performed by inferring correct encoding type information during compile time. The inference of the correct encoding type information occurs in three phases during compile time: the first phase involves collecting, consolidating, and propagating the encoding-type information of input columns up the expression tree. The second phase involves pushing the encoding-type information down the tree for nodes in the expression tree that do not yet have any encoding-type assigned. The third phase involves determining which inputs to the current relational operator need to be pre-processed by a transcode operator.

Abstract: Techniques are provided for using decentralized lock synchronization to increase network throughput. In an embodiment, a first computer sends, to a second computer comprising a lock, a request to acquire the lock. In response to receiving the lock acquisition request, the second computer detects whether the lock is available. If the lock is unavailable, then the second computer replies by sending a denial to the first computer. Otherwise, the second computer sends an exclusive grant of the lock to the first computer. While the first computer has acquired the lock, the first computer sends data to the second computer. Afterwards, the first computer sends a request to release the lock to the second computer. This completes one duty cycle of the lock, and the lock is again available for acquisition.

Abstract: Embodiments comprise a distributed join processing technique that reduces the data exchanged over the network. Embodiments first evaluate the join using a partitioned parallel join based on join tuples that represent the rows that are to be joined to produce join result tuples that represent matches between rows for the join result. Embodiments fetch, over the network, projected columns from the appropriate partitions of the tables among the nodes of the system using the record identifiers from the join result tuples. To further conserve network bandwidth, embodiments perform an additional record-identifier shuffling phase based on the respective sizes of the projected columns from the relations involved in the join operation. Specifically, the result tuples are shuffled such that transmitting projected columns from the join relation with the larger payload is avoided and the system need only exchange, over the network, projected columns from the join relation with the smaller payload.

Abstract: Herein, horizontally scalable techniques efficiently configure machine learning algorithms for optimal accuracy and without informed inputs. In an embodiment, for each particular hyperparameter, and for each epoch, a computer processes the particular hyperparameter. An epoch explores one hyperparameter based on hyperparameter tuples. A respective score is calculated from each tuple. The tuple contains a distinct combination of values, each of which is contained in a value range of a distinct hyperparameter. All values of a tuple that belong to the particular hyperparameter are distinct. All values of a tuple that belong to other hyperparameters are held constant. The value range of the particular hyperparameter is narrowed based on an intersection point of a first line based on the scores and a second line based on the scores. A machine learning algorithm is optimally configured from repeatedly narrowed value ranges of hyperparameters. The configured algorithm is invoked to obtain a result.

Abstract: Techniques are described herein for performing efficient matrix multiplication in architectures with scratchpad memories or associative caches using asymmetric allocation of space for the different matrices. The system receives a left matrix and a right matrix. In an embodiment, the system allocates, in a scratchpad memory, asymmetric memory space for tiles for each of the two matrices as well as a dot product matrix. The system proceeds with then performing dot product matrix multiplication involving the tiles of the left and the right matrices, storing resulting dot product values in corresponding allocated dot product matrix tiles. The system then proceeds to write the stored dot product values from the scratchpad memory into main memory.

Abstract: Herein are techniques for automatic tuning of hyperparameters of machine learning algorithms. System throughput is maximized by horizontally scaling and asynchronously dispatching the configuration, training, and testing of an algorithm. In an embodiment, a computer stores a best cost achieved by executing a target model based on best values of the target algorithm's hyperparameters. The best values and their cost are updated by epochs that asynchronously execute. Each epoch has asynchronous costing tasks that explore a distinct hyperparameter. Each costing task has a sample of exploratory values that differs from the best values along the distinct hyperparameter. The asynchronous costing tasks of a same epoch have different values for the distinct hyperparameter, which accomplishes an exploration. In an embodiment, an excessive update of best values or best cost creates a major epoch for exploration in a subspace that is more or less unrelated to other epochs, thereby avoiding local optima.

Abstract: Techniques are provided for selection of machine learning algorithms based on performance predictions by trained algorithm-specific regressors. In an embodiment, a computer derives meta-feature values from an inference dataset by, for each meta-feature, deriving a respective meta-feature value from the inference dataset. For each trainable algorithm and each regression meta-model that is respectively associated with the algorithm, a respective score is calculated by invoking the meta-model based on at least one of: a respective subset of meta-feature values, and/or hyperparameter values of a respective subset of hyperparameters of the algorithm. The algorithm(s) are selected based on the respective scores. Based on the inference dataset, the selected algorithm(s) may be invoked to obtain a result. In an embodiment, the trained regressors are distinctly configured artificial neural networks. In an embodiment, the trained regressors are contained within algorithm-specific ensembles.

Abstract: A system and method for processing a group and aggregate query on a relation are disclosed. A database system determines whether assistance of a heterogeneous system (HS) of compute nodes is beneficial in performing the query. Assuming that the relation has been partitioned and loaded into the HS, the database system determines, in a compile phase, whether the HS has the functional capabilities to assist, and whether the cost and benefit favor performing the operation with the assistance of the HS. If the cost and benefit favor using the assistance of the HS, then the system enters the execution phase. The database system starts, in the execution phase, an optimal number of parallel processes to produce and consume the results from the compute nodes of the HS. After any needed transaction consistency checks, the results of the query are returned by the database system.

Abstract: A method for storing XML documents a hybrid navigation/streaming format is provided to allow efficient storage and processing of queries on the XML data that provides the benefits of both navigation and streaming and ameliorates the disadvantages of each. Each XML document to be stored is independently analyzed to determine a combination of navigable and streamable storage format that optimizes the processing of the data for anticipated access patterns.

Abstract: Techniques provide for hardware accelerated data movement between main memory and an on-chip data movement system that comprises multiple core processors that operate on the tabular data. The tabular data is moved to or from the scratch pad memories of the core processors. While the data is in-flight, the data may be manipulated by data manipulation operations. The data movement system includes multiple data movement engines, each dedicated to moving and transforming tabular data from main memory data to a subset of the core processors. Each data movement engine is coupled to an internal memory that stores data (e.g. a bit vector) that dictates how data manipulation operations are performed on tabular data moved from a main memory to the memories of a core processor, or to and from other memories. The internal memory of each data movement engine is private to the data movement engine.