Abstract: Unsorted sparse dictionary encodings are transformed into unsorted-dense or sorted-dense dictionary encodings. Sparse domain codes have large gaps between codes that are adjacent in order. Unlike spare codes, dense codes have smaller gaps between adjacent codes; consecutive codes are dense codes that have no gaps between adjacent codes. The techniques described herein are relational approaches that may be used to generate sparse composite codes and sorted codes.

Abstract: Techniques described herein propose a ROWID Elimination Rewrite that uses functional dependencies to substitute and/or eliminate ROWID pseudo-columns referenced in a query in order to reduce memory pressure and speed up processing.

Abstract: The present invention relates to execution optimization of database queries. Herein are techniques for optimal execution based on query interpretation by translation to a domain specific language (DSL), with optimizations such as partial evaluation, abstract syntax tree (AST) rewriting, just in time (JIT) compilation, dynamic profiling, speculative logic, and Futamura projection. In an embodiment, a database management system (DBMS) that is hosted on a computer generates a query tree that represents a database query that contains an expression that is represented by a subtree of the query tree. The DBMS generates a sequence of DSL instructions that represents the subtree. The sequence of DSL instructions is executed to evaluate the expression during execution of the database query. In an embodiment, an AST is generated from the sequence of DSL instructions. In an embodiment, the DSL AST is optimally rewritten based on a runtime feedback loop that includes dynamic profiling information.

Abstract: A partial group by operator is a group by operator that implements a fallback mechanism. The fallback mechanism is triggered whenever memory pressure reaches a certain threshold. When the fallback mechanism is triggered, a row is included in an output of the partial group by operator without including an aggregation value for a grouping value for the row to an aggregation data structure. A final group by operator computes a final aggregate value of all results, including pre-grouped results and passed through results, from the partial group by operator.

Abstract: Techniques are described for a vectorized queue, which implements a vectorized ‘contains’ function that determines whether a value is in the queue. A three-phase vectorized shortest-path graph search splits each expanding and probing iteration into three phases that utilize vectorized instructions: (1) The neighbors of nodes that are in a next queue are fetched and written into a current queue. (2) It is determined whether the destination node is among the fetched neighbor nodes in the current queue. (3) The fetched neighbor nodes that have not yet been visited are put into the next queue. According to an embodiment, a vectorized copy operation performs vector-based data copying using vectorized load and store instructions. Specifically, vectors of data are copied from a source to a destination. Any invalid data copied to the destination is overwritten, either with a vector of additional valid data or with a vector of nonce data.

Abstract: Techniques are provided for lazy push optimization, allowing for constant time push operations. A d-heap is used as the underlying data structure for indexing values being inserted. The d-heap is vectorized by storing values in a contiguous memory array. Heapify operations are delayed until a retrieve operation occurs, improving insert performance of vectorized d-heaps that use horizontal aggregation SIMD instructions at the cost of slightly lower retrieve performance.

Abstract: Herein are techniques that extend a software system to embed new guest programing languages (GPLs) that interoperate in a transparent, modular, and configurable way. In embodiments, a computer inserts an implementation of a GPL into a deployment of the system. A command registers the GPL, define subroutines for the GPL, generates a guest virtual environment, and adds a binding of a dependency to a guest module. In an embodiment, a native programing language invokes a guest programing language to cause importing intra- or inter-language dependencies. An embodiment defines a guest object that is implemented in a first GPL and accessed from a second GPL. In an embodiment, dependencies are retrieved from a virtual file system having several alternative implementation mechanisms that include: an archive file or an actual file system, and a memory buffer or a column of a database table.

Abstract: The present invention relates to execution optimization of database queries. Herein are techniques for optimal execution based on query interpretation by translation to a domain specific language (DSL), with optimizations such as partial evaluation, abstract syntax tree (AST) rewriting, just in time (JIT) compilation, dynamic profiling, speculative logic, and Futamura projection. In an embodiment, a database management system (DBMS) that is hosted on a computer generates a query tree that represents a database query that contains an expression that is represented by a subtree of the query tree. The DBMS generates a sequence of DSL instructions that represents the subtree. The sequence of DSL instructions is executed to evaluate the expression during execution of the database query. In an embodiment, an AST is generated from the sequence of DSL instructions. In an embodiment, the DSL AST is optimally rewritten based on a runtime feedback loop that includes dynamic profiling information.

Abstract: Techniques are provided for vectorizing Heapsort. A K-heap is used as the underlying data structure for indexing values being sorted. The K-heap is vectorized by storing values in a contiguous memory array containing a beginning-most side and end-most side. The vectorized Heapsort utilizes horizontal aggregation SIMD instructions for comparisons, shuffling, and moving data. Thus, the number of comparisons required in order to find the maximum or minimum key value within a single node of the K-heap is reduced resulting in faster retrieval operations.

Abstract: The manner in which tables are joined can affect the outcome of the query and database performance. Example types of join operations include semi-join and inner-join. The techniques described herein are approaches that may be used to substitute a semi-join operator with an inner-join operator and may be used to transform and optimize representations of queries.

Abstract: The present invention relates to execution optimization of database queries. Herein are techniques for optimal execution based on query interpretation by translation to a domain specific language (DSL), with optimizations such as partial evaluation, abstract syntax tree (AST) rewriting, just in time (JIT) compilation, dynamic profiling, speculative logic, and Futamura projection. In an embodiment, a database management system (DBMS) that is hosted on a computer generates a query tree that represents a database query that contains an expression that is represented by a subtree of the query tree. The DBMS generates a sequence of DSL instructions that represents the subtree. The sequence of DSL instructions is executed to evaluate the expression during execution of the database query. In an embodiment, an AST is generated from the sequence of DSL instructions. In an embodiment, the DSL AST is optimally rewritten based on a runtime feedback loop that includes dynamic profiling information.

Abstract: According to an embodiment, a method includes rewriting a particular query to generate a rewritten query. The particular query specifies a window function operator, a particular input to the window function operator, and an analytical function. Rewriting the particular query includes assigning the particular input to an intermediate relation and replacing the window function operator with a replacement operator. The method further includes assigning to the replacement operator an aggregate function corresponding to the analytical function, and the intermediate relation. In this embodiment, the method also includes placing a join operator that joins the intermediate relation with an output of the replacement operator.

Abstract: Techniques described herein propose a new RIDDecode operator in a QEP that uses ROWID lookup and fetch, instead of dictionary decoding, to retrieve decoded values, in order to reduce memory pressure and speed up processing.

Abstract: Unsorted sparse dictionary encodings are transformed into unsorted-dense or sorted-dense dictionary encodings. Sparse domain codes have large gaps between codes that are adjacent in order. Unlike spare codes, dense codes have smaller gaps between adjacent codes; consecutive codes are dense codes that have no gaps between adjacent codes. The techniques described herein are relational approaches that may be used to generate sparse composite codes and sorted codes.

Abstract: Techniques described herein propose a new RIDDecode operator in a QEP that uses ROWID lookup and fetch, instead of dictionary decoding, to retrieve decoded values, in order to reduce memory pressure and speed up processing.

Abstract: A method includes determining, whether: a first case is applicable, in which a first number of values of a first dataset and a second number of values of a second dataset total less than or equal to a third number of values of a register; a second case is applicable, in which the first and second numbers total more than the third number, and the first or second number is less than or equal to half of the third number; or a third case is applicable, in which the first and second numbers total more than the third number, and each of the first and second numbers is greater than half of the third number. In response to the determining, the method includes selectively loading to the register a first portion of the first dataset and a second portion of the second dataset, and performing conflict-detection for identifying one or more common values in the register loaded with the first portion and the second portion.

Abstract: Herein are machine learning (ML) feature processing and analytic techniques to detect anomalies in parse trees of logic statements, database queries, logic scripts, compilation units of general-purpose programing language, extensible markup language (XML), JAVASCRIPT object notation (JSON), and document object models (DOM). In an embodiment, a computer identifies an operational trace that contains multiple parse trees. Values of explicit features are generated from a single respective parse tree of the multiple parse trees of the operational trace. Values of implicit features are generated from more than one respective parse tree of the multiple parse trees of the operational trace. The explicit and implicit features are stored into a same feature vector. With the feature vector as input, an ML model detects whether or not the operational trace is anomalous, based on the explicit features of each parse tree of the operational trace and the implicit features of multiple parse trees of the operational trace.

Abstract: Unsorted sparse dictionary encodings are transformed into unsorted-dense or sorted-dense dictionary encodings. Sparse domain codes have large gaps between codes that are adjacent in order. Unlike spare codes, dense codes have smaller gaps between adjacent codes; consecutive codes are dense codes that have no gaps between adjacent codes. The techniques described herein are relational approaches that may be used to generate sparse composite codes and sorted codes.

Abstract: Techniques are provided for obtaining generic vectorized d-heaps for any data type for which horizontal aggregation SIMD instructions are not available, including primitive as well as complex data types. A generic vectorized d-heap comprises a prefix heap and a plurality of suffix heaps. Each suffix heap of the plurality of suffix heaps comprises a d-heap. A plurality of key values stored in the heap are split into key prefix values and key suffix values. Key prefix values are stored in the prefix heap and key suffix values are stored in the plurality of suffix heaps. Each entry in the prefix heap includes a key prefix value of the plurality of key values and a reference to the suffix heap of the plurality of suffix heaps that includes all key suffix values of the plurality of key values that share the respective key prefix value.

Abstract: Herein are computerized techniques for deploying JavaScript and TypeScript stored procedures and user-defined functions into a database management system (DBMS). In an embodiment, a computer generates a SQL call specification for each subroutine of one or more subroutines encoded in a scripting language. The generating is based on a signature declaration of the subroutine. Each subroutine comprises a definition of a stored procedure or a user-defined function. The computer packages the definition and the SQL call specification of each subroutine into a single bundle file. The definition and the SQL call specification of each subroutine are deployed into a DBMS from the single bundle file. Eventually, the SQL call specification of at least one subroutine is invoked to execute the definition of the subroutine in the DBMS.