Abstract: A method, a system, and a computer program product for generating a query executable plan. A query requiring access to data stored in a database system is received. Based on the received query, a query execution plan having a plurality of query execution pipelines is generated. Each query execution pipeline in the plurality of query execution pipelines is configured to execute a plurality of operations in a predetermined order associated with each query execution pipeline. The generated query execution plan is fragmented into a plurality of fragments. Each fragment has one or more query execution pipelines in the plurality of query execution pipelines. The received query is executed by executing each fragment in the plurality of fragments.

Abstract: Tiered storage may be implemented for processing data. Data processors may maintain some of a data set, including user data and metadata describing the user data, locally. The data set is also maintained a data store remote to the data processor. When processing requests are received, a determination is made as to whether the local portions of the data set can execute the processing request or one or more additional portions of the data set are needed from the remote data store. If additional portions of the data set are needed, then a request may be sent to the data store for the additional portions. Once received, the data processor may execute the processing request utilizing the additional portions. Portions of the data set maintained locally at the data processor may be selected and flushed from local storage to the remote data store.

Abstract: The present disclosure relates to computer-implemented methods, software, and systems for managing data replication between different source sections and target sections in response to received copy instructions associated with copy profiles. In response to evaluating statistical metadata identifying whether data records in relation to at least one client are included for a table from a first set of tables, a first subset of tables from the first set of tables is determined. In response to evaluating update metadata defining latest updates of tables from the first subset of tables, a second subset of tables from the first subset of tables is determined that defines tables that include updated data records relevant for copying. The second subset tables are iteratively evaluated to define corresponding operations to be performed for tables at the target section and at the source section in the database in relation to the requested copy operation.

Abstract: Embodiments of the present disclosure are directed to techniques for monitoring and orchestrating the use and generation of collaborative data in a trustee environment subject to configurable constraints. A user interface can be provided to enable tenants to specify desired computations and constraints on the use and access to their data. A constraint manager can communicate with various components in the trustee environment to implement the constraints. For example, requests to execute an executable unit of logic such as a command or function call may be issued to the constraint manager, which can grant or deny permission. Permission may be granted subject to one or more conditions that implement the constraints, such as requiring the replacement of a particular executable unit of logic with a constrained executable unit of logic. As constraints are applied, any combination of schema, constraints, and/or attribution metadata can be associated with the data.

Abstract: Embodiments for intelligent automated feature engineering for relational data in a computing environment by a processor. Indices may be automatically selected and built from one or more columns of one or more tables in a relational database using one or more automated feature engineering models that include a set of queries. One or more features may be determined using a set of queries of an automated feature engineering models to execute for a scoring operation.

Abstract: A DBMS query-optimization module receives a corpus of training data that contains data-access requests, such as SQL queries. Each request specifies data to be accessed but does not specify a query plan that the database should use to generate the requested data. The module identifies, in each received request, parameters, such as join methods and access methods, that can vary among query plans; and variables that cannot be assigned values until the query is actually processed. The system generates a set of queries, each of which implements a distinct query plan, that specify every viable permutation of values of the parameters and of the variables. The generated queries are added to the received corpus, which is forwarded to a machine-learning module in order to train the query-optimizer to select query plans that incur the lowest resource costs when servicing a particular type of query.

Abstract: In some aspects, there is provided a method including receiving an execution plan file, the execution plan file utilizing at least one operator of interest and further utilizing other actions separate from the at least one operator of interest. The method further includes forming an execution plan object by modifying the execution plan file by isolating the at least one operator of interest from the other actions. The method further includes performing a series of tests executing an extended execution plan object. The series of tests can include receiving the input data identified by the one or more pointers in the extended execution plan object, executing the extended execution plan object using the received input data, measuring, based on the execution of the extended execution plan object, at least one cost metric representative of execution of the at least one operator of interest, and outputting the measured cost metric.

Abstract: Sub-plans are executed in parallel using a plurality of execution nodes, which can be part of a data platform. In particular, sub-plans (e.g., fragments or portions of one or more child operators) of a root operator are identified in a query plan such that the identified sub-plans that are candidates for execution on a single execution node, determine a cost estimate for causing the candidate sub-plans to be executed in parallel using multiple execution nodes, and cause the candidate sub-plans to be executed in parallel based on the cost estimate.

Abstract: A system and method for displaying data using temporal granularities. The method includes determining at least one first dataset of a plurality of datasets based on at least one temporal data requirement, wherein the plurality of datasets is generated based on a data model, wherein each of the plurality of datasets is generated based further on a distinct temporal granularity of a plurality of temporal granularities, wherein the distinct temporal granularity of each of the at least one first dataset meets at least one of the at least one temporal data requirement; and querying the determined at least one first dataset in order to obtain at least one query result.

Abstract: A federated database-management system receives an SQL query or other type of data-access request. The federated system's host DBMS parses, rewrites, and optimizes the request into an optimal data-access plan, then determines which portions of the plan require access to data stored on the federated systems' remote databases. The federated host partitions the plan into subplans that each represent instructions of the original data-access request that were directed to a corresponding remote database of the federated DBMS. Each subplan is then transmitted to its corresponding remote database, which directly executes the subplan and returns results to the host. If necessary, a subplan is translated from an original generic access-plan format into a database-specific format required by its corresponding remote database.

Abstract: A query processing system generates and employs an inverted index of predicates for predicate statement evaluation. The inverted index maps values for variables to predicates that evaluate to true for the corresponding values. When querying input data, the query processing system identifies a value for each variable in the input data. For each value and variable pair, the query processing system identifies predicates mapped to the value for the variable in the inverted index. The query processing system evaluates the predicate statements by treating each predicate identified from the inverted index as true. In some configurations, the query processing system represents each predicate statement using a bit string and evaluates the predicate statements for the input data by setting bits to one for predicates identified from the inverted index and determining predicate statements that evaluate to true based on the bit strings.

Abstract: Techniques for a system capable of performing low-latency database query processing are disclosed herein. The system includes a gateway server and a plurality of worker nodes. The gateway server is configured to divide a database query, for a database containing data stored in a distributed storage cluster having a plurality of data nodes, into a plurality of partial queries and construct a query result based on a plurality of intermediate results. Each worker node of the plurality of worker nodes is configured to process a respective partial query of the plurality of partial queries by scanning data related to the respective partial query that stored on at least one data node of the distributed storage cluster and generate an intermediate result of the plurality of intermediate results that is stored in a memory of that worker node.

Abstract: Embodiments of the present disclosure may provide a dynamic query execution model. This query execution model may provide acceleration by scaling out parallel parts of a query (also referred to as a fragment) to additional computing resources, for example computing resources leased from a pool of computing resources. Execution of the parts of the query may be coordinated by a parent query coordinator, where the query originated, and a fragment query coordinator.

Abstract: A system for data migration is disclosed. The system may receive a migration request comprising a source file path and a target file location. The system may capture source file metadata based on the source file path and the migration request. The system may transfer a source file from a first data environment to an intermediate data environment via a first transfer process. The system may transfer the source file from the intermediate data environment to a second data environment via a second transfer process.

Abstract: An approach is provided for optimizing data fetching. A query employing a method to fetch data from a JSON document is received. An amount of time required to execute the query and a number of nested layers in a traversal of the JSON document required to fetch the data are determined. Based on the amount of time and the number of nested layers, a cost associated with an execution of the query is calculated. The cost is determined to exceed a threshold value. Responsive to the determination that the cost exceeds the threshold value and using historical query patterns and historical query execution times, a schema of the JSON document is re-designed. The data is fetched from the JSON document using the re-designed schema.

Abstract: An original query execution plan of a database query is received. The original query execution plan represents a tree of operators. Source code for the original query execution plan is generated by a single traversal of the tree of operators. The generated source code is compiled into native machine code. The native machine code represents a simplified native access plan (SNAP).

Abstract: Techniques for managing database workloads using similarity measures based on queries executed are described. Classical techniques from information retrieval are applied to the domain of database workload management. Specifically, the technique of using document term vectors to compute similarity measures are applied using the conceptual mapping of SQL workloads as “documents” composed of SQL queries as “terms.” The techniques include generating two or more sets of workloads with each workload representing a set of queries executed on at least one database. Based on the sets of workloads, workload term vectors are calculated that represent the set of queries executed on the database. Then, based on the calculated workload vectors, a similarity score is generated between the two or more sets of workloads.

Abstract: In some embodiments, a system is provided, comprising: memory storing instructions that, when executed, cause a processor to: submit a first database query; receive a runtime to execute the first database query using a plan selected by a query optimizer; receive runtimes to execute the first database query using a plurality of test plans; determine, based on the runtimes, a metric indicative of the effectiveness of the query optimizer; and cause the metric indicative of the effectiveness of the query optimizer to be presented to a user.

Abstract: A novel distributed graph database is provided that is designed for efficient graph data storage and processing on modern computing architectures. In particular a single node graph database and a runtime & communication layer allows for composing a distributed graph database from multiple single node instances.

Abstract: The present disclosure relates to a method of managing requests to a key-value database. A non-limiting example of the method includes receiving a request that includes a number of keys. The number of keys can be compared with a first threshold number and second threshold number. If the number of keys exceeds the first threshold number, the request can be split. If the number of keys is smaller than the second threshold number, the request can be merged with at least one previous or subsequent request. Requests resulting from the splitting and merging steps can be submitted to the key-value database for further processing of the submitted requests.

Abstract: System, method, and various embodiments for providing a data access and recommendation system are described herein. An embodiment operates by identifying a column access of one or more data values of a first column of a plurality of columns of a table of a database during a sampling period. A count of how many of the one or more data values are accessed during the column access are recorded. A first counter, corresponding to the first column and stored in a distributed hash table, is incremented by the count. The sampling period is determined to have expired. A load recommendation on how to load data values into the first column based on the first counter is computed. The load recommendation for implementation into the database for one or more subsequent column accesses is provided.

Abstract: Software is increasingly being developed as a collection of loosely coupled applications. Loosely coupled applications exchange data by publishing data to and retrieving data from a data store, such as a database, a file located on a storage cluster, etc. Data produced by one application and consumed by another is referred to as a data dependency. In some embodiments, an application's data dependencies are identified by analyzing cached query plans associated with the application. Query plans include a hierarchical representation of a query, where non-leaf nodes represent commands and leaf nodes identify data dependencies. An application's data dependencies are identified by traversing the hierarchical representation of the query. Data dependencies consumed by the application are identified by finding leaf nodes that descend from a read command, while data dependencies produced by the application are identified by finding leaf nodes that descend from a write command.

Abstract: Systems and methods for computer memory management by a memory coordinator and a plurality of memory consumers. An urgency and memory quota of each memory consumer is initialized by the memory coordinator, which then adjusts the memory quota of each memory consumer such that the sum of the memory quota of each memory consumer does not exceed a finite amount of computer memory. Each memory consumer adjusts its memory usage in response to the quota input and urgency input from the memory coordinator.

Abstract: Using a step editor for data preparation includes: receiving an indication of a user input with respect to at least some of a set of sequenced data preparation operations on a set of data; generating, using one or more processors, a signature based at least in part on the set of sequenced data preparation operations, references to the set of data, and the user input; using the generated signature to determine whether there exists a cached result associated with the set of sequenced data preparation operations, the references to the set of data, and the user input; based at least in part on the determination, obtaining a data traversal program representing a result associated with the set of sequenced operations, the references to the set of data, and the user input; and providing output based at least in part on the result represented by the obtained data traversal program.

Abstract: A database engine receives a query batch of database queries from a client. The database engine identifies one or more object model queries from the query batch. Each object model query includes an outer-most outer-join that joins a respective dimension subquery and respective aggregated measure subqueries. The database engine forms a plurality of candidate subqueries by peeling off the respective outer-most outer-join for each of the object model queries. The database engine then fuses at least some of the plurality of candidate subqueries to form a set of optimized subqueries. The set of optimized subqueries has fewer subqueries than the plurality of candidate queries. The database engine also forms an optimized execution plan based on the set of one or more optimized subqueries. The database engine subsequently obtains a result set from the database based on the optimized execution plan, and returns the result set to the client.

Abstract: The embodiments of this application provide an uncorrelated subquery optimization method and apparatus, and a storage medium. The method includes determining whether there is an uncorrelated subquery statement in a target clause in a database query statement. In response to the determination that there is the uncorrelated subquery statement in the target clause in the database query statement, the method includes obtaining an estimated number of rows of an execution result set corresponding to the target clause; and determining whether the estimated number of rows is less than a preset threshold. In response to the determination that the estimated number of rows is less than a preset threshold, the method includes executing the uncorrelated subquery statement, and rewriting the target clause according to an execution result set of the uncorrelated subquery statement, to eliminate the uncorrelated subquery statement.

Abstract: A data-instantiator method handle is configured to create a target object based on a stream object. One type of data-instantiator method handles is a Stream Object Processor method handle (SOP_mh). A SOP_mh is a runtime-computed constant in a runtime constant pool. A runtime environment resolves the SOP_mh lazily responsive to a request to access the SOP_mh. The runtime environment invokes the SOP_mh to create a target object based on a stream object. By virtue of being a constant in the runtime constant pool, the SOP_mh is a candidate for optimization by a dynamic compiler in the runtime environment. The dynamic compiler may elect to constant fold the value of the SOP_mh and inline any code or executable logic that the SOP_mh refers to.

Abstract: Embodiments utilize trained query performance machine learning (QP-ML) models to predict an optimal compute node cluster size for a given in-memory workload. The QP-ML models include models that predict query task runtimes at various compute node cardinalities, and models that predict network communication time between nodes of the cluster. Embodiments also utilize an analytical model to predict overlap between predicted task runtimes and predicted network communication times. Based on this data, an optimal cluster size is selected for the workload. Embodiments further utilize trained data capacity machine learning (DC-ML) models to predict a minimum number of compute nodes needed to run a workload. The DC-ML models include models that predict the size of the workload dataset in a target data encoding, models that predict the amount of memory needed to run the queries in the workload, and models that predict the memory needed to accommodate changes to the dataset.

Abstract: A data input sub-system of a large scale application specific computing system receives a data set that includes a plurality of records, each with a plurality of data fields, and divides the data set into a plurality of data segments. The data input sub-system further restructures records of data segments based on a key field of the plurality of data fields to produce restructured data segments and generates storage instructions for storing the restructured data segments. A data storage and processing sub-system of the computing system interprets the storage instructions to determine resources to engage and stores the restructured data segments using engaged resources. A query and results sub-system of the computing system generates an initial query plan based on a data processing request, optimizes the initial query plan to produce an optimized query plan, and sends the optimized query plan to the data storage and processing sub-system for execution.

Abstract: Predictive execution of query flows in an application aware database environment. A repository of previously received and registered database queries along with at least corresponding metadata having information about database query flows generating the database queries is maintained. Application metadata corresponding to a subsequent database query is received. The repository is checked to determine if the application metadata matches one of the previously received and registered database query flows. One or more queries corresponding to the query flow from the repository is/are retrieved if a match is determined. Execution of the retrieved one or more database queries is started prior to receiving the query from outside the repository.

Abstract: Systems and methods are disclosed for executing a query that includes an indication to process data managed by an external data system. The system identifies the external data system that manages the data to be processed, and generates a subquery for the external data system indicating that the results of the subquery are to be sent to multiple worker nodes. The system also generates instructions for multiple worker nodes to receive and process results of the subquery from the external data system.

Abstract: A system and method for managing data storage and data access with querying data in a distributed system without buffering the results on intermediate operations in disk storage.

Abstract: A tool for combining common processes shared by at least two or more sub-queries within a query is provided. The tool determines whether one or more sub set relationships are shared between the at least two or more sub-queries. Responsive to a determination that one or more sub set relationships are shared between the at least two or more sub-queries, the tool determines an order class for the at least two or more sub-queries based on the one or more sub set relationships, wherein determining the order class includes transforming the query to include one or more differing aspects within the single shared common process, with the one or more differing aspects arranged based, at least in part, on a query style, a query type, and a query function. Responsive to determining an access path for the query, the tool executes the access path during run-time for data accessing.

Abstract: A method, system and computer program product are provided. A first index scan of records of a database table is performed. A first index used by the first index scan is specified in a DBMS query statement as being ordered by one or more field values of one or more fields of the first index. A second scan of the database table is performed to determine whether each respective record of the second scan is qualified based on one or more fields of the each respective record of the second scan and a predicate. A lookup data structure is built indicating whether each of the respective records of the second scan is qualified. The lookup data structure is used to determine whether a record of the first index scan is qualified when the lookup data structure becomes available.

Abstract: Methods, non-transitory computer readable media, and infrastructure management devices that inventory, and facilitate improved security of, public cloud databases are illustrated. With this technology, access data for one or more accounts associated with one or more public cloud networks is obtained. This technology then periodically identifies, stores an indication of, and obtains and stores a state of, a plurality of databases hosted by the public cloud networks using the obtained access data and one or more application programming interfaces (APIs) provided by the public cloud networks for each of one or more types of the databases. A dashboard comprising an inventory of the databases for a historical time period is then generated, based on the stored indication and state of each of the databases, and output.

Abstract: A unified access layer (UAL) and scalable query engine receive queries from various interfaces and executes the queries with respect to non-heterogeneous data management and analytic computing platforms that are sources of record for data they store. Query performance is monitored and used to generate a query performance model. The query performance model may be used to generate alternatives for queries of users or groups of users or to generate policies for achieving a target performance. Performance may be improved by monitoring queries and retrieving catalog data for databases referenced and generating a recommendation model according to them. Duplicative or overlapping sources may be identified based on the monitoring and transformations to improve accuracy and security may be suggested. A recommendation model may be generated based on analysis of queries received through the UAL. Transformations may be performed according to the recommendation model in order to improve performance.

Abstract: A system and method for managing data storage and data access with querying data in a distributed system without buffering the results on intermediate operations in disk storage.

Abstract: Queries executed against a materialized view can execute up to orders of magnitude faster than equivalent queries on a source (or base) table. However, although a query can reference a materialized view directly, a user (e.g., query author) may not know about a relevant materialized view. Moreover, if a source table has multiple materialized views generated, the user may not know which materialized view to reference in the query. Thus, embodiments of the present disclosure provide techniques for automatically rewriting queries directed to a source table to utilize existing materialized views.

Abstract: Implementation of the present disclosure relates to a method, system and computer program product for improvement of query execution. According to one embodiment of the present invention, there is provided a method. In the method a query on data organized in a first data structure is received first. And then based on at least one property associated with the query, a second data structure is selected from a plurality of candidate data structures, wherein a performance measure of the query on the data organized in the second data structure is better than the performance measure of the query on the data organized in the first data structure. And at last the query on the data organized in the second data structure is executed. In other implementations, a system and a computer program product are disclosed.

Abstract: A first query execution plan generated for a query on a second time the query was processed by a database is compared against a dynamically generated second query plan generated based on statistics only dynamic feedback for the second time the query is processed by the database. A determination is made on the second time as to whether to cache the first query execution plan, the second query execution plan, or no plan for third or more times the query is processed by the database. The query can be non-parameterized or parameterized.

Abstract: Techniques for implementing mechanisms for chaining continuous queries are provided. In some examples, results of a first continuous query may be stored in a data object. For example, the first continuous query may be configured to process business event data and store the results in the data object. Additionally, a second continuous query may then be processed based at least in part on the results stored in the data object.

Abstract: Disclosed are a method and a device for processing distributed data. The method includes: integrating and configuring data analysis services of multiple users with different data analysis requirements into a distributed computing engine program to obtain an analysis service data package; configuring a distributed scheduler in the cluster server according to the analysis service data package, and calling the distributed scheduler to monitor a message content transmitted by a message middleware including multiple data analysis services to be executed; and generating a distributed data execution plan according to the message content, and performing distributed scheduling calculation on the distributed data execution plan to obtain a distributed calculation result.

Abstract: Methods, systems, and computer program products for dynamically managing predicate expression columns in an encrypted database are provided herein.

Abstract: An approach is provided for accessing data in a database. A query which accesses item(s) in the data is received. The query has multiple conditions whose execution produces a final result. Amounts of filtering of portions of the data caused by utilizing different types of indexes to filter the portions are determined. Greatest amounts of filtering for the portions are determined. Types of indexes are selected so that a utilization of a given type of index causes a greatest amount of filtering for a given portion. Data access methods for the portions of data are generated. During runtime, process tasks are generated for the multiple conditions in the query. An execution flow is controlled. The execution flow executes the process tasks by utilizing the data access methods to generate initial results. The final result of the query is generated by combining the initial results from the process tasks.

Abstract: A method, a system, and a computer program product for executing a query. A query plan for execution of a query is generated. The query requires access to at least one table stored in a database system. The query includes one or more filter predicates. A filter predicate in the one or more filter predicates is selected. For the selected filter predicate, a plurality of cost function values associated executing a filter evaluation of the selected filter predicate are determined. Filter evaluation of the selected predicate is executed in accordance with at least one determined cost function value in the plurality of cost function values.

Abstract: A query referencing an external table function provided by a remote software component is received. Requests to execute the external table function on input data are sent to a proxy service. A first request includes a batch of input rows from the input data. A first response to the first request received from the proxy service includes a first portion of result data and a pagination token. The pagination token indicates that at least a second portion of the result data corresponding to the first batch of input rows is to be obtained from the remote software component. Based on the pagination token, a second request is sent to obtain the second portion of the result data. One or more responses are received from the proxy service that comprise at least the second portion of the result data. The result data is processed according to the query.

Abstract: Disclosed aspects include the database management system detecting that a first data storage medium includes a first set of data of a database. The first data storage medium has a first access-factor. The database management system detects that a second data storage medium includes a second set of data of the database. The second data storage medium has a second access-factor which differs from the first access-factor. Based on a data storage relationship, the database management system determines a query plan to access both the first set of data with respect to the first data storage medium and the second set of data with respect to the second data storage medium. The database management system establishes the query plan.

Abstract: A computing system determines a response to a query. A bin start value and a bin stop value is defined for each bin based on an input bin option. End nodes are split based on the bin start value and the bin stop value of each bin to define a second plurality of end nodes. Each start node of a plurality of start nodes that is connected to each end node of the second plurality of end nodes is identified based on the respective link attributes of a plurality of link attributes. Overlapping start nodes of the plurality of start nodes that overlap at an end node of the second plurality of end nodes are identified based on a predefined overlap query graph that defines a connectivity to identify between a start node and the end node. The identified overlapping start nodes are output as a response to the predefined overlap query graph.

Abstract: A database system may receive a query and form a plan for executing the query using a hypothetical index selected based on columns referenced by the query. The database system may compare the hypothetical index to indexes on the computing nodes of the database system. Based on the comparison, the database system may form a new index, based on the hypothetical index, on one of the computing nodes. An additional computing node may be added to the database system for the purpose of maintaining the new index. Subsequent queries having commonality with the first query may be processed on a node maintaining the first index.

Abstract: A computer-implemented automated review method for structured query language (SQL) source code files seeking to access a data store comprises the steps of calculating an SQL complexity score; setting an issue flag if the SQL complexity score is greater than a threshold; checking the SQL source code for SQL-92 compliance; setting the issue flag if any of the SQL source code files is not SQL-92 compliant; comparing a type of data in the SQL source code files to be input to the data store with a type of a field in the data store set to receive the input data; setting the issue flag if the type of any input data does not match the type of a corresponding data store field; checking the SQL source code for SQL triggers set to activate when selected SQL commands are executed; and setting the issue flag if there are any SQL triggers.