Abstract: A serverless query processing system receives a query and determines whether the query is a recurring query or a non-recurring query. The system may predict, in response to determining that the query is the recurring query, a peak resource requirement during an execution of the query. The system may compute, in response to determining that the query is the non-recurring query, a tight resource requirement corresponding to an amount of resources that satisfy a performance requirement over the execution of the query, where the tight resource requirement is less than the peak resource requirement. The system allocates resources to the query based on an applicable one of the peak resource requirement or the tight resource requirement. The system then starts the execution of the query using the resources.

Abstract: In a set of data analytics queries, at least a one of the queries comprising more than one operator, and each query being at least one of i) a producer of data for an other query in the set, and ii) a consumer of data from an other query in the set. In such examples, one or more computing devices identify each producer/consumer relationship between the queries. The one or more computing devices identify one or more optimizations among the queries based on the identified relationships. The one or more computing devices then apply at least one identified optimization to at least one of the queries.

Abstract: In a set of data analytics queries, at least a one of the queries comprising more than one operator, and each query being at least one of i) a producer of data for an other query in the set, and ii) a consumer of data from an other query in the set. In such examples, one or more computing devices identify each producer/consumer relationship between the queries. The one or more computing devices identify one or more optimizations among the queries based on the identified relationships. The one or more computing devices then apply at least one identified optimization to at least one of the queries.

Abstract: Methods of machine learning for system deployments without performance regressions are performed by systems and devices. A performance safeguard system is used to design pre-production experiments for determining the production readiness of learned models based on a pre-production budget by leveraging big data processing infrastructure and deploying a large set of learned or optimized models for its query optimizer. A pipeline for learning and training differentiates the impact of query plans with and without the learned or optimized models, selects plan differences that are likely to lead to most dramatic performance difference, runs a constrained set of pre-production experiments to empirically observe the runtime performance, and finally picks the models that are expected to lead to consistently improved performance for deployment. The performance safeguard system enables safe deployment not just for learned or optimized models but also for additional of other ML-for-Systems features.

Abstract: Methods of machine learning for system deployments without performance regressions are performed by systems and devices. A performance safeguard system is used to design pre-production experiments for determining the production readiness of learned models based on a pre-production budget by leveraging big data processing infrastructure and deploying a large set of learned or optimized models for its query optimizer. A pipeline for learning and training differentiates the impact of query plans with and without the learned or optimized models, selects plan differences that are likely to lead to most dramatic performance difference, runs a constrained set of pre-production experiments to empirically observe the runtime performance, and finally picks the models that are expected to lead to consistently improved performance for deployment. The performance safeguard system enables safe deployment not just for learned or optimized models but also for additional of other ML-for-Systems features.

Abstract: Methods for optimization in query plans are performed by computing systems via a query optimizer advisor. A query optimizer advisor (QO-Advisor) is configured to steer a query plan optimizer towards more efficient plan choices by providing rule hints to improve navigation of the search space for each query in formulation of its query plan. The QO-Advisor receives historical information of a distributed data processing system as an input, and then generates a set of rule hint pairs based on the historical information. The QO-Advisor provides the set of rule hint pairs to a query plan optimizer, which then optimizes a query plan of an incoming query through application of a rule hint pair in the set. This application is based at least on a characteristic of the incoming query matching a portion of the rule hint pair.

Abstract: Implementations described herein relate to systems and methods to provide platform agnostic query acceleration. In some implementations, a method includes receiving, at a processor associated with a query acceleration service, a request from an client/application, wherein the request conforms to a particular wire protocol of a plurality of supported wire protocols, and wherein the request includes header data and body content data, analyzing the request to identify at least one of a query and a command in the body content data, determining an optimal matched model of the one or more query acceleration models, rewriting the query based on the optimal matched model, transmitting the rewritten query to the query processing platform, receiving a response to the rewritten query or the query from the query processing platform, and transmitting the received response to the application, wherein the transmission is configured based on the particular wire protocol.

Abstract: A serverless query processing system receives a query and determines whether the query is a recurring query or a non-recurring query. The system may predict, in response to determining that the query is the recurring query, a peak resource requirement during an execution of the query. The system may compute, in response to determining that the query is the non-recurring query, a tight resource requirement corresponding to an amount of resources that satisfy a performance requirement over the execution of the query, where the tight resource requirement is less than the peak resource requirement. The system allocates resources to the query based on an applicable one of the peak resource requirement or the tight resource requirement. The system then starts the execution of the query using the resources.

Abstract: A serverless query processing system receives a query and determines whether the query is a recurring query or a non-recurring query. The system may predict, in response to determining that the query is the recurring query, a peak resource requirement during an execution of the query. The system may compute, in response to determining that the query is the non-recurring query, a tight resource requirement corresponding to an amount of resources that satisfy a performance requirement over the execution of the query, where the tight resource requirement is less than the peak resource requirement. The system allocates resources to the query based on an applicable one of the peak resource requirement or the tight resource requirement. The system then starts the execution of the query using the resources.

Abstract: Solutions for optimizing job runtimes via prediction-based token allocation includes receiving training data comprising historical run data, the historical run data comprising job characteristics, runtime results, and a token count for each of a plurality of prior jobs, and the job characteristics comprising an intermediate representation and job graph data; based at least on the training data, training a token estimator, the token estimator comprising a machine learning (ML) model; receiving job characteristics for a user-submitted job; based at least on the received job characteristics, generating, with the token estimator, token prediction data for the user-submitted job; selecting a token count for the user-submitted job, based at least on the token prediction data; identifying the selected token count to an execution environment; and executing, with the execution environment, the user-submitted job in accordance with the selected token count.

Abstract: Shuffling of into partitions by first grouping input vertices of a limited number. Each group of input vertices may then be simply shuffled into a corresponding group of intermediate vertices, such as by broadcasting. A second grouping occurs in which the intermediate vertices are grouped by partition. The intermediate vertices then shuffle into corresponding output vertices for the respective partitions of that group. If the intermediate vertices are still too large, then this shuffling may involve recursively performing the shuffling just described, until ultimately the number of intermediate vertices shuffling into the output vertices is likewise limited. Thus, the final shuffling into the output vertices might also be simply performed by broadcasting.

Abstract: Methods of machine learning for system deployments without performance regressions are performed by systems and devices. A performance safeguard system is used to design pre-production experiments for determining the production readiness of learned models based on a pre-production budget by leveraging big data processing infrastructure and deploying a large set of learned or optimized models for its query optimizer. A pipeline for learning and training differentiates the impact of query plans with and without the learned or optimized models, selects plan differences that are likely to lead to most dramatic performance difference, runs a constrained set of pre-production experiments to empirically observe the runtime performance, and finally picks the models that are expected to lead to consistently improved performance for deployment. The performance safeguard system enables safe deployment not just for learned or optimized models but also for additional of other ML-for-Systems features.

Abstract: Described herein is a system and method for detecting and reusing overlapping computations. Overlapping subgraphs of the query are determined using a normalized signature for a particular subgraph that identifies a particular subgraph across recurring instances of data. A normalized signature for each overlapping subgraph for the determined overlapping subgraphs of the query is provided. For each overlapping subgraph determined to be materialized: whether or not the particular subgraph has been materialized is determined using a precise signature corresponding to a normalized signature of the particular overlapping subgraph. The precise signature identifies a particular subgraph corresponding to the normalized signature within a particular recurring instance of data. When the particular subgraph has not been materialized, the subgraph is materialized and used to respond to the query. When the particular subgraph has been materialized, the materialized subgraph is used to respond to the query.

Abstract: A serverless query processing system receives a query and determines whether the query is a recurring query or a non-recurring query. The system may predict, in response to determining that the query is the recurring query, a peak resource requirement during an execution of the query. The system may compute, in response to determining that the query is the non-recurring query, a tight resource requirement corresponding to an amount of resources that satisfy a performance requirement over the execution of the query, where the tight resource requirement is less than the peak resource requirement. The system allocates resources to the query based on an applicable one of the peak resource requirement or the tight resource requirement. The system then starts the execution of the query using the resources.

Abstract: The cloud-based query workload optimization system disclosed herein the cloud-based query workloads optimization system receives query logs from various query engines to a cloud data service, extracts various query entities from the query logs, parses query entities to generate a set of common workload features, generates intermediate representations of the query workloads, wherein the intermediate representations are agnostic to the language of the plurality of the queries, identifies a plurality of workload patterns based on the intermediate representations of the query workloads, categorizes the workloads in one or more workload type categories based on the workload patterns and the workload features, and selects an optimization scheme based on the category of workload pattern.

Abstract: Methods, systems, apparatuses, and computer program products are provided for evaluating a resource consumption of a query. A logical operator representation of a query generated to be executed (e.g., obtained from a query generating entity) may be determined. The logical operator representation may be transformed to a plurality of different physical operator representations for executing the query. A plurality of resource consumption models may be applied to each of the physical operator representations to determine a resource consumption estimate for the physical operator representation. The resource consumption models may be trained in different manners based at least on a history of query executions, such that each model may have different granularity, coverage and/or accuracy characteristics in estimating a resource consumption of a query.

Abstract: Described herein is a system and method for detecting and reusing overlapping computations. Overlapping subgraphs of the query are determined using a normalized signature for a particular subgraph that identifies a particular subgraph across recurring instances of data. A normalized signature for each overlapping subgraph for the determined overlapping subgraphs of the query is provided. For each overlapping subgraph determined to be materialized: whether or not the particular subgraph has been materialized is determined using a precise signature corresponding to a normalized signature of the particular overlapping subgraph. The precise signature identifies a particular subgraph corresponding to the normalized signature within a particular recurring instance of data. When the particular subgraph has not been materialized, the subgraph is materialized and used to respond to the query. When the particular subgraph has been materialized, the materialized subgraph is used to respond to the query.

Abstract: Shuffling of into partitions by first grouping input vertices of a limited number. Each group of input vertices may then be simply shuffled into a corresponding group of intermediate vertices, such as by broadcasting. A second grouping occurs in which the intermediate vertices are grouped by partition. The intermediate vertices then shuffle into corresponding output vertices for the respective partitions of that group. If the intermediate vertices are still too large, then this shuffling may involve recursively performing the shuffling just described, until ultimately the number of intermediate vertices shuffling into the output vertices is likewise limited. Thus, the final shuffling into the output vertices might also be simply performed by broadcasting.