Abstract: Provided herein are systems, methods, and computer-storage media for managing data skew in hash join operations. A skew manager partitions build-side row data into multiple sets corresponding to hash-join-build (HJB) instances based on hash values. The skew manager detects skew in a build-side row set associated with a first HJB instance by analyzing the number of rows. Upon detecting skew, the skew manager redirects data rows to at least a second HJB instance. The method involves configuring skew caches, generating histograms, and detecting frequent hash values to identify skew. It also includes communicating skew notifications, broadcasting probe-side row data, and adjusting partitioning of probe-side data. The disclosed techniques further include buffering build-side row sets in streams and performing join operations based on these streams, enhancing efficiency in distributed computing environments.

Abstract: An input file is split into a multiple chunks. Each chunk is assigned to one of multiple worker nodes assigned to processing the input file. Refined boundaries for the multiple chunks are calculated. The refined boundaries define multiple adjusted chunks. The calculating of the refined boundaries comprises determining summary statistics for the multiple chunks, determining start positions of the multiple chunks based on the summary statistics, and determining end positions of the multiple chunks based on the start positions of the multiple chunks. Data scan passes are performed on the multiple adjusted chunks based on the refined boundaries to populate one or more data structures for storing information extracted from the input file.

Abstract: The subject technology receives a query plan, the query plan comprising a set of query operations, the set of query operations including at least one aggregation and at least one join operation. The subject technology analyzes the query plan to identify an aggregation that is redundant. The subject technology removes the aggregation based at least in part on the analyzing. The subject technology determines at least one aggregation property corresponding to at least one query operation of the query plan. The subject technology inserts at least one adaptive aggregation operator in the query plan based at least in part on the at least one aggregation property, the at least one aggregation property comprising a set of aggregation properties. The subject technology provides a modified query plan based at least in part on the inserted at least one adaptive aggregation operator in the query plan.

Abstract: A method includes generating, by at least one hardware processor of a first computing node, a plurality of hash values using build-side row data. A frequent hash value of the plurality of hash values is detected based on row size associated with a plurality of build-side row sets including the build-side row data. A plurality of hash partitions of the build-side row data is generated using a build-side row set of the plurality of build-side row sets that includes the frequent hash value. The plurality of hash partitions of the build-side row data is distributed to a corresponding plurality of hash-join-build (HJB) instances associated with a plurality of join operations.

Abstract: The subject technology sends, from a child rowset operator (RSO) instance, a first request for performing a user defined aggregate function (UDAF) to a user defined function (UDF) server to initialize an aggregate state for a set of aggregation groups and update aggregated states for each aggregation group from the set of aggregation groups, the first request including a set of input rows. The subject technology receives information comprising a computation status of the UDAF. The subject technology sends, by the child RSO instance, a second request to the UDF server to update the aggregated states for each aggregation group from the set of aggregation groups, the second request including a second set of input rows. The subject technology receives an aggregate states vector with one entry per aggregation group. The subject technology sends, by the child RSO instance, the aggregate states vector to a parent RSO instance.

Abstract: A join decision manager (JDM) generates a data processing pipeline. The data processing pipeline includes at least one join operation associated with build-side row data and probe-side row data. The JDM determines the maximum cardinality associated with the probe-side row data. The JDM determines size of the build-side row data at a decision node of the at least one join operation. The JDM configures execution of the at least one join operation as one of a broadcast join or a hash-hash join based on the size of the build-side row data and the maximum cardinality.

Abstract: A system includes at least one hardware processor and memory storing instructions. The processor generates a query plan for a received query. The query plan includes multiple hash-join-build and hash-join-probe operations. A primary decision node is configured in the query plan. The primary decision node receives build-side data information from the hash-join-build operations. For each hash-join-build operation, a memory amount for performing a broadcast is determined. A subset of hash-join-build operations is selected for broadcast join distribution by comparing the memory amount to a broadcast memory threshold. The system selects a broadcast join distribution for the subset and a hash-hash join distribution for the remaining hash-join-build operations. The query plan is executed using the broadcast join distribution for the selected subset and the hash-hash join distribution for the remaining operations. This approach optimizes memory usage and join distribution during query execution.

Abstract: Provided herein are systems and methods for hash-join broadcast decision making. For example, a method includes generating a query plan for a received query. The query plan includes a plurality of join operations with a plurality of hash-join-build (HJB) operations and a plurality of hash-join-probe (HJP) operations. A decision node of a plurality of decision nodes of the query plan is configured as a primary decision node. Build-side data information associated with build-side data and received from the plurality of HJB operations is decoded by the primary decision node. A data distribution method is determined by the primary decision node for each HJB operation of the plurality of HJB operations based on the build-side data information. The query plan is executed based on distributing the build-side data to the plurality of HJP operations using the data distribution method for each HJB operation of the plurality of HJB operations.

Abstract: Provided herein are systems and methods for hash-join broadcast decision making. For example, a method includes generating a query plan for a received query. The query plan includes a plurality of join operations with a plurality of hash-join-build (HJB) operations and a plurality of hash-join-probe (HJP) operations. A decision node of a plurality of decision nodes of the query plan is configured as a primary decision node. Build-side data information associated with build-side data and received from the plurality of HJB operations is decoded by the primary decision node. A data distribution method is determined by the primary decision node for each HJB operation of the plurality of HJB operations based on the build-side data information. The query plan is executed based on distributing the build-side data to the plurality of HJP operations using the data distribution method for each HJB operation of the plurality of HJB operations.

Abstract: A method implementing a fault-tolerant data warehouse using availability zones includes allocating a plurality of processing units to a data warehouse, the processing units located in different availability zones, an availability zone comprising one or more data centers. The method further includes routing a query to a processing unit within the data warehouse, the query having a common session identifier with a query previously provided to the processing unit, the processing unit determined to be caching a data segment associated with a cloud storage resource independent of the plurality of processing units. The method further includes, as a result of monitoring a number of queries running at an input degree of parallelism, determining that the processing capacity of the processing units has reached a threshold; and changing a total number of processing units using the input degree of parallelism and the number of queries.

Abstract: A method implementing a fault-tolerant data warehouse including allocating a plurality of processing units to a data warehouse, the processing units located in different availability zones, an availability zone comprising one or more data centers. The method further includes, as a result of monitoring a number of queries running at an input degree of parallelism on the plurality of processing units of the data warehouse, determining that the number of queries is serviceable by one fewer processing units. The method further includes routing a query from a first processing unit to a second processing unit within the data warehouse, the query having a common session identifier with another query previously provided to the second processing unit, the second processing unit determined to be caching a data segment associated with a cloud storage resource, usable by the query, and removing the first processing unit from the data warehouse.

Abstract: Provided herein are systems, methods, and computer-storage media for managing data skew in hash join operations. A skew manager partitions build-side row data into multiple sets corresponding to hash-join-build (HJB) instances based on hash values. The skew manager detects skew in a build-side row set associated with a first HJB instance by analyzing the number of rows. Upon detecting skew, the skew manager redirects data rows to at least a second HJB instance. The method involves configuring skew caches, generating histograms, and detecting frequent hash values to identify skew. It also includes communicating skew notifications, broadcasting probe-side row data, and adjusting partitioning of probe-side data. The disclosed techniques further include buffering build-side row sets in streams and performing join operations based on these streams, enhancing efficiency in distributed computing environments.

Abstract: Joining data using a disjunctive operator is described. An example computer-implemented method can include generating, with a processing device, a query plan for a query, the query comprising a join operator expression for a disjunctive predicate, wherein the join operator expression includes a conjunctive predicate and a disjunctive operator. The method may further include generating a bloom filter for the disjunctive operator. Additionally, the method may include generating a result set as a result of evaluating the join operator expression using the disjunctive operator and bloom filter for the disjunctive predicate.

Abstract: A method includes generating, by at least one hardware processor of a first computing node, a plurality of hash values using build-side row data. A frequent hash value of the plurality of hash values is detected based on row size associated with a plurality of build-side row sets including the build-side row data. A plurality of hash partitions of the build-side row data is generated using a build-side row set of the plurality of build-side row sets that includes the frequent hash value. The plurality of hash partitions of the build-side row data is distributed to a corresponding plurality of hash-join-build (HJB) instances associated with a plurality of join operations.

Abstract: Joining data using a disjunctive operator using a lookup table is described. An example computer-implemented method can include receiving a query with a set of conjunctive predicates and a set of disjunctive predicates. The method may also include generating a lookup table for each predicate in the sets of conjunctive predicates and disjunctive predicates. The method, for each row in a probe-side table, may also further include looking up a value associated with that row in each of the lookup tables and adding the row to a results set when there is a match. Additionally, the method may also include returning the results set.

Abstract: The subject technology receives a query plan, the query plan comprising a set of query operations, the set of query operations including at least one aggregation and at least one join operation. The subject technology analyzes the query plan to identify an aggregation that is redundant. The subject technology removes the aggregation based at least in part on the analyzing. The subject technology determines at least one aggregation property corresponding to at least one query operation of the query plan. The subject technology inserts at least one adaptive aggregation operator in the query plan based at least in part on the at least one aggregation property, the at least one aggregation property comprising a set of aggregation properties. The subject technology provides a modified query plan based at least in part on the inserted at least one adaptive aggregation operator in the query plan.

Abstract: Provided herein are systems and methods for handling build-side skew. For example, a method includes computing a plurality of hash values for a join operation. The join operation uses a corresponding plurality of row sets. The plurality of hash values are sampled to detect a frequent hash value. A build-side row set is partitioned using the frequent hash value to generate a partitioned build-side row set. The build-side row set is selected from the plurality of row sets. The partitioned build-side row set is distributed to a plurality of hash-join-build (HJB) instances executing at a corresponding plurality of servers.

Abstract: A method implementing a fault-tolerant data warehouse including allocating a plurality of processing units to a data warehouse, the processing units located in different availability zones, an availability zone comprising one or more data centers. The method further includes, as a result of monitoring a number of queries running at an input degree of parallelism on the plurality of processing units of the data warehouse, determining that the number of queries is serviceable by one fewer processing units. The method further includes routing a query from a first processing unit to a second processing unit within the data warehouse, the query having a common session identifier with another query previously provided to the second processing unit, the second processing unit determined to be caching a data segment associated with a cloud storage resource, usable by the query, and removing the first processing unit from the data warehouse.

Abstract: Data in a micro-partition of a table is stored in a compressed form. In response to a database query on the table comprising a filter, the portion of the data on which the filter operates is decompressed, without decompressing other portions of the data. Using the filter on the decompressed portion of the data, the portions of the data that are responsive to the filter are determined and decompressed. The responsive data is returned in response to the database query. When a query is run on a table that is compressed using dictionary compression, the uncompressed data may be returned along with the dictionary look-up values. The recipient of the data may use the dictionary look-up values for memoization, reducing the amount of computation required to process the returned data.

Abstract: Provided herein are systems and methods for handling build-side skew. For example, a method includes computing a plurality of hash values for a join operation. The join operation uses a corresponding plurality of row sets. The plurality of hash values are sampled to detect a frequent hash value. A build-side row set is partitioned using the frequent hash value to generate a partitioned build-side row set. The build-side row set is selected from the plurality of row sets. The partitioned build-side row set is distributed to a plurality of hash-join-build (HJB) instances executing at a corresponding plurality of servers.