Abstract: A shared database platform implements dynamic masking on data shared between users where specific data is masked, transformed, or otherwise modified based on preconfigured functions that are associated with user roles. The shared database platform can implement the masking at runtime dynamically in response to users requesting access to a database object that is associated with one or more masking policies.

Abstract: Techniques for join elimination are described herein. Join elimination can identify and eliminate unnecessary joins in a query plan node. For example, join elimination can involve a semantic query optimization technique, which removes reference to a table whose columns are only referenced in join predicates if the joins do not filter/expand rows in the result. Such joins can cause significant performance issues on larger datasets if not optimized.

Abstract: The logging techniques described herein can enable using logging tools without having to use different methods for sandbox implementations and push out the log data to storage without problems. The log data is treated as sensitive data and is protected according to the defined security policies. Further, the results may be compressed and encrypted.

Abstract: Embodiments of the present disclosure may provide dynamic and fair assignment techniques for allocating resources on a demand basis. Assignment control may be separated into at least two components: a local component and a global component. Each component may have an active dialog with each other; the dialog may include two aspects: 1) a demand for computing resources, and 2) a total allowed number of computing resources. The global component may allocate resources from a pool of resources to different local components, and the local components in turn may assign their allocated resources to local competing requests. The allocation may also be throttled or limited at various levels.

Abstract: A query directed to database data stored across a set of files is received. The query includes predicates applied against metadata of the set of files. One or more files are removed from the set of files whose metadata does not satisfy a predicate of the plurality of predicates. Rows of the one or more remaining files of the set of files are filtered based on the one or more predicates.

Abstract: Embodiments of the present disclosure may provide a dynamic query execution model with fault tolerance and failure recovery techniques. Embodiments of the present disclosure may utilize checkpoints to map processed output files to their corresponding input files. Therefore, if an error occurs in processing one or more files, the system may only need to reschedule processing of selected file(s).

Abstract: The cloud data platform receives a first query directed towards a shared dataset, the first query identifying a first operation. The platform accesses a first set of data from the shared dataset to perform the first operation, the first set of data including data accessed from a first table of the shared dataset. The cloud data platform determines that an aggregation constraint policy is attached to the first table, the aggregation constraint policy restricts output of data values stored in the first table and enforces the aggregation constraint policy on the first query based on a context of the first query. The cloud data platform generates an output to the first query based on the first set of data and the first operation, based on enforcing the aggregation constraint policy on the first query.

Abstract: A system and method for managing data storage and data access with querying data in a distributed system without buffering the results of intermediate operations in persistent storage. The method includes pushing, by a first operator, as a result of executing a query plan, an intermediate result to a plurality of secondary operators. Processing of the intermediate result by the plurality of secondary operators is delayed to coordinate generation of a plurality of second results based on beginning a final consumption operation of the plurality of second results.

Abstract: Embodiments of the present disclosure related to sharing applications within a data sharing platform. An example method includes replicating a database from a provider account of a data sharing platform to a consumer account of the data sharing platform. The method further includes executing an installation script within the consumer account to install an application in the consumer account of the data sharing platform responsive to the replicating. The method further includes creating, by a processing device, a set of database roles to manage execution of the application in the consumer account, wherein one or more of the set of database roles determine access privileges granted to the application for accessing the database inside the consumer account. The application is to perform a data processing service relative to a data asset in the database.

Abstract: A shared database platform implements dynamic masking on data shared between users where specific data is masked, transformed, or otherwise modified based on preconfigured functions that are associated with user roles. The shared database platform can implement the masking at runtime dynamically in response to users requesting access to a database object that is associated with one or more masking policies.

Abstract: Joining data using a disjunctive operator using a lookup table is described. An example computer-implemented method can include receiving a query with a set of conjunctive predicates and a set of disjunctive predicates. The method may also include generating a lookup table for each predicate in the sets of conjunctive predicates and disjunctive predicates. The method, for each row in a probe-side table, may also further include looking up a value associated with that row in each of the lookup tables and adding the row to a results set when there is a match. Additionally, the method may also include returning the results set.

Abstract: Example caching systems and methods are described. In one implementation, a method receives a set of queries to be processed by a set of virtual warehouses. The method distributes the set of queries to the set of virtual warehouses to be executed and creates, during the processing of the set of queries by the set of virtual warehouses, a new virtual warehouse, wherein cache resources associated with the new virtual warehouse are populated with data files associated with the set of queries at the time the virtual warehouse is created and the cache resources vary among the processors, wherein a first subset of the processors comprises minimal cache resources and a second subset of processors comprises cache resources providing faster input-output operations. The method redistributes the set of queries across the set of virtual warehouses.

Abstract: Example resource management systems and methods are described. In one implementation, a resource manager is to be coupled to one or more virtual data warehouse instances that each comprise a plurality of computing resources. A resource manager receives a query directed to database data. The resource manager determines a task associated with the query. The resource manager distributes the task to a first virtual data warehouse instance of the plurality of virtual data warehouse instances. The resource manager determines that the first virtual data warehouse instance of the plurality of virtual data warehouse instances no longer accesses the database data. The resource manager reconfigures the first virtual data warehouse instance to delete a communication between the first virtual data warehouse instance and the database.

Abstract: Interleaved infrastructure can interleave fragment execution during query compilation. For example, compilation can be paused, and a plan fragment is executed. The results of the plan fragment can be fed back into the compiler, which can then restart the compilation and modify the compilation based on the results of the plan fragment execution. The interleaved workflow provides a flexible, robust infrastructure, which can enable and improve a variety of database features and query optimizations.

Abstract: The subject technology provides embodiments for supporting a unified table which may be a managed table or an unmanaged table. Managed tables are those where the subject technology manages the metastore/catalog for the table, whereas unmanaged tables are tables where an external catalog controls the table and the subject technology integrates with that catalog to work with the table, but does not assume control of the table.

Abstract: Techniques for join elimination are described herein. Join elimination can identify and eliminate unnecessary joins in a query plan node. For example, join elimination can involve a semantic query optimization technique, which removes reference to a table whose columns are only referenced in join predicates if the joins do not filter/expand rows in the result. Such joins can cause significant performance issues on larger datasets if not optimized.

Abstract: A system and method of scaling capacity of data warehouses to user-defined levels. The method includes monitoring a processing of one or more queries by a data warehouse, the data warehouse including one or more processing resources and a storage platform. The method includes receiving a request to process database data stored on the storage platform. The method includes determining a performance of the processed request as observed by a user device. The method includes adjusting, by a processing device, a timing associated with the processing of the one or more queries based on the performance of the processed request.

Abstract: The subject technology receives a query plan, the query plan comprising a set of query operations, the set of query operations including at least one aggregation and at least one join operation. The subject technology analyzes the query plan to identify an aggregation that is redundant. The subject technology removes the aggregation based at least in part on the analyzing. The subject technology determines at least one aggregation property corresponding to at least one query operation of the query plan. The subject technology inserts at least one adaptive aggregation operator in the query plan based at least in part on the at least one aggregation property, the at least one aggregation property comprising a set of aggregation properties. The subject technology provides a modified query plan based at least in part on the inserted at least one adaptive aggregation operator in the query plan.

Abstract: A method implementing a fault-tolerant data warehouse including allocating a plurality of processing units to a data warehouse, the processing units located in different availability zones, an availability zone comprising one or more data centers. The method further includes, as a result of monitoring a number of queries running at an input degree of parallelism on the plurality of processing units of the data warehouse, determining that the number of queries is serviceable by one fewer processing units. The method further includes routing a query from a first processing unit to a second processing unit within the data warehouse, the query having a common session identifier with another query previously provided to the second processing unit, the second processing unit determined to be caching a data segment associated with a cloud storage resource, usable by the query, and removing the first processing unit from the data warehouse.

Abstract: The subject technology receives, at a first execution node, a first transaction, the first transaction to be executed on linearizable storage. The subject technology determines whether the first execution node corresponds to a rank indicating a leader worker. The subject technology, in response to the first execution node corresponding to the rank indicating the leader worker, performs, by the first execution node, an initialization process for executing the first transaction. The subject technology broadcasts a first read timestamp associated with the first transaction to a set of execution nodes, the set of execution nodes being different than the first execution node. The subject technology executes, by the first execution node, at least a first operation from the first transaction.