Abstract: Embodiments of the present disclosure enable users of a data sharing system to build native applications that can be shared with other users of the data sharing system. The native applications can be published and discovered in the data sharing system like any other data listing, and consumers can install them in their local data sharing system account to serve their data processing needs. A provider may define an installation script for installing an application and create a share object to which the installation script may be attached. In response to an imported database being created in a consumer account based on the share object, a native application framework may automatically execute the installation script in the consumer account and may create a set of database roles to manage execution of the application in the consumer account.

Abstract: Example resource management systems and methods are described. In one implementation, a resource manager is configured to manage data processing tasks associated with multiple data elements. An execution platform is coupled to the resource manager and includes multiple execution nodes configured to store data retrieved from multiple remote storage devices. Each execution node includes a cache and a processor, where the cache and processor are independent of the remote storage devices. A metadata manager is configured to access metadata associated with at least a portion of the multiple data elements.

Abstract: A source table organized into a set of micro-partitions is accessed by a network-based data warehouse. A pruning index is generated based on the source table. The pruning index comprises a set of filters that indicate locations of distinct values in each column of the source table. A query directed at the source table is received at the network-based data warehouse. The query is processed using the pruning index. The processing of the query comprises pruning the set of micro-partitions of the source table to scan for data matching the query, the pruning of the plurality of micro-partitions comprising identifying, using the pruning index, a sub-set of micro-partitions to scan for the data matching the query.

Abstract: A system and method for managing data storage and data access with querying data in a distributed system without buffering the results on intermediate operations in disk storage.

Abstract: Joining data using a disjunctive operator is described. An example computer-implemented method can include receiving a query that includes a first disjunctive predicate involving a first table and a second table. The method may also include determining a first set of rows from the first table and generating a filter from the first set of rows. The method may also further include applying the filter to the second table to generate a second set of rows. Additionally, the method may also include joining the first set of rows and the second set of rows using a first disjunctive operator of the first disjunctive predicate to generate a first results set.

Abstract: Example resource provisioning systems and methods are described. In one implementation, multiple processing resources are provided within a data warehouse. The processing resources include at least one processor and at least one storage device. At least one query to process database data is received. At least some of the processing resources may process the database data. When a processing capacity of the processing resources has reached a threshold processing capacity, the processing capacity is automatically scaled by adding at least one additional processor to the data warehouse.

Abstract: Disclosed herein are embodiments of systems and methods for partition-based scanning of external tables for query processing. In an example embodiment, a database platform receives a query that includes one or more predicates, where the query is directed at least to data in an external table that is stored in an external storage platform that is external to the database platform. The database platform identifies, based on metadata that summarizes the data in the external table, one or more partitions of the external table that potentially include data that satisfies the one or more predicates. The database platform also identifies, from the one or more identified partitions, data that satisfies the one or more predicates. The database platform sends a response to the query to the client, the response comprising the data satisfying the one or more predicates.

Abstract: Resource provisioning systems and methods are described. In an embodiment, a system includes a plurality of shared storage devices collectively storing database data, an execution platform, and a compute service manager. The compute service manager is configured to determine a task to be executed in response to a trigger event and determine a query plan for executing the task, wherein the query plan comprises a plurality of discrete subtasks. The compute service manager is further configured to assign the plurality of discrete subtasks to one or more nodes of a plurality of nodes of the execution platform, determine whether execution of the task is complete, and in response to determining the execution of the task is complete, store a record in the plurality of shared storage devices indicating the task was completed.

Abstract: Example resource management systems and methods are described. In one implementation, a processing device is to: provide a plurality of virtual processors, each virtual processor comprising a cache memory to cache a portion of database data accessed from a storage platform; receive a query referencing the database data stored in the storage platform; determine at least one task to process the received query; and allocate the at least one task to a subset of the plurality of virtual processors to process the at least one task. The allocation may be based at least in part on the respective portions of the database data that are cached within the respective cache memories of the subset of the plurality of virtual processors.

Abstract: The subject technology determines whether a table is sufficiently clustered. The subject technology in response to determining the table is not sufficiently clustered, selects one or more micro-partitions of the table to be reclustered. The subject technology constructs a data structure for the table. The subject technology extracts minimum and maximum endpoints for each micro-partition in the data structure. The subject technology sorts each of one or more peaks in the data structure based on height. The subject technology sorts overlapping micro-partitions based on width. The subject technology selects based on which micro-partitions are within the tallest peaks of the one or more peaks and further based on which of the overlapping micro-partitions have the widest widths.

Abstract: Disclosed herein are systems and methods for pruning external data. In an embodiment, a database platform receives a query directed at least in part to external data in an external table on an external data storage platform. The external table includes partitions that correspond to storage locations in a source directory of the external data storage platform. The storage locations contain files that contain the external data. The database platform identifies, from external-table metadata that is stored by the database platform and that maps the partitions of the external table to the storage locations in the source directory, a subset of the partitions as including data that potentially satisfies the query. The database platform identifies data that satisfies the query by scanning the identified subset of the partitions, and responds to the query at least in part with the identified data that satisfies the query.

Abstract: A system and method for managing data storage and data access with querying data in a distributed system without buffering the results on intermediate operations in disk storage.

Abstract: A method and apparatus managing a set of processors for a set of queries is described. In an exemplary embodiment, a device receives a set of queries for a data warehouse, the set of queries including one or more queries to be processed by the data warehouse. The device further provisions a set of processors from a first plurality of processors, where the set of processors to process the set of queries, and a set of storage resources to store data for the set of queries. In addition, the device monitors a utilization of the set of processors as the set of processors processes the set of queries. The device additionally updates a number of the processors in the set of processors provisioned based on the utilization/Furthermore, the device processes the set of queries using the updated set of processors.

Abstract: A method for sharing data in a multi-tenant database includes generating a share object in a first account comprising a share role. The method includes associating one or more access rights with the share role, wherein the one or more access rights indicate which objects in the first account are accessible based on the share object. The method includes granting, to a second account, cross-account access rights to the share role or share object in the first account. The method includes receiving a request from the second account to access data or services of the first account. The method further includes providing a response to the second account based on the data or services of the first account.

Abstract: Embodiments of the present disclosure may provide a dynamic query execution model. This query execution model may provide acceleration by scaling out parallel parts of a query (also referred to as a fragment) to additional computing resources, for example computing resources leased from a pool of computing resources. Execution of the parts of the query may be coordinated by a parent query coordinator, where the query originated, and a fragment query coordinator.

Abstract: Systems, methods, and devices for generating a secure join of database data are disclosed. A method includes hashing datapoints of a consumer account and creating a secure view of the hashed datapoints of the consumer account. The method further includes processing, using a secure user defined function (UDF), the hashed datapoints of the consumer account and datapoints of a provider account to generate a secure join key, wherein the hashed data points of the consumer account are provided to the secure UDF using the secure view.

Abstract: A method for sharing data in a multi-tenant database includes receiving, by a target account of a multiple tenant database, access rights of a share object in a first account of the multiple tenant database, wherein the share object having access rights to a database object of the first account and wherein access to the database object of the first account by the target account is based on the access rights of the share object. The method also includes receiving, by one or more processors of the target account, access rights to an alias object, wherein the alias object references the database object of the first account.

Abstract: A query directed at a source table organized into a set of batch units is received. The query includes a pattern matching predicate that specifies a search pattern. A set of N-grams are generated based on the search pattern. A pruning index associated with the source table is accessed. The pruning index comprises a set of filters that index distinct N-grams in each column of the source table. The pruning index is used to identify a subset of batch units to scan for matching data based on the set of N-grams generated for the search pattern. The query is processed by scanning the subset of batch units.

Abstract: Resource provisioning systems and methods are described. In an embodiment, a system includes a plurality of shared storage devices collectively storing database data, an execution platform, and a compute service manager. The compute service manager is configured to determine a task to be executed in response to a trigger event and determine a query plan for executing the task, wherein the query plan comprises a plurality of discrete subtasks. The compute service manager is further configured to assign the plurality of discrete subtasks to one or more nodes of a plurality of nodes of the execution platform, determine whether execution of the task is complete, and in response to determining the execution of the task is complete, store a record in the plurality of shared storage devices indicating the task was completed.

Abstract: A system and method for pruning data based on metadata. The method may include receiving a query that includes a plurality of predicates and identifying one or more applicable files including database data satisfying at least one of the plurality of predicates. The identifying the one or more applicable files including reading metadata stored in a metadata store that is separate from the database data. The method further includes pruning inapplicable files including database data that does not satisfy at least one of the plurality of predicates to create a reduced set of files and reading the reduced set of files to execute the query.