Abstract: Resource provisioning systems and methods are described. In an embodiment, a system includes a plurality of shared storage devices collectively storing database data, an execution platform, and a compute service manager. The compute service manager is configured to determine a task to be executed in response to a trigger event and determine a query plan for executing the task, wherein the query plan comprises a plurality of discrete subtasks. The compute service manager is further configured to assign the plurality of discrete subtasks to one or more nodes of a plurality of nodes of the execution platform, determine whether execution of the task is complete, and in response to determining the execution of the task is complete, store a record in the plurality of shared storage devices indicating the task was completed.

Abstract: Embodiments of the present disclosure may provide a dynamic query execution model with fault tolerance and failure recovery techniques. Embodiments of the present disclosure may utilize checkpoints to map processed output files to their corresponding input files. Therefore, if an error occurs in processing one or more files, the system may only need to reschedule processing of selected file(s).

Abstract: Embodiments of the present disclosure may provide a dynamic query execution model. This query execution model may provide acceleration by scaling out parallel parts of a query (also referred to as a fragment) to additional computing resources, for example computing resources leased from a pool of computing resources. Execution of the parts of the query may be coordinated by a parent query coordinator, where the query originated, and a fragment query coordinator.

Abstract: Embodiments of the present disclosure may provide dynamic and fair assignment techniques for allocating resources on a demand basis. Assignment control may be separated into at least two components: a local component and a global component. Each component may have an active dialog with each other; the dialog may include two aspects: 1) a demand for computing resources, and 2) a total allowed number of computing resources. The global component may allocate resources from a pool of resources to different local components, and the local components in turn may assign their allocated resources to local competing requests. The allocation may also be throttled or limited at various levels.

Abstract: The subject technology determines whether a table is sufficiently clustered. The subject technology in response to determining the table is not sufficiently clustered, selects one or more micro-partitions of the table to be reclustered. The subject technology constructs a data structure for the table. The subject technology extracts minimum and maximum endpoints for each micro-partition in the data structure. The subject technology sorts each of one or more peaks in the data structure based on height. The subject technology sorts overlapping micro-partitions based on width. The subject technology selects based on which micro-partitions are within the tallest peaks of the one or more peaks and further based on which of the overlapping micro-partitions have the widest widths.

Abstract: Embodiments of the present disclosure may provide a dynamic query execution model with fault tolerance and failure recovery techniques. Embodiments of the present disclosure may utilize checkpoints to map processed output files to their corresponding input files. Therefore, if an error occurs in processing one or more files, the system may only need to reschedule processing of selected file(s).

Abstract: Resource provisioning systems and methods are described. In an embodiment, a system includes a plurality of shared storage devices collectively storing database data, an execution platform, and a compute service manager. The compute service manager is configured to determine a task to be executed in response to a trigger event and determine a query plan for executing the task, wherein the query plan comprises a plurality of discrete subtasks. The compute service manager is further configured to assign the plurality of discrete subtasks to one or more nodes of a plurality of nodes of the execution platform, determine whether execution of the task is complete, and in response to determining the execution of the task is complete, store a record in the plurality of shared storage devices indicating the task was completed.

Abstract: Resource provisioning systems and methods are described. In an embodiment, a system includes a plurality of shared storage devices collectively storing database data, an execution platform, and a compute service manager. The compute service manager is configured to determine a task to be executed in response to a trigger event and determine a query plan for executing the task, wherein the query plan comprises a plurality of discrete subtasks. The compute service manager is further configured to assign the plurality of discrete subtasks to one or more nodes of a plurality of nodes of the execution platform, determine whether execution of the task is complete, and in response to determining the execution of the task is complete, store a record in the plurality of shared storage devices indicating the task was completed.

Abstract: Systems, methods, and devices for generating and updating cross-account materialized views in multiple tenant database systems. A method includes defining a share object in a first account wherein the share object includes data associated with the first account. The method includes granting cross-account access rights to the share object to a second account such that the second account has access to the share object without copying the share object. The method includes generating a materialized view over the share object. The method includes updating the data associated with the first account. The method includes identifying whether the materialized view is stale with respect to the share object by merging the materialized view and the share object.

Abstract: The subject technology defines a materialized view over a source table that is associated with a first account of a database. The subject technology defines cross-account access rights to the materialized view for a second account. The subject technology modifies the source table for the materialized view. The subject technology identifies that the materialized view is stale with respect to the source table by merging the materialized view and the source table. The subject technology causes the materialized view to be refreshed with respect to the source table.

Abstract: The subject technology generates a materialized view over a share object, the share object including data associated with a first account. The subject technology merges the materialized view and the share object. The subject technology determines that the materialized view is stale with respect to the share object based at least in part on the merging. The subject technology generates a refreshed materialized view with respect to the share object in response to the determining. The subject technology shares the refreshed materialized view to a second account in response to a request from the second account.

Abstract: Disclosed herein are systems and methods for incremental reclustering of database tables based on local maxima of partition overlap. In an embodiment, a database platform makes a determination, based on one or more incremental-reclustering criteria, to incrementally recluster a database table, which has a clustering key and which is stored across a plurality of partitions. In response to making the determination, the database platform selects a subset of the partitions, and at least incrementally reclusters the selected subset. The selecting of the subset includes identifying a local maximum of a quantity of overlapping partitions in the plurality of partitions with respect to a domain of the clustering key of the table, where the overlapping partitions overlap with respect to the clustering key.

Abstract: Systems, methods, and devices for incrementally refreshing a materialized view are disclosed. A method includes generating a materialized view based on a source table. The method includes merging the source table and the materialized view to generate a merged table to identify whether an update has been executed on the source table that is not reflected in the materialized view. The method includes, in response to detecting an update made to the source table that is not reflected in the materialized view, applying the update to the materialized view.

Abstract: Embodiments of the present disclosure may provide dynamic and fair assignment techniques for allocating resources on a demand basis. Assignment control may be separated into at least two components: a local component and a global component. Each component may have an active dialog with each other; the dialog may include two aspects: 1) a demand for computing resources, and 2) a total allowed number of computing resources. The global component may allocate resources from a pool of resources to different local components, and the local components in turn may assign their allocated resources to local competing requests. The allocation may also be throttled or limited at various levels.

Abstract: The subject technology determines whether a table is sufficiently clustered. The subject technology in response to determining the table is not sufficiently clustered, selects one or more micro-partitions of the table to be reclustered. The subject technology constructs a data structure for the table. The subject technology extracts minimum and maximum endpoints for each micro-partition in the data structure. The subject technology sorts each of one or more peaks in the data structure based on height. The subject technology sorts overlapping micro-partitions based on width. The subject technology selects based on which micro-partitions are within the tallest peaks of the one or more peaks and further based on which of the overlapping micro-partitions have the widest widths.

Abstract: Systems, methods, and devices for incrementally refreshing a materialized view are disclosed. A method includes generating a materialized view based on a source table. The method includes merging the source table and the materialized view to generate a merged table to identify whether an update has been executed on the source table that is not reflected in the materialized view. The method includes, in response to detecting an update made to the source table that is not reflected in the materialized view, applying the update to the materialized view.

Abstract: Using one or more processors, generating a materialized view over a share object, the share object, including data associated with a first account. Using the one or more processors, merging the materialized view and the share object. Using the one or more processors, updating the data associated with the first account subsequent to generating the materialized view. Using the one or more processors, generating a refreshed materialized view with respect to the share object in response to detecting the updated data. Using the one or more processors, sharing the refreshed materialized view to a second account in response to a request from the second account.

Abstract: Automatic clustering of a database table is disclosed. A method for automatic clustering of a database table includes receiving an indication that a data modification task has been executed on a table and determining whether the table is sufficiently clustered. The method includes, in response to determining the table is not sufficiently clustered, selecting one or more micro-partitions of the table to be reclustered. The method includes assigning each of the one or more micro-partitions to an execution node to be reclustered.

Abstract: The subject technology generates, using one or more processors, a materialized view over a share object, the share object including data associated with a first account. The subject technology merges, using the one or more processors, the materialized view and the share object. The subject technology updates, using the one or more processors, the data associated with the first account subsequent to generating the materialized view. The subject technology generates, using the one or more processors, a refreshed materialized view with respect to the share object in response to detecting the updated data. The subject technology shares, using the one or more processors, the refreshed materialized view to a second account in response to a request from the second account.

Abstract: The subject technology generates a materialized view over a share object, the share object including data associated with a first account. The subject technology merges the materialized view and the share object. The subject technology determines that the materialized view is stale with respect to the share object based at least in part on the merging. The subject technology generates a refreshed materialized view with respect to the share object in response to the determining. The subject technology shares the refreshed materialized view to a second account in response to a request from the second account.