Abstract: Techniques for continuous ingestion of files using custom file formats are described. A custom file format may include formats not natively supported by a data system. Unstructured files (e.g., images) may also be considered custom file formats. A custom file format may be set using a user defined table function and scanner options.

Abstract: Techniques for sharing query results in a multi-tenant database system are described. The query results can be shared between users of the same account or organization in the multi-tenant network-based database system using security tokens. A first user executes a query, and the results are stored in the network-based database system. The first user can invoke a function to create a security token to provide access to the stored query results to other users in the same account. The first user can share the security token with the other users, who can directly access the stored results in the network-based database system instead of having to download local copies of the query results.

Abstract: A database platform receives an object identifier from a client in association with a database session. The client is associated with a customer account of the database platform, and the database session is associated with the client. The customer account includes multiple disjoint account-level namespaces, each of which represents a distinct context for resolution of object identifiers, such that matching object identifiers in different account-level namespaces in the customer account do not collide with respect to object-identifier resolution. The database platform determines that the object identifier does not specify an account-level namespace, and responsively resolves the object identifier with reference to a current account-level namespace of the database session by identifying an object corresponding to the object identifier in the customer account.

Abstract: The subject technology receives a query plan, the query plan comprising a set of query operations, the set of query operations including at least one aggregation and at least one join operation. The subject technology analyzes the query plan to identify an aggregation that is redundant. The subject technology removes the aggregation based at least in part on the analyzing. The subject technology determines at least one aggregation property corresponding to at least one query operation of the query plan. The subject technology inserts at least one adaptive aggregation operator in the query plan based at least in part on the at least one aggregation property, the at least one aggregation property comprising a set of aggregation properties. The subject technology provides a modified query plan based at least in part on the inserted at least one adaptive aggregation operator in the query plan.

Abstract: Example caching systems and methods are described. In one implementation, a method identifies multiple files used to process a query and distributes each of the multiple files to a particular execution node to execute the query. Each execution node determines whether the distributed file is stored in the execution node's cache. If the execution node determines that the file is stored in the cache, it processes the query using the cached file. If the file is not stored in the cache, the execution node retrieves the file from a remote storage device, stores the file in the execution node's cache, and processes the query using the file.

Abstract: Disclosed herein are systems and methods for efficiently querying external tables. In an embodiment, a database platform receives a query that is directed at least in part to external data in an external table stored on a data storage platform that is external to the database platform. The external table includes a plurality of partitions. The database platform identifies, from external-table metadata, a subset of the plurality of partitions of the external table as including data that potentially satisfies the query. The external-table metadata is stored by the database platform. The database platform identifies data that satisfies the query by scanning the identified subset of the partitions, and responds to the query at least in part with the identified data that satisfies the query.

Abstract: A method, apparatus, and system for join operations of a plurality of relations that are distributed over a plurality of storage locations over a network of computing components.

Abstract: Systems and methods for generating transient object references are provided. The systems and methods perform operations including establishing a session between a first entity and a second entity. The operations include identifying an object that the first entity is authorized to access according to a first set of access privileges. The operations include generating a reference associated with the object. The operations include temporarily authorizing the second entity to access the object using the reference according to a second set of access privileges, the second set of access privileges being derived from the first set of access privileges.

Abstract: A constraint system enforces projection constraints on data values stored in specified columns of a shared dataset when queries are received by a database system. A projection constraint identifies that the data in a column may be restricted from being projected (e.g., presented, read, outputted) in an output to a received query, while allowing specified operations to be performed on the data and a corresponding output to be provided. For example, the projection constraint may indicate a context for a query that triggers the constraint, such as based on the user that submitted the query. Enforcing projection constraints on queries received at the database system allows for data to be shared and used anonymously by entities to perform various operations without the need to tokenize the data.

Abstract: Embodiments of the present disclosure may provide dynamic and fair assignment techniques for allocating resources on a demand basis. Assignment control may be separated into at least two components: a local component and a global component. Each component may have an active dialog with each other; the dialog may include two aspects: 1) a demand for computing resources, and 2) a total allowed number of computing resources. The global component may allocate resources from a pool of resources to different local components, and the local components in turn may assign their allocated resources to local competing requests. The allocation may also be throttled or limited at various levels.

Abstract: The subject technology receives, at a first execution node, a first transaction, the first transaction to be executed on linearizable storage. The subject technology determines whether the first execution node corresponds to a rank indicating a leader worker. The subject technology, in response to the first execution node corresponding to the rank indicating the leader worker, performs, by the first execution node, an initialization process for executing the first transaction. The subject technology broadcasts a first read timestamp associated with the first transaction to a set of execution nodes, the set of execution nodes being different than the first execution node. The subject technology executes, by the first execution node, at least a first operation from the first transaction.

Abstract: The subject technology receives, at a first execution node, a first transaction, the first transaction to be executed on linearizable storage. The subject technology determines whether the first execution node corresponds to a rank indicating a leader worker. The subject technology, in response to the first execution node corresponding to the rank indicating the leader worker, performs, by the first execution node, an initialization process for executing the first transaction. The subject technology broadcasts a first read timestamp associated with the first transaction to a set of execution nodes, the set of execution nodes being different than the first execution node. The subject technology executes, by the first execution node, at least a first operation from the first transaction.

Abstract: Systems, methods, and devices for sharing materialized views in multiple tenant database systems. A method includes defining a materialized view over a source table that is associated with a first account of a multiple tenant database. The method includes defining cross-account access rights to the materialized view to a second account such that that second account can read the materialized view without copying the materialized view. The method includes modifying the source table for the materialized view. The method includes identifying whether the materialized view is stale with respect to the source table by merging the materialized view and the source table.

Abstract: Resource provisioning systems and methods are described. In an embodiment, a system includes a plurality of shared storage devices collectively storing database data, an execution platform, and a compute service manager. The compute service manager is configured to determine a task to be executed in response to a trigger event and determine a query plan for executing the task, wherein the query plan comprises a plurality of discrete subtasks. The compute service manager is further configured to assign the plurality of discrete subtasks to one or more nodes of a plurality of nodes of the execution platform, determine whether execution of the task is complete, and in response to determining the execution of the task is complete, store a record in the plurality of shared storage devices indicating the task was completed.

Abstract: The subject technology receives a command to commit a table in a different table format on an external volume. The subject technology generates a first snapshot of the table on internal storage. The subject technology generates a first list of metadata files on the internal storage. The subject technology generates, based on the first list of metadata files, a first set of metadata files on the internal storage. The subject technology generates a second snapshot of the table on the external volume. The subject technology generates a second list of metadata files on the external volume. The subject technology generates, based on the second list of metadata files, a second set of metadata files on the external volume. The subject technology generates a first set of data files in a different file format on the external volume.

Abstract: A source table organized into a set of batch units is accessed. The source table comprises a column of data corresponding to a semi-structured data type. One or more indexing transformations for an object in the column are generated. The generating of the one or more indexing transformation includes converting the object to one or more stored data types. A pruning index is generated for the source table based in part on the one or more indexing transformations. The pruning index comprises a set of filters that index distinct values in each column of the source table, and each filter corresponds to a batch unit in the set of batch units. The pruning index is stored in a database with an association with the source table.

Abstract: Embodiments of the present disclosure may provide a dynamic query execution model. This query execution model may provide acceleration by scaling out parallel parts of a query (also referred to as a fragment) to additional computing resources, for example computing resources leased from a pool of computing resources. Execution of the parts of the query may be coordinated by a parent query coordinator, where the query originated, and a fragment query coordinator.

Abstract: A data platform that implements memoizable functions for database objects. The data platform detects a first execution of a memoizable function and generates a first key based on metadata of one or more database objects operated on by the memoizable function and generates a first result for the memoizable function based on the one or more database objects. The data platform detects a second execution of the memoizable function and generates a second key based on the metadata of the one or more database objects operated on by the memoizable function. When the first key and the second key are equal, the data platform reuses the first result of the memoizable function. When the first key and second key do not match, the data platform generates a second result for the second execution of the memoizable function.

Abstract: Example caching systems and methods are described. In one implementation, a method receives a query, at an execution platform, directed to data stored across a plurality of shared storage devices, the execution platform comprising one or more execution nodes, an execution node comprising a plurality of processors. The method processes the query using the one or more execution nodes of the execution platform, and in response to a determination of a change in a number of execution nodes of the execution platform, wherein the change is creating a new execution node, wherein a first subset of the plurality of processors comprises a minimal cache and a second subset of the plurality of processors comprises a cache providing faster input-output operations, reassigns processing of the query, among the changed number of execution nodes of the execution platform.

Abstract: Systems, methods, and devices for incrementally refreshing a materialized view are disclosed. A method includes generating a materialized view based on a source table. The method includes merging the source table and the materialized view to generate a merged table to identify whether an update has been executed on the source table that is not reflected in the materialized view. The method includes, in response to detecting an update made to the source table that is not reflected in the materialized view, applying the update to the materialized view.