Abstract: Systems, methods, and devices for automated maintenance of external tables in database systems are disclosed. A method includes receiving, by a database platform, read access to content in an external data storage platform that is separate from the database platform. The method includes defining an external table based on the content in the external data storage platform. The method includes connecting the database platform to the external table such that the database platform has read access for the external table and does not have write access for the external table. The method includes generating metadata for the external table, the metadata comprising information about data stored in the external table. The method includes receiving a notification that a modification has been made to the content in the external data storage platform, the modification comprising one or more of an addition of a file, a deletion of a file, or an update to a file in a source location for the external table.

Abstract: Disclosed herein are systems and methods for processing queries over external tables. In an embodiment, a database platform receives a query directed at least to data in an external table stored in a storage platform that is external to the database platform. The database platform uses metadata that summarizes the data in the external table to identify one or more partitions of the external table as potentially including data satisfying the query, and generates a query plan that includes a plurality of discrete subtasks that collectively include instructions to scan the identified one or more partitions of the external table for data satisfying the query. The database platform assigns, based on the metadata, the plurality of discrete subtasks to one or more nodes in an execution platform, and refreshes the metadata in response to a threshold number of modifications being made to the external table.

Abstract: A database platform receives an object identifier from a client in association with a database session. The client is associated with a customer account of the database platform, and the database session is associated with the client. In response to receiving the object identifier, the database platform identifies a resolution namespace for the object identifier, where the resolution namespace for the object identifier is a namespace that is specified in the object identifier if the object identifier includes a specified namespace, and where the resolution namespace is otherwise a current account-level namespace of the database session. The database platform resolves the object identifier with reference to the identified resolution namespace for the object identifier, including identifying an object corresponding to the object identifier in the customer account.

Abstract: Disclosed herein are systems and methods for processing queries over external tables. In an embodiment, a database platform receives a query directed at least to data in an external table stored in a storage platform that is external to the database platform. The database platform uses metadata that summarizes the data in the external table to identify one or more partitions of the external table as potentially including data satisfying the query, and generates a query plan that includes a plurality of discrete subtasks that collectively include instructions to scan the identified one or more partitions of the external table for data satisfying the query. The database platform assigns, based on the metadata, the plurality of discrete subtasks to one or more nodes in an execution platform, and refreshes the metadata in response to a threshold number of modifications being made to the external table.

Abstract: Systems, methods, and devices for transactional processing of change tracking data for a database are discussed. A method includes generating a micro-partition based on execution of a transaction on a table of a database, the micro-partition reflecting changes made to the table by the transaction. A change tracking entry is generated in response to the execution of the transaction. The change tracking entry includes an indication of one or more modifications made to the table by the transaction and an indication of the micro-partition generated based on the execution of the transaction. The change tracking entry is stored in the micro-partition as metadata. At least one existing micro-partition is removed from the table, responsive to storing the change tracking entry.

Abstract: The subject technology receives a notification that a modification has been made to an external table, the modification comprising inserting at least one row of new data to the external table, the at least one row corresponding to a first micro-partition that includes a first portion of data from the external table prior to the inserting. The subject technology, in response to the notification indicating the modification to the external table, generates a new micro-partition different from the first micro-partition, the new micro-partition including the inserted at least one row of new data and the first portion of data from the external table. The subject technology generates a refreshed materialized view based at least in part on the generated new micro-partition such that the refreshed materialized view comprises a representation of the external table after the modification has been made.

Abstract: Systems, methods, and devices for automated maintenance of external tables in database systems are disclosed. A method includes receiving, by a database platform, read access to content in an external data storage platform that is separate from the database platform. The method includes defining an external table based on the content in the external data storage platform. The method includes connecting the database platform to the external table such that the database platform has read access for the external table and does not have write access for the external table. The method includes generating metadata for the external table, the metadata comprising information about data stored in the external table. The method includes receiving a notification that a modification has been made to the content in the external data storage platform, the modification comprising one or more of an addition of a file, a deletion of a file, or an update to a file in a source location for the external table.

Abstract: A method of implementing sub-table replication starts with the processor detecting an update to an entitlements table. The processor performs filtering of a data table based on the update to the entitlements table. The data table including an entitlements column. The processor detects an update to the entitlements column and performs incremental replication of the data table by causing a version-based replication to be executed. Other embodiments are also described herein.

Abstract: A method of implementing sub-table replication starts with the processor detecting a first update to an entitlements table. The processor filters of a data table based on the first update. The processor then detects a second update to the entitlements table and performs incremental replication of the data table by causing a full inventory replication at a next refresh to be executed for provider accounts associated with the provider account identifiers in the entitlements table. Other embodiments are also described herein.

Abstract: A database platform authenticates a system user for access via an application to a database that is associated with a customer account of the database platform. The system user is a first object in a first account-level namespace of the customer account, and the first account-level namespace is distinct from a default account-level namespace of the customer account. The database platform sends, as the system user, a query to the database via the application. The database platform receives, as the system user, results of the query from the database, and stores, as the system user, the results of the query in a first-namespace stage, which is a second object in the first account-level namespace.

Abstract: A database platform receives an object identifier from a client in association with a database session. The client is associated with a customer account of the database platform, and the database session is associated with the client. In response to receiving the object identifier, the database platform identifies a resolution namespace for the object identifier, where the resolution namespace for the object identifier is a namespace that is specified in the object identifier if the object identifier includes a specified namespace, and where the resolution namespace is otherwise a current account-level namespace of the database session. The database platform resolves the object identifier with reference to the identified resolution namespace for the object identifier, including identifying an object corresponding to the object identifier in the customer account.

Abstract: Systems, methods, and devices for generating a transactional change tracking summary for a database. A method includes executing a transaction on a table of a database, wherein the table includes a micro-partition and the transaction is executed on the micro-partition. The method includes, in response to the transaction being fully executed, generating a change tracking entry comprising an indication of one or modifications made to the table by the transaction. The method includes storing the change tracking entry in a change tracking stream, wherein the change tracking stream includes one or more sequential change tracking entries that each correspond to a different transaction that is fully executed on the table.

Abstract: Disclosed herein are systems and methods for database replication to a remote deployment with automated fulfillment. In an embodiment, a data platform detects receipt of a database-replication request associated with a data consumer, requesting availability of a local instance of a primary-deployment database in a remote-deployment account of a data provider at a remote deployment of the data platform. The data platform responsively performs remote-deployment-activation operations including provisioning the remote-deployment account with a set of one or more replication-preparation objects associated with the primary-deployment database, and also including invoking at least one refresh command with respect to the set of one or more replication-preparation objects, which makes available to the data consumer a local instance of the primary-deployment database in the remote-deployment account of the data provider at the remote deployment of the data platform.

Abstract: Disclosed herein are systems and methods for database replication to a remote deployment with automated fulfillment. In an embodiment, a data platform provisions a remote-deployment account of a data provider at a remote deployment of the data platform with a set of one or more replication-preparation objects associated with a primary-deployment database that is resident in a primary-deployment account of the data provider at a primary deployment of the data platform. After the provisioning, the data platform detects receipt of a database-replication request associated with a data consumer, requesting availability of a local instance of the primary-deployment database in the remote-deployment account of the data provider. The data platform responsively performs one or more remote-deployment-activation operations, which makes available to the data consumer a local instance of the primary-deployment database in the remote-deployment account of the data provider at the remote deployment of the data platform.

Abstract: Systems, methods, and devices for querying over an external table are disclosed. A method includes connecting a database platform to an external table such that the database platform has read access for the external table and does not have write access for the external table. The method includes receiving a query comprising a predicate, the query directed at least to data in the external table. The method includes determining, based on metadata, one or more partitions in the external table comprising data satisfying the predicate. The method includes pruning, based on the metadata, all partitions in the external table that do not comprise any data satisfying the predicate. The method includes generating a query plan comprising a plurality of discrete subtasks. The method includes assigning, based on the metadata, the plurality of discrete subtasks to one or more nodes in an execution platform.

Abstract: Systems, methods, and devices for querying over an external table are disclosed. A method includes connecting a database platform to an external table such that the database platform has read access for the external table and does not have write access for the external table. The method includes receiving a query comprising a predicate, the query directed at least to data in the external table. The method includes determining, based on metadata, one or more partitions in the external table comprising data satisfying the predicate. The method includes pruning, based on the metadata, all partitions in the external table that do not comprise any data satisfying the predicate. The method includes generating a query plan comprising a plurality of discrete subtasks. The method includes assigning, based on the metadata, the plurality of discrete subtasks to one or more nodes in an execution platform.

Abstract: Systems, methods, and devices for generating a transactional change tracking summary for a database. A method includes executing a transaction on a table of a database, wherein the table includes a micro-partition and the transaction is executed on the micro-partition. The method includes, in response to the transaction being fully executed, generating a change tracking entry comprising an indication of one or modifications made to the table by the transaction. The method includes storing the change tracking entry in a change tracking stream, wherein the change tracking stream includes one or more sequential change tracking entries that each correspond to a different transaction that is fully executed on the table.

Abstract: Embodiments of the present disclosure may provide cross cloud auto-ingestion techniques. A deployment may monitor multiple queues across different cloud providers and may classify the queues based on their cloud. provider type. The deployment may receive notifications from those queues regarding new data ready for ingestion. The deployment may maintain a pool of credentials and may assign appropriate credentials to each queue. The deployment may route the notifications to appropriate receivers based on cloud provider types. The receivers may then auto-ingest new data in the corresponding queue.

Abstract: Systems, methods, and devices for generating a materialized view over an external table. A method includes connecting a database platform to an external table such that the database platform has read access for the external table and does not have write access for the external table. The method includes generating, by the database platform, a materialized view over the external table. The method includes receiving a notification that a modification has been made to the external table, the modification comprising one or more of an addition of a file, a deletion of a file, or an update to a file in a source location for the external table. The method includes, in response to the external table being modified, refreshing the materialized view such that the materialized view comprises an accurate representation of the external table.

Abstract: A shared database platform implements dynamic masking on data shared between users where specific data is masked, transformed, or otherwise modified based on preconfigured functions that are associated with user roles. The shared database platform can implement the masking at runtime dynamically in response to users requesting access to a database object that is associated with one or more masking policies.