Abstract: Techniques for triggering pipeline execution based on data change (transaction commit) are described. The pipelines can be used for data ingestion or other specified tasks. These tasks can be operational across account, organization, cloud region, and cloud provider boundaries. The tasks can be triggered by commit post-processing. Gates in the tasks can be set up to reference change data capture information. If the gate is satisfied, tasks can be executed to set up data pipelines.

Abstract: Systems, methods, and devices for generating a transactional change tracking summary for a database. A method includes executing a transaction on a table of a database, wherein the table includes a micro-partition and the transaction is executed on the micro-partition. The method includes, in response to the transaction being fully executed, generating a change tracking entry comprising an indication of one or modifications made to the table by the transaction. The method includes storing the change tracking entry in a change tracking stream, wherein the change tracking stream includes one or more sequential change tracking entries that each correspond to a different transaction that is fully executed on the table.

Abstract: Provided herein are systems and methods for a stream object configuration, including configuring and processing streams on views and streams on tables. For example, a method includes parsing, by at least one hardware processor, a received query to determine a plurality of data processing operations associated with a base table. A stream object on the base table is generated. The stream object includes a plurality of query results from executing the plurality of data processing operations and a corresponding plurality of time stamps of times the plurality of query results are obtained. A stream expansion of the stream object is performed. The stream expansion is based on computing changes to the base table from the plurality of data processing operations.

Abstract: A streaming ingest platform can improve latency and expense issues related to uploading data into a cloud data system. The streaming ingest platform can organize the data to be ingested into per-table chunks and per-account blobs. This data may be committed and may be made available for query processing before it is ingested into the target source tables. This significantly improves latency issues. The streaming ingest platform can also accommodate uploading data from various sources with different processing and communication capabilities, such as Internet of Things (IOT) devices.

Abstract: Embodiments of the present disclosure enable users of a data sharing system to build native applications that can be shared with other users of the data sharing system. The native applications can be published and discovered in the data sharing system like any other data listing, and consumers can install them in their local data sharing system account to serve their data processing needs. A provider may define an installation script for installing an application and create a share object to which the installation script may be attached. In response to an imported database being created in a consumer account based on the share object, a native application framework may automatically execute the installation script in the consumer account and may create a set of database roles to manage execution of the application in the consumer account.

Abstract: A database platform authenticates a system user for access via an application to a database that is associated with a customer account of the database platform. The system user is a first object in a first account-level namespace of the customer account, and the first account-level namespace is distinct from a default account-level namespace of the customer account. The database platform sends, as the system user, a query to the database via the application. The database platform receives, as the system user, results of the query from the database, and stores, as the system user, the results of the query in a first-namespace stage, which is a second object in the first account-level namespace.

Abstract: The subject technology determines, using a connection to an external data source, a set of shards stored in an external data source, the connection to the external data source being established using an external integration, the external integration including security and configuration information. The subject technology determines a set of offsets of each shard of the set of shards. The subject technology generates a query plan indicating a degree of parallelism based at least in part on a size of the set of offsets. The subject technology, based on the set of shards and the set of offsets, performs an operation on the external data source by performing, using the connection to the external data source, a write operation from a query statement on the external data source, the external data source being different than a storage platform associated with the system.

Abstract: Techniques described herein can accelerate change data capture determinations such as stream reads, which show changes made to a table between two points in time. Three distinct row bitests that mark deleted, updated, inserted, rows in micro-partitions can be added as metadata for the table. These bitsets can be generated during DML operations and then stored as metadata of the new partition generated by the DML operations. The bitsets can then be used to generate streams showing the changes in the table between two points in time (changes interval).

Abstract: Hybrid tables can be used in different use-case scenarios. Hybrid tables provide a flexible mechanism to support files and data in different formats while providing access to the different types of data as part of one table. This flexibility can allow the use of hybrid tables in data lake or other similar environments.

Abstract: The subject technology generates, by executing first code in a first programming language, a resilient distributed dataset (RDD). The subject technology generates, by executing the first code in the first programming language, a second RDD based on the generated RDD. The subject technology generates, by executing the first code in the first programming language, a directed acyclic graph (DAG) including the generated RDD and the generated second RDD. The subject technology serializes the DAG. The subject technology generates a database table based on the serialized DAG. The subject technology converts the first code to second code in a second programming language different than the first programming language, and at least one database query in a second format different than the first programming language. The subject technology sends the converted first code and the at least one database query to a database system for execution.

Abstract: A query referencing an external table function provided by a remote software component is received. Requests to execute the external table function on input data are sent to a proxy service. A first request includes a batch of input rows from the input data. A first response to the first request received from the proxy service includes a first portion of result data and a pagination token. The pagination token indicates that at least a second portion of the result data corresponding to the first batch of input rows is to be obtained from the remote software component. Based on the pagination token, a second request is sent to obtain the second portion of the result data. One or more responses are received from the proxy service that comprise at least the second portion of the result data. The result data is processed according to the query.

Abstract: The subject technology determines, using a connection to an external data source, a set of shards stored in an external data source, the connection to the external data source being established using an external integration, the external integration including security and configuration information. The subject technology determines a set of offsets of each shard of the set of shards. The subject technology generates a query plan indicating a degree of parallelism based at least in part on a size of the set of offsets. The subject technology, based on the set of shards and the set of offsets, performs an operation on the external data source by performing, using the connection to the external data source, a write operation from a query statement on the external data source, the external data source being different than a storage platform associated with the system.

Abstract: A streaming ingest platform can improve latency and expense issues related to uploading data into a cloud data system. The streaming ingest platform can organize the data to be ingested into per-table chunks and per-account blobs. This data may be committed and may be made available for query processing before it is ingested into the target source tables. This significantly improves latency issues. The streaming ingest platform can also accommodate uploading data from various sources with different processing and communication capabilities, such as Internet of Things (IOT) devices.

Abstract: A database platform can track changes using a stream that records changes made by database transactions. Streams on views can be implemented by applying a stream to source data, and applying the view to the streamed data. Users of the database platform can query the stream on the view to review recent changes to the view, such as changes made since the last database transaction.

Abstract: Provided herein are systems and methods for a stream object configuration, including configuring and processing streams on views. For example, a method includes configuring by at least one hardware processor at a first computing node, a view on a source table for shared access with at least a second computing node. The view includes a query associated with the source table. The query associated with the source table is executed to obtain a plurality of query results. A stream object is generated on the view. The stream object includes the plurality of query results of executing the query and a corresponding plurality of time stamps associated with times the plurality of query results are obtained. The generated stream object is configured for the shared access with at least the second computing node.

Abstract: Systems, methods, and devices for generating a materialized view over an external table. A method includes connecting a database platform to an external table such that the database platform has read access for the external table and does not have write access for the external table. The method includes generating, by the database platform, a materialized view over the external table. The method includes receiving a notification that a modification has been made to the external table, the modification comprising one or more of an addition of a file, a deletion of a file, or an update to a file in a source location for the external table. The method includes, in response to the external table being modified, refreshing the materialized view such that the materialized view comprises an accurate representation of the external table.

Abstract: Systems, methods, and devices for scheduling of data processing tasks are disclosed. A method includes performing a modification on a first set of immutable partitions storing database data to generate a second set of immutable partitions. The second set is associated with a modified version of the database data. A change tracking entry is entered in a stream of tracking entries based on committing the modification. The change tracking entry includes an indication of the modification on the first set of immutable partitions. A stream offset of the stream of tracking entries is advanced based on the entering of the change tracking entry in the stream of tracking entries. The stream offset indicates a timestamp associated with a latest committed modification to the database data. A data processing task is scheduled for execution on the modified version of the database data based on the advancing of the stream offset.

Abstract: Disclosed herein are systems and methods for efficiently querying external tables. In an embodiment, a database platform receives a query that is directed at least in part to external data in an external table stored on a data storage platform that is external to the database platform. The external table includes a plurality of partitions. The database platform identifies, from external-table metadata, a subset of the plurality of partitions of the external table as including data that potentially satisfies the query. The external-table metadata is stored by the database platform. The database platform identifies data that satisfies the query by scanning the identified subset of the partitions, and responds to the query at least in part with the identified data that satisfies the query.

Abstract: A query referencing an external table function provided by a remote software component is received. Requests to execute the external table function on input data are sent to a proxy service. A first request includes a batch of input rows from the input data. A first response to the first request received from the proxy service includes a first portion of result data and a pagination token. The pagination token indicates that at least a second portion of the result data corresponding to the first batch of input rows is to be obtained from the remote software component. Based on the pagination token, a second request is sent to obtain the second portion of the result data. One or more responses are received from the proxy service that comprise at least the second portion of the result data. The result data is processed according to the query.

Abstract: The subject technology receives first code in a first programming language, the first code including a reference to a first dataset. The subject technology converting the first code to: second code in a second programming language different than the first programming language, and at least one data query in a second format different than the first programming language. Further the subject technology sends the converted first code and the at least one data query to a data system for execution.