Abstract: A database platform receives an object identifier from a client in association with a database session. The client is associated with a customer account of the database platform, and the database session is associated with the client. The customer account includes multiple disjoint account-level namespaces, each of which represents a distinct context for resolution of object identifiers, such that matching object identifiers in different account-level namespaces in the customer account do not collide with respect to object-identifier resolution. The database platform determines that the object identifier does not specify an account-level namespace, and responsively resolves the object identifier with reference to a current account-level namespace of the database session by identifying an object corresponding to the object identifier in the customer account.

Abstract: Techniques for triggering pipeline execution based on data change (transaction commit) are described. The pipelines can be used for data ingestion or other specified tasks. These tasks can be operational across account, organization, cloud region, and cloud provider boundaries. The tasks can be triggered by commit post-processing. Gates in the tasks can be set up to reference change data capture information. If the gate is satisfied, tasks can be executed to set up data pipelines.

Abstract: A streaming ingest platform can improve latency and expense issues related to uploading data into a cloud data system. The streaming ingest platform can organize the data to be ingested into per-table chunks and per-account blobs. This data may be committed and may be made available for query processing before it is ingested into the target source tables. This significantly improves latency issues. The streaming ingest platform can also accommodate uploading data from various sources with different processing and communication capabilities, such as Internet of Things (IOT) devices.

Abstract: Disclosed herein are systems and methods for efficiently querying external tables. In an embodiment, a database platform receives a query that is directed at least in part to external data in an external table stored on a data storage platform that is external to the database platform. The external table includes a plurality of partitions. The database platform identifies, from external-table metadata, a subset of the plurality of partitions of the external table as including data that potentially satisfies the query. The external-table metadata is stored by the database platform. The database platform identifies data that satisfies the query by scanning the identified subset of the partitions, and responds to the query at least in part with the identified data that satisfies the query.

Abstract: A method includes detecting, by at least one hardware processor, a change request for a table of a database, the table comprising a plurality of micro-partitions. A transaction associated with the change request is executed at a first timestamp. The transaction causes replacement of a first micro-partition of the plurality of micro-partitions with a second micro-partition. A change tracking column is generated in the second micro-partition. The change tracking column comprises metadata for the transaction. A delta for the table between the first timestamp and a second timestamp is generated using the metadata in the change tracking column. The delta indicates changes made to one or more rows of the table between the first time stamp and the second timestamp.

Abstract: A system includes one or more processors and data storage containing instructions executable by the one or more processors to perform operations. The operations include detecting a first executed transaction causing a first modification to table data stored in a table. The table data is associated with a corresponding metadata file with metadata information of the table. A new metadata file is generated responsive to the first executed transaction. The new metadata file includes the metadata information and additional metadata associated with the first modification. A second executed transaction causing a second modification to the table data is detected. The table data is updated with a change tracking column. The change tracking column includes lineage of executed transactions on the table data. The lineage indicates at least the first transaction and the second transaction.

Abstract: The subject technology receives a command to commit a table in a different table format on an external volume. The subject technology generates a first snapshot of the table on internal storage. The subject technology generates a first list of metadata files on the internal storage. The subject technology generates, based on the first list of metadata files, a first set of metadata files on the internal storage. The subject technology generates a second snapshot of the table on the external volume. The subject technology generates a second list of metadata files on the external volume. The subject technology generates, based on the second list of metadata files, a second set of metadata files on the external volume. The subject technology generates a first set of data files in a different file format on the external volume.

Abstract: A streaming ingest platform can improve latency and expense issues related to uploading data into a cloud data system. The streaming ingest platform can organize the data to be ingested into per-table chunks and per-account blobs. This data may be committed and may be made available for query processing before it is ingested into the target source tables. This significantly improves latency issues. The streaming ingest platform can also accommodate uploading data from various sources with different processing and communication capabilities, such as Internet of Things (IOT) devices.

Abstract: A data platform for running a subset of cells in a notebook is provided. The data platform receives a run cells message from a notebook user interface (UI) application specifying the subset of cells to run. For each cell in the subset, the data platform runs the cell to generate a set of results, generates a cell execution stream using the results, stores the stream, and transmits the stream to the notebook UI application. The notebook UI application generates a display for the user using the cell execution stream. The data platform provides an efficient way to run specific cells in a notebook and display the results to the user.

Abstract: A method includes partitioning a database table into a first plurality of partitions. A plurality of changes are executed on the database table in response to a transaction. The changes occur at a corresponding plurality of timestamps and result in a second plurality of partitions. The database table is updated to include a log with the plurality of changes. For each change of the plurality of changes, the log includes and an identification of a portion of the database table the change is applied to. A selection of a first timestamp and a second timestamp of the plurality of timestamps is detected. A delta is generated in response to the selection. The delta indicates a total change occurring to the database table between a first change of the plurality of changes corresponding to the first timestamp and a second change of the plurality of changes corresponding to the second timestamp.

Abstract: An external function system can be implemented on a database to perform processing on one or more external network services. The external function system can comprise a particular external function for a particular external service, an outbound serializer function, and an inbound serializer function that are linked with the particular external function. The outbound serializer function can be configured to transform the data of a query from a database format to a different format of the particular external network service. The inbound deserializer function can be configured to receive data returned from the external service and transform the data to the format of the database.

Abstract: Provided herein are systems and methods for a stream object configuration, including configuring and processing streams on views and streams on tables. For example, a method includes parsing, by at least one hardware processor, a received query to determine a plurality of data processing operations associated with a base table. A stream object on the base table is generated. The stream object includes a plurality of query results from executing the plurality of data processing operations and a corresponding plurality of time stamps of times the plurality of query results are obtained. A stream expansion of the stream object is performed. The stream expansion is based on computing changes to the base table from the plurality of data processing operations.

Abstract: Provided herein are systems and methods for a stream object configuration, including query processing of stream objects using stream expansion. For example, a method includes decoding a query to obtain a first data processing operation and a first stream object. The first stream object is associated with a view on a base table. A first stream expansion on the first stream object is performed. The first stream expansion is based on generating a second stream object on the base table. A second stream expansion of the second stream object is performed. The second stream expansion is based on replacing the second stream object with at least a second data processing operation. The query is executed based on completing the first data processing operation and the at least a second data processing operation.

Abstract: The subject technology receives a command to commit a table in a different table format on an external volume. The subject technology generates a first snapshot of the table on internal storage. The subject technology generates a first list of metadata files on the internal storage. The subject technology generates, based on the first list of metadata files, a first set of metadata files on the internal storage. The subject technology generates a second snapshot of the table on the external volume. The subject technology generates a second list of metadata files on the external volume. The subject technology generates, based on the second list of metadata files, a second set of metadata files on the external volume. The subject technology generates a first set of data files in a different file format on the external volume.

Abstract: In one aspect, a computer-implemented method includes detecting a committed version of recurrently executed tasks of a first data pipeline on a primary deployment that is hosted on a first cloud service, and replicating the committed version of the recurrently executed tasks to a second data pipeline on a secondary deployment that is hosted on a second cloud service.

Abstract: Techniques for schema mismatch detection and evolution are described. When data is being uploaded into a source table, schema of the data to be uploaded can be compared with the schema for the source table. If a schema mismatch is detected, the schema of the source table can be modified, and the upload can be continued without data loss.

Abstract: In an embodiment, a database platform receives a query that is directed at least in part to external data in an external table stored on a data storage platform that is external to the database platform. The external table includes a plurality of partitions. The database platform identifies, from external-table metadata, a subset of the plurality of partitions of the external table as including data that potentially satisfies the query. The external-table metadata is stored by the database platform. The database platform identifies data that satisfies the query by scanning the identified subset of the partitions, and responds to the query at least in part with the identified data that satisfies the query.

Abstract: A system or persistent table may be generated storing changelog information of a primary base table. The system table may then be used to create streams of relevant information. In some examples, the streams may read from the system table for information past a retention period of the primary table while reading from the primary table information in the retention period.

Abstract: An external function system can be implemented on a database to perform processing on one or more external network services. The external function system can comprise a particular external function for a particular external service, an outbound serializer function, and an inbound serializer function that are linked with the particular external function. The outbound serializer function can be configured to transform the data of a query from a database format to a different format of the particular external network service. The inbound deserializer function can be configured to receive data returned from the external service and transform the data to the format of the database.

Abstract: Techniques described herein can enable stream replication. A first deployment can store a table including one or more streams. The techniques described herein can be used to replicate the table at a second deployment while replicating the one or more streams associated with the table. Select prior table versions and partitions in the table are copied to the second deployment to enable stream replication.