Abstract: A query referencing a function associated with a remote software component is received by a network-based data warehouse system. Temporary security credentials corresponding to a role at a cloud computing service platform are obtained. The role has permission to send calls to a web endpoint corresponding to the remote software component. A request comprising input data and electronically signed using the temporary security credentials is sent to a web Application Programming Interface (API) management system of the cloud computing service platform. The request, when received by the web API management system, causes the web API management system to invoke external functionality provided by the remote software component at the web endpoint with respect to the input data. A response comprising a result of invoking the external functionality is received from the web API management system, and the result data is processed according to the query.

Abstract: The subject technology determines a set of shards corresponding to an external data source accessible via a network, the external data source being hosted by an external system separate from a network-based database system. The subject technology determines, using a stream object, a set of offsets of each shard of the set of shards. The subject technology identifies an operation to perform on the set of shards, the operation comprising a read operation or a write operation. The subject technology, based on the set of shards and the set of offsets, performs the operation on the external data source.

Abstract: Systems, methods, and devices for querying over an external table are disclosed. A method includes connecting a database platform to an external table such that the database platform has read access for the external table and does not have write access for the external table. The method includes receiving a query comprising a predicate, the query directed at least to data in the external table. The method includes determining, based on metadata, one or more partitions in the external table comprising data satisfying the predicate. The method includes pruning, based on the metadata, all partitions in the external table that do not comprise any data satisfying the predicate. The method includes generating a query plan comprising a plurality of discrete subtasks. The method includes assigning, based on the metadata, the plurality of discrete subtasks to one or more nodes in an execution platform.

Abstract: The subject technology receives a task for executing in a network-based data warehouse system. The subject technology determines a number of execution nodes to execute the task. The subject technology, based on the number of execution nodes, selects a virtual warehouse that includes at least the number of execution nodes. The subject technology sends the task to the virtual warehouse for execution. The subject technology receives an indication that execution of the task is complete. Further, the subject technology provides an indication that the virtual warehouse is idle.

Abstract: The subject technology executes a change on an existing micro-partition of a table of a database, the executing of the change comprising generating a new micro-partition that embodies the change. The subject technology receives a request for a delta for the table between a first timestamp and a second timestamp. The subject technology queries at least one change tracking column to determine the delta between the first timestamp and the second timestamp, the delta including information indicating at least one database operation that was performed to at least one row of a set of rows of the table, without including information as to intermediate changes made to at least one row of the set of rows of the table between the first timestamp and the second timestamp, that facilitates a reduction in storage of historical versions of the set of rows of the table.

Abstract: Systems, methods, and devices for querying over an external table are disclosed. A method includes connecting a database platform to an external table such that the database platform has read access for the external table and does not have write access for the external table. The method includes receiving a query comprising a predicate, the query directed at least to data in the external table. The method includes determining, based on metadata, one or more partitions in the external table comprising data satisfying the predicate. The method includes pruning, based on the metadata, all partitions in the external table that do not comprise any data satisfying the predicate. The method includes generating a query plan comprising a plurality of discrete subtasks. The method includes assigning, based on the metadata, the plurality of discrete subtasks to one or more nodes in an execution platform.

Abstract: Systems, methods, and devices for generating a transactional change tracking summary for a database. A method includes executing a transaction on a table of a database, wherein the table includes a micro-partition and the transaction is executed on the micro-partition. The method includes, in response to the transaction being fully executed, generating a change tracking entry comprising an indication of one or modifications made to the table by the transaction. The method includes storing the change tracking entry in a change tracking stream, wherein the change tracking stream includes one or more sequential change tracking entries that each correspond to a different transaction that is fully executed on the table.

Abstract: The subject technology executes a change on an existing micro-partition of a table, the executing of the change comprising generating a new micro-partition that embodies the change and deleting the existing micro-partition. The subject technology receives a request for a delta for the table between a first timestamp and a second timestamp. The subject technology responsively determines the delta based on a table history, the delta including information indicating at least one operation that was performed to at least one row of a set of rows of the table, without including information as to intermediate changes made to at least one row of the set of rows of the table between the first timestamp and the second timestamp, that facilitates a reduction in storage of historical versions of the set of rows of the table.

Abstract: The subject technology receives one or more requests to execute one or more requested transactions on a journal table of a database, the journal table comprising a snapshot and a log table. The subject technology, based on the one or more requests, inserts a new row into the log table for each requested transaction, each new row reflecting a corresponding requested transaction. The subject technology receives, after the one or more requested transactions have been executed, a second request to execute a second requested transaction on the journal table. The subject technology generates, prior to executing the second requested transaction, a second snapshot, the second snapshot comprising a second representation of data in the journal table after the one or more requested transactions have been executed, the second snapshot stored in a third micro-partition different than the first micro-partition and the second micro-partition.

Abstract: Systems, methods, and devices for generating a transactional change tracking summary for a database. A method includes executing a transaction on a table of a database, wherein the table includes a micro-partition and the transaction is executed on the micro-partition. The method includes, in response to the transaction being fully executed, generating a change tracking entry comprising an indication of one or modifications made to the table by the transaction. The method includes storing the change tracking entry in a change tracking stream, wherein the change tracking stream includes one or more sequential change tracking entries that each correspond to a different transaction that is fully executed on the table.

Abstract: A query referencing a function associated with a remote software component is received by a network-based data warehouse system. Temporary security credentials corresponding to a role at a cloud computing service platform are obtained. The role has permission to send calls to a web endpoint corresponding to the remote software component. A request comprising input data and electronically signed using the temporary security credentials is sent to a web Application Programming Interface (API) management system of the cloud computing service platform. The request, when received by the web API management system, causes the web API management system to invoke external functionality provided by the remote software component at the web endpoint with respect to the input data. A response comprising a result of invoking the external functionality is received from the web API management system, and the result data is processed according to the query.

Abstract: Systems, methods, and devices for automated maintenance of external tables in database systems are disclosed. A method includes receiving, by a database platform, read access to content in an external data storage platform that is separate from the database platform. The method includes defining an external table based on the content in the external data storage platform. The method includes connecting the database platform to the external table such that the database platform has read access for the external table and does not have write access for the external table. The method includes generating metadata for the external table, the metadata comprising information about data stored in the external table. The method includes receiving a notification that a modification has been made to the content in the external data storage platform, the modification comprising one or more of an addition of a file, a deletion of a file, or an update to a file in a source location for the external table.

Abstract: Systems, methods, and devices for tracking a series of changes to database data are disclosed. A method includes executing a transaction to modify data in a micro-partition of a table of a database by generating a new micro-partition that embodies the transaction. The method includes associating transaction data with the new micro-partition, wherein the transaction data comprises a timestamp when the transaction was fully executed, and further includes associating modification data with the new micro-partition that comprises an indication of one or more rows of the table that were modified by the transaction. The method includes joining the transaction data with the modification data to generate joined data and querying the joined data to determine a listing of intermediate modifications made to the table between a first timestamp and a second timestamp.

Abstract: A query referencing a function associated with a remote software component is received by a network-based data warehouse system. Temporary security credentials corresponding to a role at a cloud computing service platform are obtained. The role has permission to send calls to a web endpoint corresponding to the remote software component. A request comprising input data and electronically signed using the temporary security credentials is sent to a web Application Programming Interface (API) management system of the cloud computing service platform. The request, when received by the web API management system, causes the web API management system to invoke external functionality provided by the remote software component at the web endpoint with respect to the input data. A response comprising a result of invoking the external functionality is received from the web API management system, and the result data is processed according to the query.

Abstract: The subject technology defines a journal table of a database, the journal table comprising a snapshot and a log table, the snapshot comprising a representation of data in the journal table at a particular time, the log table comprising a listing of requested changes to the journal table since the particular time, the snapshot stored in a first micro-partition, the log table stored in a second micro-partition. The subject technology receives, after at least one first requested transaction has been executed, a request to execute a second requested transaction on the journal table. The subject technology generates, prior to executing the second requested transaction, a second snapshot, the second snapshot comprising a second representation of data in the journal table after the at least one first requested transaction has been executed, the second snapshot stored in a third micro-partition different than the first micro-partition and the second micro-partition.

Abstract: The subject technology executes a change on an existing micro-partition of a table of a database, the executing of the change comprising generating a new micro-partition that embodies the change. The subject technology receives a request for a delta for the table between a first timestamp and a second timestamp. The subject technology queries at least one change tracking column to determine the delta between the first timestamp and the second timestamp, the delta including information indicating at least one database operation that was performed to at least one row of a set of rows of the table, without including information as to intermediate changes made to at least one row of the set of rows of the table between the first timestamp and the second timestamp, that facilitates a reduction in storage of historical versions of the set of rows of the table.

Abstract: Systems, methods, and devices for tracking a series of changes to database data are disclosed. A method includes executing a transaction to modify data in a micro-partition of a table of a database by generating a new micro-partition that embodies the transaction. The method includes associating transaction data with the new micro-partition, wherein the transaction data comprises a timestamp when the transaction was fully executed, and further includes associating modification data with the new micro-partition that comprises an indication of one or more rows of the table that were modified by the transaction. The method includes joining the transaction data with the modification data to generate joined data and querying the joined data to determine a listing of intermediate modifications made to the table between a first timestamp and a second timestamp.

Abstract: The subject technology receives one or more requests to execute one or more requested transactions on a journal table of a database, the journal table comprising a snapshot and a log table. The subject technology, based on the one or more requests, inserts a new row into the log table for each requested transaction, each new row reflecting a corresponding requested transaction. The subject technology receives, after the one or more requested transactions have been executed, a second request to execute a second requested transaction on the journal table. The subject technology generates, prior to executing the second requested transaction, a second snapshot, the second snapshot comprising a second representation of data in the journal table after the one or more requested transactions have been executed, the second snapshot stored in a third micro-partition different than the first micro-partition and the second micro-partition.

Abstract: The subject technology defines a journal table of a database, the journal table comprising a snapshot and a log table, the snapshot comprising a representation of data in the journal table at a particular time, the log table comprising a listing of requested changes to the journal table since the particular time, the snapshot stored in a first micro-partition, the log table stored in a second micro-partition. The subject technology receives, after at least one first requested transaction has been executed, a request to execute a second requested transaction on the journal table. The subject technology generates, prior to executing the second requested transaction, a second snapshot, the second snapshot comprising a second representation of data in the journal table after the at least one first requested transaction has been executed, the second snapshot stored in a third micro-partition different than the first micro-partition and the second micro-partition.

Abstract: Systems, methods, and devices for storing database data in journal tables comprising a snapshot and a log table. A method includes defining a journal table comprising a snapshot and a log table, the snapshot comprising an up-to-date representation of data in the journal table at a point in time. The method includes assigning a timestamp to the snapshot indicating when the snapshot was generated. The method includes receiving a request to execute a transaction on the journal table to modify the data in the journal table, the transaction comprising one or more of an insert, a delete, an update, or a merge. The method includes inserting a new row into the log table in lieu of executing the transaction on the snapshot of the journal table, the new row comprising an indication of a change requested to be made to the journal table based on the transaction.