Abstract: Techniques for triggering pipeline execution based on data change (transaction commit) are described. The pipelines can be used for data ingestion or other specified tasks. These tasks can be operational across account, organization, cloud region, and cloud provider boundaries. The tasks can be triggered by commit post-processing. Gates in the tasks can be set up to reference change data capture information. If the gate is satisfied, tasks can be executed to set up data pipelines.

Abstract: In one aspect, a computer-implemented method includes detecting a committed version of recurrently executed tasks of a first data pipeline on a primary deployment that is hosted on a first cloud service, and replicating the committed version of the recurrently executed tasks to a second data pipeline on a secondary deployment that is hosted on a second cloud service.

Abstract: Techniques for triggering pipeline execution based on data change (transaction commit) are described. The pipelines can be used for data ingestion or other specified tasks. These tasks can be operational across account, organization, cloud region, and cloud provider boundaries. The tasks can be triggered by commit post-processing. Gates in the tasks can be set up to reference change data capture information. If the gate is satisfied, tasks can be executed to set up data pipelines.

Abstract: Hybrid tables can be used in different use-case scenarios. Hybrid tables provide a flexible mechanism to support files and data in different formats while providing access to the different types of data as part of one table. This flexibility can allow the use of hybrid tables in data lake or other similar environments.

Abstract: Hybrid tables can be used in different use-case scenarios. Hybrid tables provide a flexible mechanism to support files and data in different formats while providing access to the different types of data as part of one table. This flexibility can allow the use of hybrid tables in data lake or other similar environments.

Abstract: Techniques for triggering pipeline execution based on data change (transaction commit) are described. The pipelines can be used for data ingestion or other specified tasks. These tasks can be operational across account, organization, cloud region, and cloud provider boundaries. The tasks can be triggered by commit post-processing. Gates in the tasks can be set up to reference change data capture information. If the gate is satisfied, tasks can be executed to set up data pipelines.

Abstract: Hybrid tables can be used in different use-case scenarios. Hybrid tables provide a flexible mechanism to support files and data in different formats while providing access to the different types of data as part of one table. This flexibility can allow the use of hybrid tables in data lake or other similar environments.

Abstract: A database platform can track changes using a stream that records changes made by database transactions. Streams on views can be implemented by applying a stream to source data, and applying the view to the streamed data. Users of the database platform can query the stream on the view to review recent changes to the view, such as changes made since the last database transaction.

Abstract: Hybrid tables can be used in different use-case scenarios. Hybrid tables provide a flexible mechanism to support files and data in different formats while providing access to the different types of data as part of one table. This flexibility can allow the use of hybrid tables in data lake or other similar environments.

Abstract: Hybrid tables can be used in different use-case scenarios. Hybrid tables provide a flexible mechanism to support files and data in different formats while providing access to the different types of data as part of one table. This flexibility can allow the use of hybrid tables in data lake or other similar environments.

Abstract: The subject technology determines, using a stream object provided by a network-based database system separate from an external system, a set of offsets of each shard of a set of shards, the set of offsets being determined by executing a user defined function (UDF) included in the stream object, the stream object and the UDF being implemented in a particular programming language different than a database query language. The subject technology identifies an operation to perform on the set of shards. The subject technology, based on the set of shards and the set of offsets, performs, using the UDF included in the stream object, the operation on an external data source hosted by the external system. The subject technology increments a property indicating a different version of the stream object after performing the operation on the external data source, the property being included in the stream object.

Abstract: A database platform can track changes using a stream that records changes made by database transactions. Streams on views can be implemented by applying a stream to source data, and applying the view to the streamed data. Users of the database platform can query the stream on the view to review recent changes to the view, such as changes made since the last database transaction.

Abstract: The subject technology determines, using a stream object provided by a network-based database system separate from an external system, a set of offsets of each shard of a set of shards, the set of offsets being determined by executing a user defined function (UDF) included in the stream object, the stream object and the UDF being implemented in a particular programming language different than a database query language. The subject technology identifies an operation to perform on the set of shards. The subject technology, based on the set of shards and the set of offsets, performs, using the UDF included in the stream object, the operation on an external data source hosted by the external system. The subject technology increments a property indicating a different version of the stream object after performing the operation on the external data source, the property being included in the stream object.

Abstract: Embodiments of the present disclosure may provide cross cloud auto-ingestion techniques. A deployment may monitor multiple queues across different cloud providers and may classify the queues based on their cloud. provider type. The deployment may receive notifications from those queues regarding new data ready for ingestion. The deployment may maintain a pool of credentials and may assign appropriate credentials to each queue. The deployment may route the notifications to appropriate receivers based on cloud provider types. The receivers may then auto-ingest new data in the corresponding queue.

Abstract: A database platform can track changes using a stream that records changes made by database transactions. Streams on views can be implemented by applying a stream to source data, and applying the view to the streamed data. Users of the database platform can query the stream on the view to review recent changes to the view, such as changes made since the last database transaction.

Abstract: A database platform can track changes using a stream that records changes made by database transactions. Streams on views can be implemented by applying a stream to source data, and applying the view to the streamed data. Users of the database platform can query the stream on the view to review recent changes to the view, such as changes made since the last database transaction.

Abstract: The subject technology determines a set of shards corresponding to an external data source accessible via a network, the external data source being hosted by an external system separate from a network-based database system. The subject technology determines, using a stream object, a set of offsets of each shard of the set of shards. The subject technology identifies an operation to perform on the set of shards, the operation comprising a read operation or a write operation. The subject technology, based on the set of shards and the set of offsets, performs the operation on the external data source.

Abstract: An operation (such as a relational query) may be processed on a processing engine (such as a relational database server) on behalf of a client. A conventional processing involves the delivery of the operation to the processing engine, which executes the entire operation to completion and returns a result data set. It may be more efficient to allocate part of the operation to be performed on the client, but a developer may be unable or unavailable to rewrite the operation in a distributed manner. Instead, the operation may be automatically partitioned into a pre-engine client portion, a processing engine portion, and a client portion, and the instructions of each portion may be automatically allocated respectively to the client, the server, and the client. The partitioning may be adjusted to conserve computing resources, such as bandwidth and storage, and the instructions may be reordered to improve the processing of the operation.

Abstract: A compiler supporting a language in which selected semantic objects are represented as data objects. The data objects may be used in multiple ways to expand the capabilities of the programming language. Data objects may be passed to applications and used to create executable instructions for that application. In this way, instructions written in the native language of the compiler may be used to control applications that accept programs in a language inconsistent with the native language of the compiler. The syntax checking and variable binding capabilities of the compiler may be used for those instructions that will be executed by an application separate from the object code generated by the compiler. The semantic objects represented as data objects may be selected based on express operations included in the source code or may be based on implicit type conversion.

Abstract: Queries against data sources (such as language-integrated queries to be applied against relational databases) may be prepared for processing by a query processing pipeline. This pipeline performs services including transforming the query into a form more easily applied by the data source, e.g., into SQL, by matching query terms with query operators configured to handle various query operations. Many query processing pipelines include single-item query operators that perform individual operations (e.g., an updating query operator that specifies an updating of individual records of a table.) An integrated query pipeline may be devised that includes both single-item query operators and multiple-item query operators that specify operations on sets of data items.