Abstract: A constraint system enforces projection constraints on data values stored in specified columns of a shared dataset when queries are received by a database system. A projection constraint identifies that the data in a column may be restricted from being projected (e.g., presented, read, outputted) in an output to a received query, while allowing specified operations to be performed on the data and a corresponding output to be provided. For example, the projection constraint may indicate a context for a query that triggers the constraint, such as based on the user that submitted the query. Enforcing projection constraints on queries received at the database system allows for data to be shared and used anonymously by entities to perform various operations without the need to tokenize the data.

Abstract: Methods, systems, and computer programs are presented for enabling automated secure data sharing from a private cloud region to a public cloud region and vice versa. A cloud data platform confirms a relationship establishment procedure between a provider and a consumer is recorded with a cloud data platform, the provider being associated with a private cloud deployment and the consumer being associated with a public cloud deployment in a public region. The cloud data platform enables disabling of a firewall policy that is preventing data traffic between the private cloud deployment and the public cloud deployment and enables data sharing between the private cloud deployment and the public cloud deployment. The cloud data platform enables data sharing in a database of the cloud data platform.

Abstract: Techniques for creating and using error tables to track errors associated with a base table are described. A command to perform an operation on a base table stored in a network-based data system can be received and executed, causing at least one error. At least one error record corresponding to the at least one error can be inputted into an error table, which is nested with the base table. Contextual information can be added to the at least one error record.

Abstract: Provided herein are systems and methods for identity resolution and data enrichment. An example method performed by at least one hardware processor includes detecting at an account of a data provider, an update to personally identifiable information (PII). The PII is stored in a source table managed by an account of a data consumer. An identity resolution process is performed based on detecting the update. The identity resolution process includes generating a secure identifier of a user associated with the PII. The method further includes generating at the account of the data provider, a result table including the secure identifier. The result table is shared with the account of the data consumer.

Abstract: Techniques for fetching query result data using result batches includes generating a plurality of result batches based on query result information. The query result information is associated with query result data generated from execution of a query. Each result batch of the plurality of result batches includes a result data retrieval function for a corresponding data portion of a plurality of data portions of the query result data. The plurality of result batches are encoded for distribution to a corresponding plurality of computing nodes. The techniques further include causing retrieving of the plurality of data portions of the query result data by the corresponding plurality of computing nodes based on the result data retrieval function for each of the plurality of data portions.

Abstract: The subject technology receives, at a first execution node, a first transaction, the first transaction to be executed on linearizable storage. The subject technology determines whether the first execution node corresponds to a rank indicating a leader worker. The subject technology, in response to the first execution node corresponding to the rank indicating the leader worker, performs, by the first execution node, an initialization process for executing the first transaction. The subject technology broadcasts a first read timestamp associated with the first transaction to a set of execution nodes, the set of execution nodes being different than the first execution node. The subject technology executes, by the first execution node, at least a first operation from the first transaction.

Abstract: A method includes parsing a query to determine a plurality of data processing operations associated with the query and including an AS OF JOIN operation between first time series data in a first table and second time series data in a second table. A query plan of the query is generated. The query plan includes a plurality of nodes corresponding to the plurality of data processing operations. At least one of the plurality of nodes corresponding to the AS OF JOIN operation is modified to generate a modified query plan of the query. The modifying is based on applying a UNION operation on at least a first portion of column data in the first table and the second table to obtain a combined table. Execution of the query by at least one of a plurality of computing nodes is scheduled based on the modified query plan.

Abstract: Systems and methods for generating a machine-learning (ML) model for extracting information from one or more electronic documents, where the ML model can be used as a data object, which can be part of a database command or as part of a document information extraction process that is continuously running (e.g., document information extraction pipeline).

Abstract: Provided herein are systems and methods for global data objects on a data platform where the global data objects are accessible at an organization level. In particular, an organization-level global data object provided by various embodiments can be used as a generic organization object that is owned by a specific organization, and can be managed (e.g., created, deleted, or modified) by use of a leader-based model.

Abstract: Systems, methods, and devices for sharing materialized views in multiple tenant database systems. A method includes defining a materialized view over a source table that is associated with a first account of a multiple tenant database. The method includes defining cross-account access rights to the materialized view to a second account such that that second account can read the materialized view without copying the materialized view. The method includes modifying the source table for the materialized view. The method includes identifying whether the materialized view is stale with respect to the source table by merging the materialized view and the source table.

Abstract: Resource provisioning systems and methods are described. In an embodiment, a system includes a plurality of shared storage devices collectively storing database data, an execution platform, and a compute service manager. The compute service manager is configured to determine a task to be executed in response to a trigger event and determine a query plan for executing the task, wherein the query plan comprises a plurality of discrete subtasks. The compute service manager is further configured to assign the plurality of discrete subtasks to one or more nodes of a plurality of nodes of the execution platform, determine whether execution of the task is complete, and in response to determining the execution of the task is complete, store a record in the plurality of shared storage devices indicating the task was completed.

Abstract: Systems and methods for an organization-level account for an organization on a data platform, users of which can possess administrative or management privileges with respect to the organization and across one or more others accounts of the organization.

Abstract: Techniques for identity resolution and data enrichment include configuring, during an onboarding process at an account of a data provider, at least one parameter associated with access to identity resolution functions by an account of a data consumer. A first shared data object is generated at the account of the data provider. The first shared data object corresponds to a second shared data object at the account of the data consumer. The second shared data object at the account of the data consumer is enabled for sharing of log data associated with an application executing at the account of the data consumer. The application is enabled for an identity resolution process based on the detecting of the second shared data object. Source data associated with the identity resolution functions is encoded for communication to the application at the account of the data consumer based on the enabling.

Abstract: Sub-plans are executed in parallel using a plurality of execution nodes, which can be part of a data platform. In particular, various embodiments identify sub-plans (e.g., fragments or portions of one or more child operators) of a root operator in a query plan such that the identified sub-plans that are candidates for execution on a single execution node, determine a cost estimate for causing the candidate sub-plans to be executed in parallel using multiple execution nodes, and cause the candidate sub-plans to be executed in parallel based on the cost estimate.

Abstract: Provided herein are systems and methods for configuring trace events. A system includes at least one hardware processor coupled to a memory and configured to instantiate a user code runtime to execute user-defined function (UDF) code. The user code runtime is instantiated within a sandbox process of an execution node. An application programming interface (API) call is detected during execution of the UDF code. The API call includes one or more configurations of a trace event. Telemetry information is collected based on the one or more configurations. The telemetry information is associated with the trace event using a telemetry API. The telemetry API corresponds to the API call. The telemetry information is formatted using the telemetry API, to generate structured telemetry information. The at least one hardware processor causes ingestion of the structured telemetry information into an event table.

Abstract: The subject technology receives a command to commit a table in a different table format on an external volume. The subject technology generates a first snapshot of the table on internal storage. The subject technology generates a first list of metadata files on the internal storage. The subject technology generates, based on the first list of metadata files, a first set of metadata files on the internal storage. The subject technology generates a second snapshot of the table on the external volume. The subject technology generates a second list of metadata files on the external volume. The subject technology generates, based on the second list of metadata files, a second set of metadata files on the external volume. The subject technology generates a first set of data files in a different file format on the external volume.

Abstract: The subject technology inserts, by a first transaction, a new version of an object, the first transaction including a first statement to perform an update operation to a row in a first table, the object corresponding to data in the row to be updated, the first statement including information comprising an object key associated with the object. The subject technology performs, by a second transaction, a range read, the range read including information indicating the object key. The subject technology receives a set of conflicting transactions from the range read. The subject technology determines that a conflict occurred between the first transaction and a third transaction from the set of conflicting transactions. The subject technology performs a restart of the first transaction in response to determining that the conflict occurred.

Abstract: A streaming ingest platform can improve latency and expense issues related to uploading data into a cloud data system. The streaming ingest platform can organize the data to be ingested into per-table chunks and per-account blobs. This data may be committed and may be made available for query processing before it is ingested into the target source tables. This significantly improves latency issues. The streaming ingest platform can also accommodate uploading data from various sources with different processing and communication capabilities, such as Internet of Things (IOT) devices.

Abstract: The subject technology provides embodiments for enabling derivation of predicates not only from other predicates but also from metadata such as expression properties. In examples, predicates are derived, avoiding unwanted impact on cardinality estimation. In other examples, predicates are derived, avoiding artificial runtime errors and providing a way to avoid security issues with secure views.

Abstract: Disclosed are systems, methods, and non-transitory computer-readable media for sharing, on a distributed database, a database application to a first user of the distributed database, the database application generated by a second user of the distributed database. The training dataset includes a first database training dataset from the first user of the distributed database and a second database training dataset from the second user of the distributed database, the first database training dataset and the second database training dataset including non-overlapping dataset features. The database application further identifies a query from the second user to train the machine learning model on the training dataset and generates a trained machine learning model by training the machine learning model on a joined dataset according to the query. The database application generates outputs from the trained machine learning model by applying the trained machine learning model on new data.