Abstract: In an embodiment, a data platform creates an application in a data-provider account. The application includes one or more APIs corresponding to one or more underlying code blocks. The data platform shares provider data with the application in the data-provider account, and also installs, in a data-consumer account, an application instance of the application. The application instance includes one or more APIs corresponding to the one or more APIs in the application in the data-provider account. The data platform shares consumer data with the application instance in the data-consumer account, and invokes one or more of the APIs of the application instance to execute respective associated underlying code blocks, which are not visible to the data-consumer account. The data platform also saves output of the one or more respective associated underlying code blocks locally within the data-consumer account.

Abstract: Interleaved infrastructure can interleave fragment execution during query compilation. For example, compilation can be paused, and a plan fragment is executed. The results of the plan fragment can be fed back into the compiler, which can then restart the compilation and modify the compilation based on the results of the plan fragment execution. The interleaved workflow provides a flexible, robust infrastructure, which can enable and improve a variety of database features and query optimizations.

Abstract: Techniques for creating, sharing, and using bundles (also referred to as packages) in a multi-tenant database are described herein. A bundle is a schema object with associated hidden schemas. A bundle can be created by a provider user and can be shared with a plurality of consumer users. The bundle can be used to enable code sharing and distribution without losing control while maintaining security protocols.

Abstract: Techniques for active invalidation of cache entries are described. A first timestamp and a second timestamp are stored by a compute instance, the first timestamp corresponding to an initial time the computing instance read from a version update log in a database and the second timestamp corresponding to a last time the computing instance read from the version update log in the database. The compute instance can read records from the version update log that are associated with version timestamps greater than the second timestamp. The compute instance can invalidate a first set of cache entries with older versions stored in a cache based on reading the records, retaining a second set of cache entries, and then execute a query using the second set of cache entries.

Abstract: The subject technology generates, by a database system, cell data for a particular table based on values from a source table, the values being based on raw input data, the source table comprising multiple rows and multiple columns, the raw input data comprising values in a first format, the values comprising input features corresponding to datasets included in the raw input data for machine learning models, the source table being provided by an external environment, the external environment comprising an external system from the database system. The subject technology performs a database operation to generate the particular table including table metadata, column metadata, and the generated cell data, the generated particular table comprising a second format that causes more efficient processing of data by the database system using a single query on the particular table compared to processing the raw input data from the source table.

Abstract: A differentially private security system communicatively coupled to a database storing restricted data receives a database query from a client. The database query includes a relation specifying a set of data in the database upon which to perform the query and privacy parameters associated with the query. The differentially private security system determines a worst-case privacy spend for the query based on the privacy parameters and the relation. The differentially private security system performs the query upon the set of data specified by the relation and decrements the determined worst-case privacy spend from a privacy budget associated with the client. The differentially private security system records the worst-case privacy spend and the query at a log and determines a privacy budget refund based on queries recorded in the log. The differentially private security system applies the determined privacy budget refund to the privacy budget associated with the client.

Abstract: A source table can be provided by a provider account in a data system. A secure view of the source table is provided to one or more consumer accounts, the secure view limiting access to a subset of data in the source table. A plan to execute a command using the secure view may be generated, the plan including a secure view boundary on a subset of operations defining the secure view. The plan may be modified to move a first operation that was outside the secure view boundary to within the secure view boundary to generate a second plan to optimize performance in view of limitations or restrictions placed by the secure view.

Abstract: An external function system can be implemented on a database to perform processing on one or more external network services. The external function system can comprise a particular external function for a particular external service, an outbound serializer function, and an inbound serializer function that are linked with the particular external function. The outbound serializer function can be configured to transform the data of a query from a database format to a different format of the particular external network service. The inbound deserializer function can be configured to receive data returned from the external service and transform the data to the format of the database.

Abstract: The subject technology sends a first statement to an execution node for executing the first statement on first storage using micro-partitions. The subject technology sends a second statement to the execution node for executing the second statement on linearizable storage. The subject technology sends a request to prepare a commit of a cross domain transaction associated with the first statement and the second statement. The subject technology generates a new version of a set of tables that were modified by the cross domain transaction and updating first metadata in a metadata database to indicate the new version. The subject technology finalizes the commit of the cross domain transaction and updates second metadata that the cross domain transaction has been committed.

Abstract: The subject technology receives a set of values for a data set corresponding to a request for determining a Pearson correlation. The subject technology determines a number of rows from the data set. The subject technology determines a number of execution nodes to execute the request. The subject technology determines a set of shards of rows from the data set. The subject technology, for each shard from the set of shards, sends a particular user defined table function (UDTF), including a particular shard of rows, to a different execution node to perform a set of operations. The subject technology provides a set of output values of each particular UDTF corresponding to each shard from the set of shards in a second UDTF. The subject technology sends the second UDTF to a particular execution node to perform an aggregate operation using the set of output values of each particular UDTF.

Abstract: The subject technology serializes, by at least one hardware processor, non-primary key data of column-organized data into compressed serialized value data that is in a row-organized sequence, the compressed serialized value data compressed using at least one bitmap, the non-primary key data comprising a schema identifier, the column-organized data being stored in a columnar database system, the column-organized data comprising primary key data and the non-primary key data. The subject technology stores the compressed serialized value data in a key-value data store of a key-value database system, the key-value database system processing key-value data in a key-value format. The subject technology receives a query by the columnar database system. The subject technology deserializes a portion of the compressed serialized value data that corresponds to the query. The subject technology processes the query using the columnar database system.

Abstract: A database system configured to manage and execute stored procedures within a secure sandbox process. The system receives a response to a database query and, through an Application Programming Interface (API) executing within the sandbox process, converts the response into a remote procedure call. The sandbox process is modified to restrict communication with external networks while enabling communication with a designated execution node, which facilitates interaction between the stored procedure and database system components. The stored procedure, executing within the confines of the sandbox process, directs the API to communicate with the execution node. The execution node, in turn, submits the database query to the database system component.

Abstract: A method, apparatus, and system for join operations of a plurality of relations that are distributed over a plurality of storage locations over a network of computing components.

Abstract: Embodiments of the present disclosure may provide a data clean room allowing secure data analysis across multiple accounts, without the use of third parties. Each account may be associated with a different company or party. The data clean room may provide security functions to safeguard sensitive information. For example, the data clean room may restrict access to data in other accounts. The data clean room may also restrict which data may be used in the analysis and may restrict the output. The overlap data may be anonymized to prevent sensitive information from being revealed.

Abstract: A method includes retrieving a plurality of materialized tables (MTs). Each of the plurality of MTs includes a lag duration and refers to a corresponding base table of a plurality of base tables. The lag duration indicates a maximum time period that a result of a prior refresh of a query on the corresponding base table can lag behind a current time instance. A plurality of time instances for the MT is determined based on the lag duration and a number of prior refreshes of the corresponding base table. A plurality of aligned time instances for the plurality of MTs is determined based on the plurality of time instances for each of the plurality of MTs. Refresh operations are scheduled for the plurality of MTs at one or more of the plurality of aligned time instances that are within the maximum time period.

Abstract: Provided herein are systems and methods for providing a secure and efficient way to manage a data exchange's availability and manage the visibility of data listings of the data exchange. For example, the method may comprise specifying, by an exchange admin of the data exchange, a set of regions where the data exchange is available, each of the set of regions comprising one or more remote deployments. The method may further comprise specifying by a data provider, one or more of the set of regions where a data listing owned by the data provider is visible. Upon receiving a request to access the data listing from a remote deployment of the one or more regions, the data provider may determine whether to reject or fulfill the request. In response to determining that the request should be fulfilled, replicating data of the data listing to the remote deployment.

Abstract: A method for secure data synchronization and sharing between distinct cloud environments is disclosed. The method involves establishing a trust relationship between a data provider's private cloud deployment and a data consumer's public cloud deployment. Utilizing hardware processors, a cross-region group sharing token is generated and validated to facilitate secure communication and data traffic between the two cloud deployments. Upon validation, a firewall policy that previously restricted data exchange is disabled, allowing for the replication of a target database from the private cloud to a secure share area accessible by the public cloud deployment. The data consumer is then enabled to access and utilize the replicated database within this secure area. The method ensures consistent data management across both deployments by managing the synchronization of the replicated database with the data provider's original database, thereby maintaining data consistency and integrity.

Abstract: Techniques for configuring event tables include retrieving, by at least one hardware processor of a computing node, log data at a first process of the computing node. The log data is associated with a function executing at a second process of the computing node. The log data is revised using a table stage to generate revised log data. The table stage is configured as a dedicated storage location of the first process. The revising includes a data enrichment process based on metadata associated with execution of the function at the second process. The revised log data is ingested into an event table.

Abstract: Techniques for creating and using error tables to track errors associated with a base table are described. A command to perform an operation on a base table stored in a network-based data system can be received and executed, causing at least one error. At least one error record corresponding to the at least one error can be inputted into an error table, which is nested with the base table. Contextual information can be added to the at least one error record.

Abstract: Hybrid tables can be used in different use-case scenarios. Hybrid tables provide a flexible mechanism to support files and data in different formats while providing access to the different types of data as part of one table. This flexibility can allow the use of hybrid tables in data lake or other similar environments.