Abstract: Provided herein are systems and methods to provide a way to share metrics regarding shared data access and accesses associated with data providers for different data listings of the data exchange. For example, the method may comprise detecting one or more client interactions with a set of data listings of a data exchange, the set of data listings associated with one or data providers. The method may further comprise collecting metrics corresponding to the one or more client interactions. In addition, the method may share metrics relevant to the one or more data providers with the one or more data providers.

Abstract: Example caching systems and methods are described. In one implementation, a method identifies multiple files used to process a query and distributes each of the multiple files to a particular execution node to execute the query. Each execution node determines whether the distributed file is stored in the execution node's cache. If the execution node determines that the file is stored in the cache, it processes the query using the cached file. If the file is not stored in the cache, the execution node retrieves the file from a remote storage device, stores the file in the execution node's cache, and processes the query using the file.

Abstract: Techniques for triggering pipeline execution based on data change (transaction commit) are described. The pipelines can be used for data ingestion or other specified tasks. These tasks can be operational across account, organization, cloud region, and cloud provider boundaries. The tasks can be triggered by commit post-processing. Gates in the tasks can be set up to reference change data capture information. If the gate is satisfied, tasks can be executed to set up data pipelines.

Abstract: The subject technology selects a particular zone among multiple zones based on a target skew to meet a global balancing of cluster instances. The subject technology deploys a particular type of cluster instance to the particular zone. The subject technology, for each zone from the multiple zones, determines a respective number of cluster instances. The subject technology identifies a second particular type of cluster instance to add based on a total number of the second particular type of cluster instance in the multiple zones and a second total number of the particular type of cluster instance in the multiple zones. The subject technology adds the second particular type of cluster instance to a second particular zone to meet the global balancing of cluster instances in the multiple zones.

Abstract: A database export system exports data using a plurality of nodes that process the data to generate structured result files that are partitioned by an export parameter in an export request. The database export system distributes the data and merges the files to avoid small file creation and increase processing speed via parallelism. The database export system generates the result files of a specified maximum size in a final format, where the files are processed merged in a temporary file format. The parallel processing is optimized and constrained per the amount of processing nodes, available memory, requested final file sizes, and operation based ordering to complete data exports in a scalable multi-stage approach.

Abstract: Embodiments of the present disclosure describe systems, methods, and computer program products for redacting sensitive data within a database. An example method can include receiving a masking policy for a column of a database, the masking policy identifying a category of sensitive data, examining a column of a database to identify a category of sensitive data in a first location of the column, and, in response to a data query accessing the column, the first location of the column exceeding a threshold probability of comprising sensitive data, executing a redaction operation to redact the category of sensitive data from the first location of the column to generate redacted data for a response to the data query.

Abstract: A method for identity resolution and data enrichment is performed by at least one hardware processor and includes detecting at an account of a data provider, a shared data object that is shared by an account of a data consumer with the account of the data provider. An application executing at the account of the data consumer is enabled for an identity resolution process based on the detecting of the shared data object. A request for source data received from the application is detected at the account of the data provider. The source data is managed by the account of the data provider. The source data is communicated to the application executing at the account of the data consumer, based on a verification that the application is enabled for the identity resolution process. The identity resolution process is performed at the account of the data consumer using the source data.

Abstract: A method includes parsing, by at least one hardware processor, a query to determine query comments and query code associated with the query. A query execution plan is generated based on the query code. Query execution using the query code is performed at a first computing node associated with a query processing pipeline. A detection is made that the query comments are indicative of a software bug in the query code based on analysis of the query comments. The detection is performed at a second computing node associated with a query analysis pipeline. A notification of the software bug and a result of the query execution is output.

Abstract: Methods, systems, and computer programs are presented for providing stored procedures stored within a database to invoke specified functionality in a cloud data platform. A response to a database query is received at a compute node of a cloud data platform. A java database connectivity (JDBC) instance executing within a sandbox process converts the response into a remote procedure call. The remote procedure call is provided to a stored procedure executing within the sandbox process as the response to the database query.

Abstract: A method includes selecting, by at least one hardware processor of a first database deployment, data for transmission to a second database deployment. The data includes a plurality of files forming a data stream. At least a first file of the plurality of files is encoded using a first encryption key which is stored in at least a second file of the plurality of files. The at least a first file is further encoded to include a second encryption key associated with at least a third file of the plurality of files. A set of nodes is identified in a virtual private network connected to the first and second database deployments. The data stream is transmitted from the first database deployment to the second database deployment via the set of nodes of the virtual private network that is connected to the first database deployment and the second database deployment.

Abstract: The subject technology determines a derived encryption key using a cryptographic hash function applied to a hybrid tenant master encryption key and a local random generated identifier. The subject technology encrypts a record value and a key value associated with a transaction using the derived encryption key. The subject technology determines a non-leaf node using a tenant prefix of a tenant. The subject technology inserts the encrypted record value at a leaf node below a non-leaf node of a tree structure associated with the tenant. The subject technology receives a second transaction for performing a read operation on a distributed database. The subject technology retrieves a set of encryption keys based at least in part on an account and the tenant. The subject technology decrypts, using the set of encryption keys, data from the distributed database. The subject technology provides the decrypted data as a result of the second transaction.

Abstract: Systems and methods for managing input and output error of a machine learning (ML) model in a database system are presented herein. A set of test queries is executed on a first version of a database system to generate first test data, wherein the first version of the system comprises a ML model to generate an output corresponding to a function of the database system. An error model is trained based on the first test data and second test data generated based on a previous version of the system. The error model determines an error associated with the ML model between the first and previous versions of the system. The first version of the system is deployed with the error model, which corrects an output or an input of the ML model until sufficient data has been produced by the error model to retrain the ML model.

Abstract: Systems and methods for generating transient object references are provided. The systems and methods perform operations including establishing a session between a first entity and a second entity. The operations include identifying an object that the first entity is authorized to access according to a first set of access privileges. The operations include generating a reference associated with the object. The operations include temporarily authorizing the second entity to access the object using the reference according to a second set of access privileges, the second set of access privileges being derived from the first set of access privileges.

Abstract: Embodiments of the present disclosure may provide a data clean room allowing encryption based data analysis across multiple accounts of different database users. The data clean room may also restrict which data may be used in the analysis and may restrict the output. A requesting user's data can be encrypted using a key and a provider user can generate a shareable database function that accepts the key to decrypt the data to generate the results data without exposing each others' data.

Abstract: Methods, systems, and computer programs are presented for enabling any sandboxed user-defined function code to securely access the Internet via a cloud data platform. A remote procedure call is received by a cloud data platform from a user-defined function (UDF) executing within a sandbox process. The UDF includes code related to at least one operation to be performed. The cloud data platform provides an overlay network to establish a secure egress path for UDF external access. The cloud data platform enables the UDF executing in the sandbox process to initiate a network call.

Abstract: A database dependency resolver system can identify different dependencies of a user application and integrate the identified dependencies in different execution environments of a distributed database system. The different execution environments can manage different versions of a given programming language, or other types of computational architectures (e.g., different CPU types). A database user can provide a database statement (e.g., query) that activates the different dependencies in the different environments to generate results data.

Abstract: Disclosed herein are systems and methods for query processing with restrictions in a database clean room. In an embodiment, a system receives a query directed to a combination of a first source dataset from a first database account of a distributed database and a second source dataset from a second database account of the distributed database. The system generates an approved statements table that contains database statement language that can be executed against the combination of the first and second source datasets. Based on determining that the approved statements table includes the query, the system executes the query to produce results data, and stores the results data in the first database account.

Abstract: A system includes one or more processors and data storage containing instructions executable by the one or more processors to perform operations. The operations include detecting a first executed transaction causing a first modification to table data stored in a table. The table data is associated with a corresponding metadata file with metadata information of the table. A new metadata file is generated responsive to the first executed transaction. The new metadata file includes the metadata information and additional metadata associated with the first modification. A second executed transaction causing a second modification to the table data is detected. The table data is updated with a change tracking column. The change tracking column includes lineage of executed transactions on the table data. The lineage indicates at least the first transaction and the second transaction.

Abstract: Example systems and methods for cloning catalog objects are described. In one implementation, a method includes creating a second catalog object by mapping a second portion of second metadata of the second catalog object to same data of a same data file as a first portion of first metadata of a first catalog object, and, in response to a data storage or data retrieval request directed to the second catalog object, deleting data associated with the second portion of the second metadata from the second catalog object independently of the first catalog object.

Abstract: Aspects of the present disclosure address systems, methods, and devices for tracking object dependencies in a cloud database system. An object dependency created between a referencing object and a referenced object is detected. Based on detecting the object dependency, a dependency record is generated. The dependency record includes dependency information describing the object dependency between the reference object and the referenced object. The dependency record is stored in a database of dependency records.