Abstract: Disclosed are techniques for deduplicating files to be ingested by a database. A bloom filter may be built for each of a first set of files to be ingested into a data exchange to generate a set of bloom filters, wherein each of the set of bloom filters is built with a number of hash functions that is based on a desired false positive rate. The set of bloom filters may be stored in the metadata storage of the data exchange. In response to receiving a set of candidate files to be ingested, identifying using the set of bloom filters, candidate files from the set of candidate files that are duplicative of a file in the first set of files and pruning from the set of candidate files, each candidate file identified as being duplicative of a file in the first set of files using the set of bloom filters.

Abstract: A constraint system enforces projection constraints on data values stored in specified columns of a shared dataset when queries are received by a database system. A projection constraint identifies that the data in a column may be restricted from being projected (e.g., presented, read, outputted) in an output to a received query, while allowing specified operations to be performed on the data and a corresponding output to be provided. For example, the projection constraint may indicate a context for a query that triggers the constraint, such as based on the user that submitted the query. Enforcing projection constraints on queries received at the database system allows for data to be shared and used anonymously by entities to perform various operations without the need to tokenize the data.

Abstract: A system includes one or more processors and data storage containing instructions executable by the one or more processors to perform operations. The operations include detecting a first executed transaction causing a first modification to table data stored in a table. The table data is associated with a corresponding metadata file with metadata information of the table. A new metadata file is generated responsive to the first executed transaction. The new metadata file includes the metadata information and additional metadata associated with the first modification. A second executed transaction causing a second modification to the table data is detected. The table data is updated with a change tracking column. The change tracking column includes lineage of executed transactions on the table data. The lineage indicates at least the first transaction and the second transaction.

Abstract: Aspects of the present disclosure address systems, methods, and devices for tracking object dependencies in a cloud database system. An object dependency created between a referencing object and a referenced object is detected. Based on detecting the object dependency, a dependency record is generated. The dependency record includes dependency information describing the object dependency between the reference object and the referenced object. The dependency record is stored in a database of dependency records.

Abstract: Techniques for creating and using error tables to track errors associated with a base table are described. A command to perform an operation on a base table stored in a network-based data system can be received and executed, causing at least one error. At least one error record corresponding to the at least one error can be inputted into an error table, which is nested with the base table. Contextual information can be added to the at least one error record.

Abstract: Disclosed is an execution information sharing system that writes execution information to a provider target (and other targets) in a secure manner. Execution information generated by an application may be written to a consumer stage, wherein the application is shared by a provider account of a data exchange with a consumer account that executes the application. A consumer exchange service (ES) of the data exchange may send a request to a copy service of the data exchange to copy the execution information from the consumer stage to the provider stage, wherein the consumer ES is a part of the data exchange and is protected from actions of the consumer account. A copy operation may be executed to copy the execution information from the consumer stage to the provider stage using the copy service of the data exchange. The execution information is ingested from the provider stage to a provider table.

Abstract: A database system can configure network devices, such as a primary database in a multi-tenant deployment and a secondary database in a private deployment, to send and receive sequence messages, such as input data indicative of a selection of a link. The database system can create a secure share area in the private deployment in response to receiving the input data indicative of the selection of the link. The database system can replicate the data from the multi-tenant deployment to the secure share area in the private deployment and share the replicated data from the secure share area to the secondary database hosted in the private deployment.

Abstract: Disclosed is an execution information sharing system that duplicates execution information to a provider target (and other targets) as it is being loaded to a consumer target. A first log information object and a second log information object are generated. The first and second log information objects comprise information indicating a consumer target and information indicating a provider target respectively where execution information generated by an application shared with a consumer account of a data exchange is written. A first event unloader and a second event unloader are generated based on the first and second log information objects respectively, wherein the first and second event unloaders are both linked to the application using a mapping. In response to receiving execution information from the application, the execution information is forwarded to the consumer target and the provider target using the first event unloader and the second event unloader respectively.

Abstract: Systems and methods for an organization-level account for an organization on a data platform, users of which can possess administrative or management privileges with respect to the organization and across one or more others accounts of the organization.

Abstract: A method of implementing sub-table replication starts with the processor detecting an update to an entitlements table. The processor performs filtering of a data table based on the update to the entitlements table. The data table including an entitlements column. The processor detects an update to the entitlements column and performs incremental replication of the data table by causing a version-based replication to be executed. Other embodiments are also described herein.

Abstract: The subject technology receives a command to commit a table in a different table format on an external volume. The subject technology generates a first snapshot of the table on internal storage. The subject technology generates a first list of metadata files on the internal storage. The subject technology generates, based on the first list of metadata files, a first set of metadata files on the internal storage. The subject technology generates a second snapshot of the table on the external volume. The subject technology generates a second list of metadata files on the external volume. The subject technology generates, based on the second list of metadata files, a second set of metadata files on the external volume. The subject technology generates a first set of data files in a different file format on the external volume.

Abstract: Embodiments of the present disclosure relate to sharing data using database roles. Database roles are generated within a database container of a provider account. Grants to a particular subset of the plurality of data objects of the database container may be assigned to each of the database roles, and each of the database roles are granted to a share object. The share object is mounted within a consumer account to generate an imported copy of each of the database roles. The imported copy of one or more of the database roles is granted to each of one or more account level roles of the consumer account. When a new object is added to a particular database role, it is immediately available for consumption by any account level roles to which the imported copy of the particular database role has been granted.

Abstract: A process of fulfilling a database deployment request for a data platform. A compute service manager of the data platform scans one or more accounts of a consumer region of the data platform for a pending listing fulfillment request, where the pending listing request includes a request for deployment of a consumer database and an associated share of grant metadata of the consumer database within the consumer region. When the compute service manager determines that an account of the one or more accounts has a pending listing fulfillment request, the compute service manager determines a listing for the pending listing fulfillment request based on listing data of the account. The compute service manager determines that no other fulfillment task is scheduled for the pending listing fulfillment request and schedules a background fulfillment task to perform the fulfillment process for the pending listing fulfillment request.

Abstract: An in-database application package and application instance for a data platform. The data platform creates an application instance of an application package having a versioned schema, creates one or more system roles for the application instance, creates a user role and an administrator role for the application instance, creates one or more objects of the application instance based on a versioned schema, and grants one or more use privileges to the one or more roles. Application instances of the application package are upgraded or patched on the data platform based on application package versions. To ensure a proper upgrade or patch, the data platform tracks versions of executing objects of application instances in a call context.

Abstract: A method of implementing object tagging framework starts with the processor receiving a tag creation command including a tag name. In response to the tag creation command, the processor creates a current tag. The processor then receives an association command, the tag name and a source object identifier. The processor determines a source object associated with the source object identifier. The source object includes a tag value. The processor associates the current tag with the source object. The processor receives a replication command including the source object and a target object. The processor causes replication of the source object to the target object that comprises replicating the current tag with the tag name and the tag value in the source object to the target object. Other embodiments are also described herein.

Abstract: A data platform that implements memoizable functions for database objects. The data platform detects a first execution of a memoizable function and generates a first key based on metadata of one or more database objects operated on by the memoizable function and generates a first result for the memoizable function based on the one or more database objects. The data platform detects a second execution of the memoizable function and generates a second key based on the metadata of the one or more database objects operated on by the memoizable function. When the first key and the second key are equal, the data platform reuses the first result of the memoizable function. When the first key and second key do not match, the data platform generates a second result for the second execution of the memoizable function.

Abstract: Various embodiments provide for tag-based application of a masking policy, which can be used in connection with a data platform. In particular, various embodiments enable enforcement of one or more masking policies against an entity (e.g., object) of a data platform, such as a database, a table, a row, or a column, based on one or more tags associated with the entity.

Abstract: Embodiments of the present disclosure describe techniques for monitoring usage of data listings and pricing data listings based thereon. Metadata comprising a pricing plan may be added to a data listing to generate a monetized data listing. The monetized data listing may be imported to the consumer account. The usage data of the monetized data listing and listing data of the monetized data listing may be exported to a data warehousing and analysis module. A usage level for a set of jobs processed using the monetized data listing during a current incremental interval of a current billing interval may be calculated and combined with a usage level of all previous incremental intervals of the current billing interval to generate a cumulative usage record. An invoice for the billing interval may be generated based on the cumulative usage record.

Abstract: Row-level security (RLS) may provide fine-grained access control based on flexible, user-defined access policies to databases, tables, objects, and other data structures. A RLS policy may be an entity or object that defines rules for row access. A RLS policy may be decoupled or independent from any specific table. This allows more robust and flexible control. A RLS policy may then be attached to one or more tables. The RLS policy may include a Boolean-valued expression.

Abstract: A system for data object replication includes at least one hardware processor and at least one memory storing instructions. The instructions cause the at least one hardware processor to perform operations including parsing a replication request to obtain a data object and identification information identifying a set of databases at a first deployment of a data provider. A dependency of the data object to one or more additional data objects stored in the set of databases is detected. A sequential replication of the data object and the one or more additional data object from the first deployment to a second deployment of the data provider is performed. The second deployment is identified by the replication request. A sequence of the sequential replication is based on the detected dependency.