Abstract: Techniques are described for budget tracking in a differentially private security system. A request to perform a query of a private database system is received by a privacy device from a client device. The request is associated with a level of differential privacy. A privacy budget corresponding to the received request is accessed by the privacy device. The privacy budget includes a cumulative privacy spend and a maximum privacy spend, the cumulative privacy spend representative of previous queries of the private database system. A privacy spend associated with the received request is determined by the privacy device based at least in part on the level of differential privacy associated with the received request. If a sum of the determined privacy spend and the cumulative privacy spend is less than the maximum privacy spend, the query is performed. Otherwise a security action is performed based on a security policy.

Abstract: A source table organized into a set of batch units is accessed. The source table comprises a column of data corresponding to a semi-structured data type. One or more indexing transformations for an object in the column are generated. The generating of the one or more indexing transformation includes converting the object to one or more stored data types. A pruning index is generated for the source table based in part on the one or more indexing transformations. The pruning index comprises a set of filters that index distinct values in each column of the source table, and each filter corresponds to a batch unit in the set of batch units. The pruning index is stored in a database with an association with the source table.

Abstract: A process of fulfilling a database deployment request for a data platform. A compute service manager of the data platform scans one or more accounts of a consumer region of the data platform for a pending listing fulfillment request, where the pending listing request includes a request for deployment of a consumer database and an associated share of grant metadata of the consumer database within the consumer region. When the compute service manager determines that an account of the one or more accounts has a pending listing fulfillment request, the compute service manager determines a listing for the pending listing fulfillment request based on listing data of the account. The compute service manager determines that no other fulfillment task is scheduled for the pending listing fulfillment request and schedules a background fulfillment task to perform the fulfillment process for the pending listing fulfillment request.

Abstract: An in-database application package and application instance for a data platform. The data platform creates an application instance of an application package having a versioned schema, creates one or more system roles for the application instance, creates a user role and an administrator role for the application instance, creates one or more objects of the application instance based on a versioned schema, and grants one or more use privileges to the one or more roles. Application instances of the application package are upgraded or patched on the data platform based on application package versions. To ensure a proper upgrade or patch, the data platform tracks versions of executing objects of application instances in a call context.

Abstract: A method of implementing object tagging framework starts with the processor receiving a tag creation command including a tag name. In response to the tag creation command, the processor creates a current tag. The processor then receives an association command, the tag name and a source object identifier. The processor determines a source object associated with the source object identifier. The source object includes a tag value. The processor associates the current tag with the source object. The processor receives a replication command including the source object and a target object. The processor causes replication of the source object to the target object that comprises replicating the current tag with the tag name and the tag value in the source object to the target object. Other embodiments are also described herein.

Abstract: A data platform for running a subset of cells in a notebook is provided. The data platform receives a run cells message from a notebook user interface (UI) application specifying the subset of cells to run. For each cell in the subset, the data platform runs the cell to generate a set of results, generates a cell execution stream using the results, stores the stream, and transmits the stream to the notebook UI application. The notebook UI application generates a display for the user using the cell execution stream. The data platform provides an efficient way to run specific cells in a notebook and display the results to the user.

Abstract: The subject technology receives a query directed to a set of source tables, each source table organized into a set of micro-partitions. The subject technology determines a set of metadata, the set of metadata comprising table metadata, query metadata, and historical data related to the query. The subject technology predicts, using a machine learning model, an indicator of an amount of computing resources for executing the query based at least in part on the set of metadata. The subject technology generates a query plan for executing the query based at least in part on the predicted indicator of the amount of computing resources. The subject technology executes the query based at least in part on the query plan.

Abstract: The subject technology receives, from a metadata database, information related to a base table. The subject technology determines a table object associated with the base table, the table object including a first set of metadata. The subject technology generates a nested object based on a second set of metadata, the second set of metadata including information linking the nested object to the table object. The subject technology generates a second table object associated with the nested object, the second table object representing a secondary index of the base table, the second table object including information linking the second table object to the nested object. The subject technology establishes a link between the second table object to the base table based on the nested object. The subject technology stores, in the metadata database, the nested object and the second table object.

Abstract: A top K query directed at a table is received. The table is organized into multiple storage units. The top K query comprises a first clause to sort a result set in order and a second clause that specifies a limit on a number of results provided in response to the query. A table scan operator identifies a first set of rows from the table based on a scan set determined for the table and provides the first set of rows to a top K operator. The top K operator determines a current boundary based on the first set of rows and provides the current boundary to the table scan operator. The table scan operator prunes the scan set based on the current boundary and identifies a second set of rows from the table based on the pruning.

Abstract: Various embodiments provide for tag-based application of a masking policy, which can be used in connection with a data platform. In particular, various embodiments enable enforcement of one or more masking policies against an entity (e.g., object) of a data platform, such as a database, a table, a row, or a column, based on one or more tags associated with the entity.

Abstract: Provided herein are systems and methods for global data objects on a data platform where the global data objects are accessible at an organization level. In particular, an organization-level global data object provided by various embodiments can be used as a generic organization object that is owned by a specific organization, and can be managed (e.g., created, deleted, or modified) by use of a leader-based model.

Abstract: A command to load or unload data at a storage location is received. In response to the command, a storage integration object associated with the storage location is identified. The storage integration object identifies a cloud identity object that corresponds to a cloud identity that is associated with a proxy identity object corresponding to a proxy identity granted permission to access the storage location. The data is loaded or unloaded at the storage location by assuming the proxy identity.

Abstract: Systems, methods, and devices for retrying a query. A method includes receiving a query directed to database data and assigning execution of the query to one or more execution nodes of an execution platform, the one or more execution nodes configured to execute the query on a first version of a database platform. The method includes determining that execution of the query was unsuccessful. The method includes assigning a first retry execution of the query to the one or more execution nodes of the execution platform and determining whether a regression or an intermittent fault caused the execution of the query to be unsuccessful based at least in part on whether the first retry execution of the query was successful or unsuccessful.

Abstract: A method includes partitioning a database table into a first plurality of partitions. A plurality of changes are executed on the database table in response to a transaction. The changes occur at a corresponding plurality of timestamps and result in a second plurality of partitions. The database table is updated to include a log with the plurality of changes. For each change of the plurality of changes, the log includes and an identification of a portion of the database table the change is applied to. A selection of a first timestamp and a second timestamp of the plurality of timestamps is detected. A delta is generated in response to the selection. The delta indicates a total change occurring to the database table between a first change of the plurality of changes corresponding to the first timestamp and a second change of the plurality of changes corresponding to the second timestamp.

Abstract: Example resource management systems and methods are described. In one implementation, a resource manager is configured to manage data processing tasks associated with multiple data elements. An execution platform is coupled to the resource manager and includes multiple execution nodes configured to store data retrieved from multiple remote storage devices. Each execution node includes a cache and a processor, where the cache and processor are independent of the remote storage devices. A metadata manager is configured to access metadata associated with at least a portion of the multiple data elements.

Abstract: Embodiments of the present disclosure describe techniques for monitoring usage of data listings and pricing data listings based thereon. Metadata comprising a pricing plan may be added to a data listing to generate a monetized data listing. The monetized data listing may be imported to the consumer account. The usage data of the monetized data listing and listing data of the monetized data listing may be exported to a data warehousing and analysis module. A usage level for a set of jobs processed using the monetized data listing during a current incremental interval of a current billing interval may be calculated and combined with a usage level of all previous incremental intervals of the current billing interval to generate a cumulative usage record. An invoice for the billing interval may be generated based on the cumulative usage record.

Abstract: An example method of tuning a machine learning operation can include receiving a data query comprising a reference to an input data set of a database, generating a plurality of unique sets of hyperparameters by varying a hyperparameter value of each set of hyperparameters of the plurality of unique sets of hyperparameters based on the input data set, in response to receiving the data query, training a plurality of machine learning models using the input data set of the data query, each of the plurality of machine learning models configured according to a respective one of a plurality of unique sets of hyperparameters, selecting a first machine learning model of the plurality of machine learning models based on an accuracy of an output of the first machine learning model, and returning the output of the first machine learning model in response to the data query.

Abstract: This disclosure provides methods and techniques of referencing row access policy (RAP) protected mapping tables in a RAP for a data table are disclosed herein. An example method of referencing a mapping table in a data table using nested RAP includes defining, by a processing device, a first access policy for the mapping table to control access by specific users or under specific conditions. The processing device further defines a second access policy attached to the data table referencing the mapping table. The processing device in response to a query, executes the second access policy of the data table to provide a response or operation of data associated with the data table and the mapping table. Executing the second access policy invokes executing the first access policy of the mapping table.

Abstract: Row-level security (RLS) may provide fine-grained access control based on flexible, user-defined access policies to databases, tables, objects, and other data structures. A RLS policy may be an entity or object that defines rules for row access. A RLS policy may be decoupled or independent from any specific table. This allows more robust and flexible control. A RLS policy may then be attached to one or more tables. The RLS policy may include a Boolean-valued expression.

Abstract: A method includes determining that an amount of available space in a first memory used by a set of relational queries is insufficient for a query, wherein the query comprises a join operation. A first partition of a set of partitions is identified, wherein the first partition possesses a smallest available probe memory size of the set of partitions and a build memory size greater than or equal to a threshold memory size, wherein the threshold memory size is a percentage of a maximum build memory size, and the largest partition of the set of partitions has the maximum build memory size. The first partition is copied from the first memory to a second memory.