Abstract: A method includes detecting, by at least one hardware processor, an upload of a user application within a database system. The user application includes user-defined function (UDF) code. A plurality of dependencies of the user application is determined by the at least one hardware processor. A plurality of execution environments corresponding to the plurality of dependencies is generated by the at least one hardware processor. The plurality of execution environments is associated with a corresponding plurality of data types of the UDF. A database query is decoded. The database query specifies database data of a data type of the plurality of data types of the UDF. The database query is processed using at least one of the plurality of execution environments to generate results data. The at least one of the plurality of execution environments corresponds to the data type of the plurality of data types of the UDF.

Abstract: Embodiments of the present disclosure may provide a data protection system that performs identification of errors from queries on a database. The data protection system can further identify corrupted data from additional errors, are difficult to detect, and occur between layers of data in the database system. The data protection system can perform corrections of the error data by rebuilding database data or removing the corrupted data.

Abstract: Using container-centric managed access, an administrator is enabled to define a set of future grants for each object that will be created in the future in a container managed by the administrator. When a user creates a database object, the system checks the future grants to determine if any apply to the user, the database object, or the combination. Any applicable future grants are applied to the database object before the user is allowed to modify it. As a result, the administrator is enabled to control the privileges associated with the database object even before the database object is created, while restricting individual object owners from managing privileges on their owned objects.

Abstract: Methods, systems, and computer programs are presented for providing a cluster view method of a database to perform compaction and clustering of database objects, such as database materialized view. A cluster view system identifies a materialized view including data from one or more base tables, a portion of the data of the materialized view including stale data. The cluster view system performs an integrated task within a maintenance operation on a database, the integrated task including compacting the materialized view, the maintenance operation including clustering the materialized view, and stores the compacted and clustered materialized view in the database.

Abstract: A data platform including an error handling framework for loading of input data. The data platform generates input data columns based on an input file and generates result data columns based on the input data columns and evaluating expressions. The data platform detects projection errors during the generating of the result data columns and stores result error indicators in error indicator arrays of the result data columns based on the projection errors. The data platform generates filtered result data columns based on the result data columns and the result error indicator arrays of the result data columns and stores the filtered result data columns in a database of the data platform.

Abstract: Techniques for secure sharing of stage data include generating a listing in a first data exchange of a data provider. The listing includes stage data stored in a stage of the data provider and metadata associated with the stage data. A secure view of the listing is configured in a second data exchange based on posting the listing from the first data exchange to the second data exchange. The stage data is retrieved in response to a request from a client device to view the stage data received in the second data exchange. A security function is applied to the stage data to generate modified stage data. The modified stage data is stored at a second location in the stage. The metadata is updated to reference the second location in the stage.

Abstract: Joining data using a disjunctive operator is described. An example computer-implemented method can include generating, with a processing device, a query plan for a query, the query comprising a join operator expression for a disjunctive predicate, wherein the join operator expression includes a conjunctive predicate and a disjunctive operator. The method may further include generating a bloom filter for the disjunctive operator. Additionally, the method may include generating a result set as a result of evaluating the join operator expression using the disjunctive operator and bloom filter for the disjunctive predicate.

Abstract: Techniques for managing cluster of devices are described. The clusters may refer to machines or instances of transactional database. A server may control cluster management. The server may operate an observability loop to manage the clusters. The observability loop may operate a plurality of observer phases in parallel, each observer phase monitoring a different parameter relating to the health of the cluster and generating a recommended action to improve the health of the cluster if an incident is present. The observability loop may rank the recommended actions from the different observer phases and may execute the highest ranked recommended action.

Abstract: Techniques for automated query retry in a database platform include decoding, by at least one hardware processor, a query directed to database data. The query is received by a first management node from a client account of a database system. The method further includes configuring an execution of the query by a first computing node associated with the first management node. The method includes detecting the execution of the query results in a failed execution. The query is transferred from the first management node to a second management node based on the failed execution. The second management node is configured to process tasks that are internal to the database system. A retry execution of the query by a second computing node is configured. The second computing node is associated with the second management node. A cause of the failed execution is determined based on a result of the retry execution.

Abstract: A system and method of dynamic task allocation and warehouse scaling. The method includes receiving a request to process a task. The method includes monitoring a plurality of execution nodes of a datastore to determine a plurality of central processing unit (CPU) utilizations. Each CPU utilization of the plurality of CPU utilizations is associated with a respective execution node of the plurality of execution nodes. The method includes identifying, by a processing device based on the plurality of CPU utilizations, a particular execution node associated with a maximum CPU utilization to process the task. The method includes allocating the task to the particular execution node.

Abstract: Methods and systems of managing database failure based on transaction request time is disclosed. A method includes copying a first dataset stored in a primary deployment to a secondary deployment to generate a second dataset. The method includes determining a first arrival time of a first request to perform a first transaction. The method includes determining a second arrival time of a second request to perform a second transaction. The method includes executing the first transaction on the first dataset at the primary deployment in response to determining the first arrival time of the first request to perform the first transaction. The method includes executing the second transaction on the second dataset at the secondary deployment in response to determining the second arrival time of the second request to perform the second transaction.

Abstract: A method for directing queries to encrypted database files includes acquiring a mapping that links a first encrypted file with a different encrypted file. The first encrypted file is generated based on a first encryption key. The method includes generating, by one or more processors based on the mapping, an updated mapping to link a second encrypted file with the different encrypted file. The second encrypted file is generated based on a second encryption key. The method includes directing, using the mapping, a first query to the first encrypted file when the first query arrives before generating the updated mapping; and. The method includes directing, using the updated mapping, a second query to the second encrypted file when a second query arrives after generating the updated mapping.

Abstract: A data platform for developing and deploying a data application. The data platform receives from a first user the data application and provider granted privileges including a consumer usage privilege and a consumer access to data privilege. The data platform authorizes the second user to access the data platform based on one or more consumer account privileges included in a set of account privileges. The data platform authorizes the second user to execute the data application based on the consumer usage privilege. During execution, the data platform authorizes the data application to access the provider database object based on the consumer access to data privilege, and authorizes the data application to access the consumer database object based on a provider access to data privilege provided by the second user.

Abstract: Disclosed is an execution information sharing system that duplicates execution information to a provider target (and other targets) as it is being loaded to a consumer target. A first log information object and a second log information object are generated. The first and second log information objects comprise information indicating a consumer target and information indicating a provider target respectively where execution information generated by an application shared with a consumer account of a data exchange is written. A first event unloader and a second event unloader are generated based on the first and second log information objects respectively, wherein the first and second event unloaders are both linked to the application using a mapping. In response to receiving execution information from the application, the execution information is forwarded to the consumer target and the provider target using the first event unloader and the second event unloader respectively.

Abstract: A method includes generating a task using a plurality of logical statements embedded in a database, the plurality of logical statements corresponding to a data modification. Database data is ingested into a staging table that is configured within the database. The task is executed based on applying the data modification to a first set of partitions storing the database data and generating a second set of partitions. The second set of partitions store modified data corresponding to the database data. A stream of committed transactions is advanced at least in part by adding an entry into the stream. The entry corresponds to committed transactions performed on the first set of partitions during the data modification. A data processing task is scheduled for execution on the modified data based on the advancing of the stream offset.

Abstract: A global and local row count limit associated with a limit query are received by a stop operator of a first execution node among a set of execution nodes that are assigned to process the limit query. Local distributed row count data is generated based on a local row count corresponding to a number of rows output by the first execution node in processing the query. Based on determining the local row count satisfies the local limit, the first execution node buffers rows produced in processing the query. The local distributed row count data is updated based on remote distributed row count data received from a second execution node. A stopping condition is detected based on determining the global limit is satisfied based on updated local distributed row count data and query processing by the first execution node based on detecting the stopping condition.

Abstract: A data platform for running a subset of cells in a notebook is provided. The data platform receives a run cells message from a notebook user interface (UI) application specifying the subset of cells to run. For each cell in the subset, the data platform runs the cell to generate a set of results, generates a cell execution stream using the results, stores the stream, and transmits the stream to the notebook UI application. The notebook UI application generates a display for the user using the cell execution stream. The data platform provides an efficient way to run specific cells in a notebook and display the results to the user.

Abstract: This disclosure provides methods and techniques of data replication involving cross replication group (RG) references. The present disclosure avoids automatic replication failing when an entity in an RG refers to another entity external to the RG. The entity to be replicated within the RG is referred to as the “referring entity,” and the entity as the dangling reference is referred to as the “referred entity.” Although the referring and referred entities are not replicated together in a replication operation, the referred entity may have already been replicated to the target account in another replication operation on a different replication group. In such cases, the data replication procedure may, according to aspects of the present disclosure, check if the referred entity has already been replicated, and if so, proceed to replicate the referring entity without fail, and link the referring and referred entities to enable normal functioning of the referring entity.

Abstract: The subject technology receives a query, the query including a statement for performing the query. The subject technology performs a first lookup operation on a multi-phase cache based on the query. The subject technology performs, in response to a first cache miss of the multi-phase cache, parsing of the statement from the query. The subject technology performs, based on the parsing, a compilation process on the query to generate a compiled query plan, the compilation process determining an optimization and a generalization for the query. The subject technology determines that the compiled query plan is cacheable. The subject technology registers, in response to the compiled query plan being cacheable, a dummy entry in the multi-phase cache.

Abstract: The subject technology generates, by a compute service manager, a schema hash value for a new schema version associated with a new schema version value, the schema hash value based on determining a sum of hash values of a set of attributes of value columns, the set of attributes comprises a column identifier, and a logical type of a column. The subject technology stores a mapping of the schema hash value to the new schema version value for a table in a metadata database. The subject technology stores a new schema entry based on the schema hash value, the new schema version value, and a new column for the table in the metadata database, the metadata database storing multiple entries for different schema versions, each entry including a particular schema hash value for mapping to a corresponding schema version from the different schema versions.