Abstract: The subject technology initiates a reinstallation process of a key-value storage device and locking the key-value storage device. The subject technology performs a bootstrap process for a blob manager and a blob worker. The subject technology performs a restoration process of a storage server. The subject technology applies a set of mutation logs to the storage server. The subject technology unlocks the key-value storage device and enabling network traffic for the key-value storage device.

Abstract: The subject technology determines that a request to commit a transaction has been received. The subject technology prior to committing the transaction, performs a write operation to store a metadata version of a table associated with the transaction in a metastore. The subject technology performs a register operation to register a background worker. The subject technology performs, by the background worker, a trigger operation to perform an execution node job on an execution node. The subject technology retrieves, by the execution node job, a set of identifiers of active transactions. The subject technology performs, by the execution node job, a write operation to store the metadata version of the table and the set of identifiers in a datastore. The subject technology receives, by the background worker, an indication that the execution node job has successfully completed. The subject technology performs an unlock operation to unlock the table.

Abstract: Various example embodiments described herein provide for systems, methods, devices, instructions, and the like for performing commit operations (e.g., write operations) with recovery on a data system, such as a database system, that is interacting with a data source external to the data system, such as an external catalog. For some example embodiments, the database system generates an operation identifier in association with each command to commit a change to an entity of the catalog linked database, where the operation identifier serves as a unique tracking mechanism for commit operations across the distributed systems.

Abstract: A data protection system is provided. The data protection system performs identification of errors from queries on a database. The data protection system can further identify corrupted data from additional errors, that are difficult to detect, and occur between layers of data in the database system. The data protection system can perform corrections of the error data by rebuilding database data or removing the corrupted data.

Abstract: The subject technology determines whether a key exists in a parent table associated with a first transaction. The subject technology performs a first write operation on a child table. The subject technology determines whether a duplicate key exists in the child table based on the key of the first write operation. The subject technology determines whether there is a conflict with the key. The subject technology, in response to determining that there is no conflict with the key, performs a second write operation on a secondary index table of the child table. The subject technology determines whether a particular duplicate key exists in the secondary index table based on a particular key from the second write operation. The subject technology, in response to determining that there is the particular duplicate key in the secondary index table, throws a uniqueness exception.

Abstract: A system includes at least one hardware processor and at least one memory storing instructions that cause the at least one hardware processor to perform operations. The operations include generating a log of changes posted to a plurality of intermediate materialized tables (MTs) during execution of a query in a network-based database system. The query is associated with a source MT that the intermediate MTs depend on. The operations include rendering the log of changes into a dependency graph. The operations include configuring a plurality of processing pipelines based on the dependency graph. The operations include performing refreshes on one or more of the plurality of intermediate MTs in at least one of the plurality of processing pipelines to complete the refresh operation. The refreshes are performed responsive to detecting an instruction for a refresh operation on the source MT.

Abstract: This disclosure provides methods and techniques of data replication involving cross replication group (RG) references. The present disclosure avoids automatic replication failing when an entity in an RG refers to another entity external to the RG. The entity to be replicated within the RG is referred to as the “referring entity,” and the entity as the dangling reference is referred to as the “referred entity.” Although the referring and referred entities are not replicated together in a replication operation, the referred entity may have already been replicated to the target account in another replication operation on a different replication group. In such cases, the data replication procedure may, according to aspects of the present disclosure, check if the referred entity has already been replicated, and if so, proceed to replicate the referring entity without fail, and link the referring and referred entities to enable normal functioning of the referring entity.

Abstract: Techniques for resharing data in multi-tenant data system includes receiving, by a resharer account, a first listing for a first set of data shared by a provider account of the multi-tenant data system, the listing defining an access grant and a resharing grant for the first set of data. The resharer account creates a first database in the resharer account, the first database comprising virtual representations of the first set of data. The resharer account transforms the first set of data to generate a second set of data and stores the second set of data in a second database in the resharer account, the second set of data comprising at least one virtual representation of the first set of data. The resharer account publishes a second listing of the second set of data to share with a consumer account based on the resharing grant for the first set of data in the first listing.

Abstract: An in-database application package and application instance for a data platform. The data platform creates an application instance of an application package having a versioned schema, creates one or more system roles for the application instance, creates a user role and an administrator role for the application instance, creates one or more objects of the application instance based on a versioned schema, and grants one or more use privileges to the one or more roles. Application instances of the application package are upgraded or patched on the data platform based on application package versions. To ensure a proper upgrade or patch, the data platform tracks versions of executing objects of application instances in a call context.

Abstract: Various example embodiments described herein provide for systems, methods, devices, instructions, and the like for using AI model cascades to filter data on a data system, such as a database system, based on an artificial intelligence model prompt (e.g., user-provided prompt). In particular, various example embodiments enable a database system to use cascaded AI models and adaptive bounds to optimize data filtering operations based on an artificial intelligence model prompt (also referred to herein as just a prompt) while balancing computational cost and accuracy, which can be useful in processing large-scale data queries.

Abstract: A data platform for executing containers is provided. In some examples, the data platform receives an application from an application package of a provider account, the application including a setup script and a manifest of a service. The data platform activates access roles based on the manifest and creates the service and a compute pool using the setup script and a specification file accessed from the application package using an access role. The service is executed in the compute pool, accessing objects of the application package and of the data platform using the access roles.

Abstract: The subject technology receives first party training data provided by an end-user of a baseline machine learning model. The subject technology determines a first set of common features based on the first party training data. The subject technology receives, from at least one data source. The subject technology determines a second set of common features based on the set of datasets. The subject technology trains, using the first set of common features and the second set of common features, a second machine learning model, the second machine learning model incorporating additional training data from the external data supplier during training compared to the baseline machine learning model. The subject technology generates a boosted machine learning model based at least in part on the training, the boosted machine learning model comprising the trained second machine learning model.

Abstract: Systems and methods are provided for generating personalized service disruption notifications. The system allocates resources of a database system to a plurality of entities, the resources of the database system being distributed in a cloud environment and analyzes a plurality of signals on the database system. The system, in response to analyzing the plurality of signals, detects a likelihood of a service availability disruption on the database system for a first entity of the plurality of entities. The system notifies the first entity of the service availability disruption in response to detecting the likelihood of the service availability disruption.

Abstract: An advanced system for refining overlap queries in a database system based on user feedback. The system monitors interactions of a first user with a first dataset on the database system, where the first dataset is associated with the first user. Feedback regarding the quality of a results dataset, generated from an executed overlap query, is received from the first user. This feedback informs the generation of a similarity score dataset that enhances the creation of new overlap queries. These new overlap queries are designed to output refined overlap datasets between the first dataset and a second dataset associated with a second user. A new joined dataset is generated by executing these overlap queries, comprising data from both the first and second datasets. A new results dataset is generated, providing the first user with refined recommendations based on additional feedback.

Abstract: Example caching systems and methods are described. In one implementation, a method receives a set of queries to be processed by a set of virtual warehouses. The method distributes the set of queries to the set of virtual warehouses to be executed and creates, during the processing of the set of queries by the set of virtual warehouses, a new virtual warehouse, wherein cache resources associated with the new virtual warehouse are populated with data files associated with the set of queries at the time the virtual warehouse is created and the cache resources vary among the processors, wherein a first subset of the processors comprises minimal cache resources and a second subset of processors comprises cache resources providing faster input-output operations. The method redistributes the set of queries across the set of virtual warehouses.

Abstract: Example caching systems and methods are described. In one implementation, a method receives a query, at an execution platform, directed to data stored across a plurality of shared storage devices, the execution platform comprising one or more execution nodes, an execution node comprising a plurality of processors. The method processes the query using the one or more execution nodes of the execution platform, and in response to a determination of a change in a number of execution nodes of the execution platform, wherein the change is creating a new execution node, wherein a first subset of the plurality of processors comprises a minimal cache and a second subset of the plurality of processors comprises a cache providing faster input-output operations, reassigns processing of the query, among the changed number of execution nodes of the execution platform.

Abstract: A computer-implemented method includes accessing virtual private cloud flow logs of network traffic data originating from a virtual private cloud, generating filtered flow logs by filtering the virtual private cloud flow logs, extracting features based on a plurality of attributes from the filtered flow logs, training one or more machine learning models based on the features, applying the one or more machine learning models to the network traffic data to identify potential beacons, generating an alert notification that identifies the potential beacons, and communicating the alert notification to an alerting system.

Abstract: Embodiments of the present disclosure provide techniques for efficient computation over a wide table. A processing device determines that a first number of columns of a first table is greater than a threshold number of columns. The processing device transforms the first table into a second table based on the determination, where the second table includes a second number of columns that is less than the first number of columns, and where the second table includes a first column that includes first fields that identify columns of the first table, a second column that includes second fields that identify data types of fields of the first table, and a third column that includes third fields that include data of the fields of the first table. The processing device executes a UDTF on the second table.

Abstract: The subject technology receives, by a first set of decoupled compute instances of a distributed database, a plurality of transactional queries against a transactional processing database stored on a first set of decoupled storage instances, at least one of the first set of decoupled compute instances bypassing a compiler of the distributed database to process the plurality of transactional queries using locally cached data. The subject technology receives at least one select query for processing on an aggregated analytical query database of the distributed database. The subject technology generates, by the compiler of the distributed database, a query plan for execution of the at least one select query using a second set of decoupled compute instances. The subject technology generates, by the second set of decoupled compute instances, an aggregated analytical query result dataset for the at least one select query according to the query plan.

Abstract: A system is disclosed comprising a memory containing instructions and one or more computer processors. When the instructions are executed, the system performs an operation to configure a Domain Name System (DNS) proxy, executing in a node of a cloud data platform associated with a first account, to perform hostname resolution of an Account Host Identifier (AHID) of the first account. The DNS proxy receives a DNS request from a process executing in a pod of the node, and the system fails to resolve the DNS request if the name in the DNS request differs from the AHID of the first account. The system returns an Internet Protocol (IP) address if the name in the DNS request matches the AHID. The process executing in the pod of the node is configured to send data to data storage of the cloud data platform using the returned IP address.