Abstract: An advanced search system leverages a pre-trained large language model to enhance user query responses. The system, equipped with hardware processors, a search query via an interface and accesses a pre-trained large language model designed to respond to the search query. The system fine-tunes the model to generate a task-specific generative model. The system employs the task-specific generative model to generate a search result to the search query and analyzes the search result based on a performance metric associated with the task-specific generative model. The system refines the task-specific generative model based on the analyzing of the search result.

Abstract: A deployment infrastructure for sharing usage metrics with customers is described. An example method includes transforming a usage view file into an anonymous block build file. The usage view file corresponds to a view of customer usage data. The method includes embedding the anonymous block build file into a task, and executing the task to produce a share object that is based on the anonymous block build file. The method includes deploying the share object to a customer account, which enables the customer account to access the view of the customer usage data.

Abstract: A cloud data platform determines a set of candidate pre-provisioning times for compute resources to be provisioned from a cloud provider service to a free pool maintained by the cloud platform, the set of candidate pre-provisioning times being based on a demand of compute resources (N) and a target time. For each candidate pre-provisioning time: the cloud data platform finds similar historical periods based on historical distribution of provisioning latencies. For each historical period of the similar historical periods, the cloud data platform determines a minimum number of compute resources (M) to provision at the candidate pre-provisioning time so that at least N compute resources are ready by the target time within a specified confidence.

Abstract: Provided herein are systems and methods for configuring automatic evolution of dynamic tables. An example method includes parsing, by at least one hardware processor, a query associated with a dynamic table to determine a current base object dependency of the dynamic table on at least a first base object. A prior base object dependency of the dynamic table on at least a second base object is retrieved. A delta between data stored by the at least first base object and data stored by the at least second base object is determined. The dynamic table is updated based on the delta.

Abstract: A top K query directed at a table is received. The table is organized into multiple storage units. The top K query comprises a first clause to sort a result set in order and a second clause that specifies a limit on a number of results provided in response to the query. A table scan operator identifies a first set of rows from the table based on a scan set determined for the table and provides the first set of rows to a top K operator. The top K operator determines a current boundary based on the first set of rows and provides the current boundary to the table scan operator. The table scan operator prunes the scan set based on the current boundary and identifies a second set of rows from the table based on the pruning.

Abstract: Various example embodiments described herein provide for systems, methods, devices, instructions, and like for translation to semantic data for artificial intelligence (AI) agent-based semantic data updates (e.g., improvements) using one or more data sources, such as golden data sources, where the updated semantic data (e.g., semantic model) that results can be used by a natural language query system to translate one or more natural language questions into one or more structure language queries (e.g., one or more structured query language (SQL) statements). Some example embodiments are implemented as a system that uses a set of AI agents working in an optimization loop to improve/refine semantic data using the one or more data sources.

Abstract: Various example embodiments described herein provide for systems, methods, devices, instructions, and the like for optimizing a query plan to execute a query by pushing down top-k information in the query plan, which can be used within a data platform (e.g., database) environment. In particular, various example embodiments enable information from a top-k operator of a query plan to be pushed down the query plan and enable pushing the information through one or more outer join operators (e.g., through multiple levels of outer join operators) of the query plan to a select operator, such as an aggregation operator, positioned below the one or more outer join operators.

Abstract: Provided herein are systems, methods, and computer-storage media for managing data skew in hash join operations. A skew manager partitions build-side row data into multiple sets corresponding to hash-join-build (HJB) instances based on hash values. The skew manager detects skew in a build-side row set associated with a first HJB instance by analyzing the number of rows. Upon detecting skew, the skew manager redirects data rows to at least a second HJB instance. The method involves configuring skew caches, generating histograms, and detecting frequent hash values to identify skew. It also includes communicating skew notifications, broadcasting probe-side row data, and adjusting partitioning of probe-side data. The disclosed techniques further include buffering build-side row sets in streams and performing join operations based on these streams, enhancing efficiency in distributed computing environments.

Abstract: A system and method for enhancing data storage security in cloud environments using multiple gateway endpoints. The system includes a cloud data platform with a plurality of networks, each associated with a different gateway endpoint. Each gateway endpoint has a respective gateway endpoint tag. The method involves creating networks and corresponding gateway endpoints, configuring each gateway endpoint to enable traffic for sessions associated with the gateway endpoint tag, and assigning execution nodes to networks with session tokens that include the gateway endpoint tag. This setup ensures that only sessions with the correct tag can access data storage, reducing the risk of unauthorized data exfiltration.

Abstract: The subject technology determines that a request to commit a transaction has been received. The subject technology prior to committing the transaction, performs a write operation to store a metadata version of a table associated with the transaction in a metastore. The subject technology performs a register operation to register a background worker. The subject technology performs, by the background worker, a trigger operation to perform an execution node job on an execution node. The subject technology retrieves, by the execution node job, a set of identifiers of active transactions. The subject technology performs, by the execution node job, a write operation to store the metadata version of the table and the set of identifiers in a datastore. The subject technology receives, by the background worker, an indication that the execution node job has successfully completed. The subject technology performs an unlock operation to unlock the table.

Abstract: The subject technology initiates a reinstallation process of a key-value storage device and locking the key-value storage device. The subject technology performs a bootstrap process for a blob manager and a blob worker. The subject technology performs a restoration process of a storage server. The subject technology applies a set of mutation logs to the storage server. The subject technology unlocks the key-value storage device and enabling network traffic for the key-value storage device.

Abstract: Various example embodiments described herein provide for systems, methods, devices, instructions, and the like for performing commit operations (e.g., write operations) with recovery on a data system, such as a database system, that is interacting with a data source external to the data system, such as an external catalog. For some example embodiments, the database system generates an operation identifier in association with each command to commit a change to an entity of the catalog linked database, where the operation identifier serves as a unique tracking mechanism for commit operations across the distributed systems.

Abstract: A data protection system is provided. The data protection system performs identification of errors from queries on a database. The data protection system can further identify corrupted data from additional errors, that are difficult to detect, and occur between layers of data in the database system. The data protection system can perform corrections of the error data by rebuilding database data or removing the corrupted data.

Abstract: The subject technology determines whether a key exists in a parent table associated with a first transaction. The subject technology performs a first write operation on a child table. The subject technology determines whether a duplicate key exists in the child table based on the key of the first write operation. The subject technology determines whether there is a conflict with the key. The subject technology, in response to determining that there is no conflict with the key, performs a second write operation on a secondary index table of the child table. The subject technology determines whether a particular duplicate key exists in the secondary index table based on a particular key from the second write operation. The subject technology, in response to determining that there is the particular duplicate key in the secondary index table, throws a uniqueness exception.

Abstract: A system includes at least one hardware processor and at least one memory storing instructions that cause the at least one hardware processor to perform operations. The operations include generating a log of changes posted to a plurality of intermediate materialized tables (MTs) during execution of a query in a network-based database system. The query is associated with a source MT that the intermediate MTs depend on. The operations include rendering the log of changes into a dependency graph. The operations include configuring a plurality of processing pipelines based on the dependency graph. The operations include performing refreshes on one or more of the plurality of intermediate MTs in at least one of the plurality of processing pipelines to complete the refresh operation. The refreshes are performed responsive to detecting an instruction for a refresh operation on the source MT.

Abstract: This disclosure provides methods and techniques of data replication involving cross replication group (RG) references. The present disclosure avoids automatic replication failing when an entity in an RG refers to another entity external to the RG. The entity to be replicated within the RG is referred to as the “referring entity,” and the entity as the dangling reference is referred to as the “referred entity.” Although the referring and referred entities are not replicated together in a replication operation, the referred entity may have already been replicated to the target account in another replication operation on a different replication group. In such cases, the data replication procedure may, according to aspects of the present disclosure, check if the referred entity has already been replicated, and if so, proceed to replicate the referring entity without fail, and link the referring and referred entities to enable normal functioning of the referring entity.

Abstract: Techniques for resharing data in multi-tenant data system includes receiving, by a resharer account, a first listing for a first set of data shared by a provider account of the multi-tenant data system, the listing defining an access grant and a resharing grant for the first set of data. The resharer account creates a first database in the resharer account, the first database comprising virtual representations of the first set of data. The resharer account transforms the first set of data to generate a second set of data and stores the second set of data in a second database in the resharer account, the second set of data comprising at least one virtual representation of the first set of data. The resharer account publishes a second listing of the second set of data to share with a consumer account based on the resharing grant for the first set of data in the first listing.

Abstract: An in-database application package and application instance for a data platform. The data platform creates an application instance of an application package having a versioned schema, creates one or more system roles for the application instance, creates a user role and an administrator role for the application instance, creates one or more objects of the application instance based on a versioned schema, and grants one or more use privileges to the one or more roles. Application instances of the application package are upgraded or patched on the data platform based on application package versions. To ensure a proper upgrade or patch, the data platform tracks versions of executing objects of application instances in a call context.

Abstract: Various example embodiments described herein provide for systems, methods, devices, instructions, and the like for using AI model cascades to filter data on a data system, such as a database system, based on an artificial intelligence model prompt (e.g., user-provided prompt). In particular, various example embodiments enable a database system to use cascaded AI models and adaptive bounds to optimize data filtering operations based on an artificial intelligence model prompt (also referred to herein as just a prompt) while balancing computational cost and accuracy, which can be useful in processing large-scale data queries.

Abstract: A data platform for executing containers is provided. In some examples, the data platform receives an application from an application package of a provider account, the application including a setup script and a manifest of a service. The data platform activates access roles based on the manifest and creates the service and a compute pool using the setup script and a specification file accessed from the application package using an access role. The service is executed in the compute pool, accessing objects of the application package and of the data platform using the access roles.