Abstract: Monitoring health metrics of computing devices in a data system can be implemented at different levels. At a first level, local background services can be run locally on the computing devices to monitor a set of health metrics on the respective computing devices. At a second level, a central health marker service can monitor a set of health metrics for the computing devices.

Abstract: Different database deployments, or other data system deployments, may want to communicate with each other without sacrificing security or control. To this end, embodiments of the present disclosure may provide secure message exchange techniques for a source and/or target deployment. Configurable rule sets may be stored in the deployments; the rule sets may define what messages may be communicated between deployments. The deployments may implement a selective filtering scheme in one or more stages based on the rule sets to filter outgoing and/or incoming messages.

Abstract: Embodiments of the present disclosure may provide a data clean room allowing secure data analysis across multiple accounts, without the use of third parties. Each account may be associated with a different company or party. The data clean room may provide security functions to safeguard sensitive information. For example, the data clean room may restrict access to data in other accounts. The data clean room may also restrict which data may be used in the analysis and may restrict the output. The overlap data may be anonymized to prevent sensitive information from being revealed.

Abstract: A method for data sharing between cloud deployments is disclosed. A provider in a virtual private cloud (VPC) deployment generates a listing of data for sharing with a consumer in a public multi-tenant deployment. After creating a public account and transmitting a link associated with the listing to the consumer, the method responds to the consumer's selection of the link by automatically creating a secure shared area in the VPC deployment. The method includes replicating both a database and share object associated with the listing into the secure shared area as respective replicas. The replicated objects are then shared with the consumer account as data share objects, enabling the consumer to launch them in a consumer mounted database. The method concludes by enabling the consumer to perform database operations on the provider's data within the VPC deployment, ensuring secure and efficient data sharing between the cloud environments.

Abstract: The subject technology receives a first query. The subject technology determines a set of rows that are modified based on performing the first DML operation on the first table, the set of rows including a first set of updated rows. The subject technology generates a first delta file based on the determined set of rows. The subject technology generates a first updated rows file based on a first set of updated rows. The subject technology determines a second set of rows of the first table that have not been modified by performing the first DML operation on the first table. The subject technology generates a first residual file based on the second set of rows. The subject technology performs a validation process to validate the first DML operation using at least the first delta file, the first residual file, and a matching CoW file.

Abstract: Described herein are techniques for data quality monitoring in a network-based data system. A data metric function used to evaluate data quality can be stored, where the data metric function is defined as schema level object. The data metric function can be attached to a table associated with an account and is evaluated on data associates with the table to generate evaluation results. The evaluation results can be stored in an account-specific central database, from which access is provided to the evaluation results to a user for the account.

Abstract: Provided herein are systems, methods, and computer-storage media for managing data object access in a catalog service account. The method includes detecting, at a catalog service account, by a catalog manager (CM), a user request to access a data object. The user request includes identification information of the user and a principal role associated with the user. The CM retrieves one or more catalog roles from a first catalog of the catalog service account. The one or more catalog roles correspond to the principal role. The CM determines a set of hierarchical data objects based on the user request. The set of hierarchical data objects comprise the data object. The CM performs a determination of whether access privileges of the set of hierarchical data objects and access privileges of the one or more catalog roles allow access to the data object. The CM grants access to the data object based on the determination.

Abstract: Methods, systems, and computer programs are described for tracking evaluation of workload stability through performance indexing. A plurality of metric source data is received by at least one hardware processor. Based on this data, a workload is identified as a stable workload candidate. A performance index is then generated, reflecting the characteristics of the identified stable workload candidate. The performance index is continuously tracked over a period of time, enabling the detection and analysis of any modifications to the workload and the subsequent impact on system performance.

Abstract: An advanced search system leverages a pre-trained large language model to enhance user query responses. The system, equipped with hardware processors, a search query via an interface and accesses a pre-trained large language model designed to respond to the search query. The system fine-tunes the model to generate a task-specific generative model. The system employs the task-specific generative model to generate a search result to the search query and analyzes the search result based on a performance metric associated with the task-specific generative model. The system refines the task-specific generative model based on the analyzing of the search result.

Abstract: A deployment infrastructure for sharing usage metrics with customers is described. An example method includes transforming a usage view file into an anonymous block build file. The usage view file corresponds to a view of customer usage data. The method includes embedding the anonymous block build file into a task, and executing the task to produce a share object that is based on the anonymous block build file. The method includes deploying the share object to a customer account, which enables the customer account to access the view of the customer usage data.

Abstract: A cloud data platform determines a set of candidate pre-provisioning times for compute resources to be provisioned from a cloud provider service to a free pool maintained by the cloud platform, the set of candidate pre-provisioning times being based on a demand of compute resources (N) and a target time. For each candidate pre-provisioning time: the cloud data platform finds similar historical periods based on historical distribution of provisioning latencies. For each historical period of the similar historical periods, the cloud data platform determines a minimum number of compute resources (M) to provision at the candidate pre-provisioning time so that at least N compute resources are ready by the target time within a specified confidence.

Abstract: Provided herein are systems and methods for configuring automatic evolution of dynamic tables. An example method includes parsing, by at least one hardware processor, a query associated with a dynamic table to determine a current base object dependency of the dynamic table on at least a first base object. A prior base object dependency of the dynamic table on at least a second base object is retrieved. A delta between data stored by the at least first base object and data stored by the at least second base object is determined. The dynamic table is updated based on the delta.

Abstract: A top K query directed at a table is received. The table is organized into multiple storage units. The top K query comprises a first clause to sort a result set in order and a second clause that specifies a limit on a number of results provided in response to the query. A table scan operator identifies a first set of rows from the table based on a scan set determined for the table and provides the first set of rows to a top K operator. The top K operator determines a current boundary based on the first set of rows and provides the current boundary to the table scan operator. The table scan operator prunes the scan set based on the current boundary and identifies a second set of rows from the table based on the pruning.

Abstract: Various example embodiments described herein provide for systems, methods, devices, instructions, and like for translation to semantic data for artificial intelligence (AI) agent-based semantic data updates (e.g., improvements) using one or more data sources, such as golden data sources, where the updated semantic data (e.g., semantic model) that results can be used by a natural language query system to translate one or more natural language questions into one or more structure language queries (e.g., one or more structured query language (SQL) statements). Some example embodiments are implemented as a system that uses a set of AI agents working in an optimization loop to improve/refine semantic data using the one or more data sources.

Abstract: Various example embodiments described herein provide for systems, methods, devices, instructions, and the like for optimizing a query plan to execute a query by pushing down top-k information in the query plan, which can be used within a data platform (e.g., database) environment. In particular, various example embodiments enable information from a top-k operator of a query plan to be pushed down the query plan and enable pushing the information through one or more outer join operators (e.g., through multiple levels of outer join operators) of the query plan to a select operator, such as an aggregation operator, positioned below the one or more outer join operators.

Abstract: Provided herein are systems, methods, and computer-storage media for managing data skew in hash join operations. A skew manager partitions build-side row data into multiple sets corresponding to hash-join-build (HJB) instances based on hash values. The skew manager detects skew in a build-side row set associated with a first HJB instance by analyzing the number of rows. Upon detecting skew, the skew manager redirects data rows to at least a second HJB instance. The method involves configuring skew caches, generating histograms, and detecting frequent hash values to identify skew. It also includes communicating skew notifications, broadcasting probe-side row data, and adjusting partitioning of probe-side data. The disclosed techniques further include buffering build-side row sets in streams and performing join operations based on these streams, enhancing efficiency in distributed computing environments.

Abstract: A system and method for enhancing data storage security in cloud environments using multiple gateway endpoints. The system includes a cloud data platform with a plurality of networks, each associated with a different gateway endpoint. Each gateway endpoint has a respective gateway endpoint tag. The method involves creating networks and corresponding gateway endpoints, configuring each gateway endpoint to enable traffic for sessions associated with the gateway endpoint tag, and assigning execution nodes to networks with session tokens that include the gateway endpoint tag. This setup ensures that only sessions with the correct tag can access data storage, reducing the risk of unauthorized data exfiltration.

Abstract: The subject technology determines that a request to commit a transaction has been received. The subject technology prior to committing the transaction, performs a write operation to store a metadata version of a table associated with the transaction in a metastore. The subject technology performs a register operation to register a background worker. The subject technology performs, by the background worker, a trigger operation to perform an execution node job on an execution node. The subject technology retrieves, by the execution node job, a set of identifiers of active transactions. The subject technology performs, by the execution node job, a write operation to store the metadata version of the table and the set of identifiers in a datastore. The subject technology receives, by the background worker, an indication that the execution node job has successfully completed. The subject technology performs an unlock operation to unlock the table.

Abstract: The subject technology initiates a reinstallation process of a key-value storage device and locking the key-value storage device. The subject technology performs a bootstrap process for a blob manager and a blob worker. The subject technology performs a restoration process of a storage server. The subject technology applies a set of mutation logs to the storage server. The subject technology unlocks the key-value storage device and enabling network traffic for the key-value storage device.

Abstract: Various example embodiments described herein provide for systems, methods, devices, instructions, and the like for performing commit operations (e.g., write operations) with recovery on a data system, such as a database system, that is interacting with a data source external to the data system, such as an external catalog. For some example embodiments, the database system generates an operation identifier in association with each command to commit a change to an entity of the catalog linked database, where the operation identifier serves as a unique tracking mechanism for commit operations across the distributed systems.