Abstract: Various example embodiments described herein provide for systems, methods, devices, instructions, and the like for using AI model cascades to filter data on a data system, such as a database system, based on an artificial intelligence model prompt (e.g., user-provided prompt). In particular, various example embodiments enable a database system to use cascaded AI models and adaptive bounds to optimize data filtering operations based on an artificial intelligence model prompt (also referred to herein as just a prompt) while balancing computational cost and accuracy, which can be useful in processing large-scale data queries.

Abstract: A data platform for executing containers is provided. In some examples, the data platform receives an application from an application package of a provider account, the application including a setup script and a manifest of a service. The data platform activates access roles based on the manifest and creates the service and a compute pool using the setup script and a specification file accessed from the application package using an access role. The service is executed in the compute pool, accessing objects of the application package and of the data platform using the access roles.

Abstract: The subject technology receives first party training data provided by an end-user of a baseline machine learning model. The subject technology determines a first set of common features based on the first party training data. The subject technology receives, from at least one data source. The subject technology determines a second set of common features based on the set of datasets. The subject technology trains, using the first set of common features and the second set of common features, a second machine learning model, the second machine learning model incorporating additional training data from the external data supplier during training compared to the baseline machine learning model. The subject technology generates a boosted machine learning model based at least in part on the training, the boosted machine learning model comprising the trained second machine learning model.

Abstract: Systems and methods are provided for generating personalized service disruption notifications. The system allocates resources of a database system to a plurality of entities, the resources of the database system being distributed in a cloud environment and analyzes a plurality of signals on the database system. The system, in response to analyzing the plurality of signals, detects a likelihood of a service availability disruption on the database system for a first entity of the plurality of entities. The system notifies the first entity of the service availability disruption in response to detecting the likelihood of the service availability disruption.

Abstract: An advanced system for refining overlap queries in a database system based on user feedback. The system monitors interactions of a first user with a first dataset on the database system, where the first dataset is associated with the first user. Feedback regarding the quality of a results dataset, generated from an executed overlap query, is received from the first user. This feedback informs the generation of a similarity score dataset that enhances the creation of new overlap queries. These new overlap queries are designed to output refined overlap datasets between the first dataset and a second dataset associated with a second user. A new joined dataset is generated by executing these overlap queries, comprising data from both the first and second datasets. A new results dataset is generated, providing the first user with refined recommendations based on additional feedback.

Abstract: Example caching systems and methods are described. In one implementation, a method receives a set of queries to be processed by a set of virtual warehouses. The method distributes the set of queries to the set of virtual warehouses to be executed and creates, during the processing of the set of queries by the set of virtual warehouses, a new virtual warehouse, wherein cache resources associated with the new virtual warehouse are populated with data files associated with the set of queries at the time the virtual warehouse is created and the cache resources vary among the processors, wherein a first subset of the processors comprises minimal cache resources and a second subset of processors comprises cache resources providing faster input-output operations. The method redistributes the set of queries across the set of virtual warehouses.

Abstract: Example caching systems and methods are described. In one implementation, a method receives a query, at an execution platform, directed to data stored across a plurality of shared storage devices, the execution platform comprising one or more execution nodes, an execution node comprising a plurality of processors. The method processes the query using the one or more execution nodes of the execution platform, and in response to a determination of a change in a number of execution nodes of the execution platform, wherein the change is creating a new execution node, wherein a first subset of the plurality of processors comprises a minimal cache and a second subset of the plurality of processors comprises a cache providing faster input-output operations, reassigns processing of the query, among the changed number of execution nodes of the execution platform.

Abstract: A computer-implemented method includes accessing virtual private cloud flow logs of network traffic data originating from a virtual private cloud, generating filtered flow logs by filtering the virtual private cloud flow logs, extracting features based on a plurality of attributes from the filtered flow logs, training one or more machine learning models based on the features, applying the one or more machine learning models to the network traffic data to identify potential beacons, generating an alert notification that identifies the potential beacons, and communicating the alert notification to an alerting system.

Abstract: Embodiments of the present disclosure provide techniques for efficient computation over a wide table. A processing device determines that a first number of columns of a first table is greater than a threshold number of columns. The processing device transforms the first table into a second table based on the determination, where the second table includes a second number of columns that is less than the first number of columns, and where the second table includes a first column that includes first fields that identify columns of the first table, a second column that includes second fields that identify data types of fields of the first table, and a third column that includes third fields that include data of the fields of the first table. The processing device executes a UDTF on the second table.

Abstract: The subject technology receives, by a first set of decoupled compute instances of a distributed database, a plurality of transactional queries against a transactional processing database stored on a first set of decoupled storage instances, at least one of the first set of decoupled compute instances bypassing a compiler of the distributed database to process the plurality of transactional queries using locally cached data. The subject technology receives at least one select query for processing on an aggregated analytical query database of the distributed database. The subject technology generates, by the compiler of the distributed database, a query plan for execution of the at least one select query using a second set of decoupled compute instances. The subject technology generates, by the second set of decoupled compute instances, an aggregated analytical query result dataset for the at least one select query according to the query plan.

Abstract: A system is disclosed comprising a memory containing instructions and one or more computer processors. When the instructions are executed, the system performs an operation to configure a Domain Name System (DNS) proxy, executing in a node of a cloud data platform associated with a first account, to perform hostname resolution of an Account Host Identifier (AHID) of the first account. The DNS proxy receives a DNS request from a process executing in a pod of the node, and the system fails to resolve the DNS request if the name in the DNS request differs from the AHID of the first account. The system returns an Internet Protocol (IP) address if the name in the DNS request matches the AHID. The process executing in the pod of the node is configured to send data to data storage of the cloud data platform using the returned IP address.

Abstract: The subject technology receives a first request to create a container service, the request indicating a service specification for creating the container service. The subject technology generates a set of endpoints based on the service specification. The subject technology generates a set of roles based on the service specification. The subject technology stores service metadata related to the set of endpoints and the set of roles in a metadata database. The subject technology instantiates the container service at a container services cluster, the container services cluster including a set of worker nodes, the container service being deployed on a worker node from the set of worker nodes, and enforces security policies based on the roles and service metadata. The subject technology coordinates with Role Based Access Control (RBAC) and network policies of the subject database system and transparently enforces the same policies over in the subject container system.

Abstract: The described system aims to reduce or eliminate inaccuracies and hallucinations in responses generated by a machine learning model when processing user queries. The data platform parses and categorizes the text within data files to create structured textual representations. The user submits multiple prompts which are collectively assessed to refine and modify the initial queries. The modified query is used to identifying segments of data files that are most relevant to the query. These relevant portions are then compiled into a Retrieval-Augmented Generation (RAG) context block. This RAG context block is fed into a prompt response machine learning model, which processes the enriched information to generate a well-informed and accurate response to the user's query. Finally, this response is displayed back to the user through the chat interface, completing a cycle that enhances the reliability and relevance of machine-generated answers.

Abstract: A system and method for optimizing data ingestion through clustering during the copy process. The system detects a data processing pipeline definition with a COPY command and determines storage location and destination table parameters. Upon detecting a COPY command, the system parses the command syntax to identify storage location and destination table parameters. The system monitors for available file batches at the storage location and executes the COPY command to transform these batches into formatted files compatible with the destination table format. The system then performs clustering operations on the formatted files, organizing data according to pre-defined clustering keys, before ingesting the clustered files into the destination table.

Abstract: A data platform having an anti-abuse analysis pipeline is provided. The anti-abuse analysis pipeline detects an image referenced in an application package and schedules an application scan to scan application source files and generate application scan results. The anti-abuse analysis pipeline also schedules an image scan to scan the image and generate image scan results. The anti-abuse analysis pipeline extracts application source files from the application package and executes an application scan using the extracted application source files to generate application scan results. The anti-abuse analysis pipeline extracts artifacts from the referenced image and executes an image scan using the artifacts to generate image scan results. The anti-abuse analysis pipeline determines a completion of the application scan and the image scan and generates a scan result using the application scan results and image scan results.

Abstract: A search engine of a data exchange may receive a query comprising a set of search terms, retrieve a plurality of data listings based on the search terms of the query, compare a first embedding generated by a large language model (LLM) from the search query to second embeddings generated by the LLM for each of the plurality of data listings to determine a respective relevance for each of the plurality of data listings to the search query, and rank the plurality of data listings based on the respective relevance for each of the plurality of data listings to the search query.

Abstract: Various embodiments described herein provide for systems, methods, devices, instructions, and like for generating synthetic data. According to various embodiments, synthetic data generation comprises receiving input specifying one or more source tables and join key columns, and generating synthetic data that preserves statistical similarity and referential integrity among columns of the source data.

Abstract: The subject technology receives a query plan, the query plan comprising a set of query operations, the set of query operations including at least one aggregation and at least one join operation. The subject technology analyzes the query plan to identify an aggregation that is redundant. The subject technology removes the aggregation based at least in part on the analyzing. The subject technology determines at least one aggregation property corresponding to at least one query operation of the query plan. The subject technology inserts at least one adaptive aggregation operator in the query plan based at least in part on the at least one aggregation property, the at least one aggregation property comprising a set of aggregation properties. The subject technology provides a modified query plan based at least in part on the inserted at least one adaptive aggregation operator in the query plan.

Abstract: This disclosure provides methods and techniques of data replication involving cross replication group (RG) references. Example methods, systems, and techniques are disclosed regarding batch database replication (e.g., backup) and failover (e.g., automatic transition to a backup) between multiple database deployments or database providers. For example, a system causes database data to be stored in a primary deployment and replicated in one or more secondary deployments. In the event that data in the primary deployment is unavailable, transactions may be executed on one or more of the secondary deployments. When the original primary deployment becomes available again, any transactions executed on secondary deployments may be propagated to the primary deployment. The system may be configured such that queries on the database data are executed on the primary deployment at any time when the primary deployment is available.

Abstract: The subject technology receives a query, the query including a statement for performing the query. The subject technology performs a first lookup operation on a multi-phase cache based on the query. The subject technology performs, in response to a first cache miss of the multi-phase cache, parsing of the statement from the query. The subject technology performs, based on the parsing, a compilation process on the query to generate a compiled query plan, the compilation process determining an optimization and a generalization for the query. The subject technology determines that the compiled query plan is cacheable. The subject technology registers, in response to the compiled query plan being cacheable, a dummy entry in the multi-phase cache.