Abstract: As described herein, a N-Gram index may be created and the search may be conducted using the index, which will lead to faster search results. The N-Gram index may also include partial N-Gram components to capture more relevant data. Moreover, as described herein, the search may also take into account recent log data that has not yet been indexed. Techniques for building an index store using log data and efficiently searching the index store and log data to process search requests are described herein.

Abstract: Techniques for configuring query result information associated with query result data include configuring execution of a query using a plurality of execution stages. Query result data is generated based on the execution of the query. The query result data includes a plurality of data portions. Each data portion of the plurality of data portions is obtained at a corresponding execution stage of the plurality of execution stages. Query result information is generated based on the execution of the query. The query result information includes authentication information associated with accessing the plurality of data portions.

Abstract: Systems and methods for managing membership in a private data exchange are provided herein. In one embodiment, the method includes generating a plurality of listings in a data exchange. A first listing of the plurality of listings may include a reference to a first version of shared data within a first database, where the first version is one of a plurality of versions of the shared data within the first database. The method further includes receiving a request from a member of the data exchange for access to the first listing and limiting, by a processing device, access for the member to the first version of the shared data of the first database referenced by the first listing.

Abstract: The subject technology receives first metadata corresponding to a set of micro-partitions. The subject technology stores a first data structure and a second data structure in storage as a first file and a second file, first data structure including the first metadata and a second data structure including second metadata, the first metadata corresponding to a set of micro-partitions, the second metadata for a grouping of the first metadata, the second data structure including information associating the second metadata to the first metadata. The subject technology stores third metadata for a table, the third metadata comprising information about data stored in a micro-partition of the table.

Abstract: Generating semantic names for a data set is described. An example method can include retrieving data from a data set, the data organized in a plurality of columns. The method may also include generating one or more candidate semantic categories for that column, wherein each of the one or more candidate semantic categories has a corresponding probability for each of the columns. The method may also further include creating a feature vector for each column from the one or more column candidate semantic categories and the corresponding probabilities. Additionally, the method may also include selecting, for each column, a column semantic category from the one or more candidate semantic categories using at least the feature vector and a trained machine learning model.

Abstract: Replication and failover of database data is disclosed. A method includes copying a first dataset of a first account stored in a primary deployment to a secondary deployment. The method includes determining that the first dataset stored in the primary deployment is unavailable. The method includes executing a first transaction on the first dataset at the secondary deployment in response to determining that the first dataset stored in the primary deployment is unavailable. The method includes executing a second transaction on a second dataset of a second account at the primary deployment while the first dataset of the first account stored in the primary deployment is unavailable.

Abstract: A method for a multi-cluster warehouse includes allocating a plurality of compute clusters as part of a virtual warehouse. The compute clusters are used to access and perform queries against one or more databases in one or more cloud storage resources. The method includes providing queries for the virtual warehouse to each of the plurality of compute clusters. Each of the plurality of compute clusters of the virtual warehouse receives a plurality of queries so that the computing load is spread across the different clusters. The method also includes dynamically adding compute clusters to and removing compute clusters from the virtual warehouse as needed based on a workload of the plurality of compute clusters.

Abstract: Signing source code using a vault device is described. An example method can include receiving, with a client device, source code that is to be committed to a repository. The method further can include sending, with a processing device to a vault device, a request to sign the source code. The method can further include receiving, from the vault device, a signed commit of the source code. In addition, the method can include pushing the source code and the signed commit to the repository, wherein the repository commits the source code and signed commit.

Abstract: A system and method for automatically routing datasets based on input and output requirements of information is disclosed. The example method includes obtaining a list of modules executing on one or more processing devices, each module to generate an output dataset of a respective output requirement based on an input dataset of a respective input requirement. The method also includes establishing connections between the modules based on the input requirements and output requirements, including a first connection between a first module and a third module and a second connection between a second module and the third module. The method also includes receiving a first output dataset from the first module and receiving a second output dataset from the second module and, responsive to determining that the first module reported an error, routing the second output dataset to the third module without routing the first output dataset to the third module.

Abstract: Techniques for schema mismatch detection and evolution are described. When data is being uploaded into a source table, schema of the data to be uploaded can be compared with the schema for the source table. If a schema mismatch is detected, the schema of the source table can be modified, and the upload can be continued without data loss.

Abstract: A method for a multi-cluster warehouse includes allocating a plurality of compute clusters as part of a virtual warehouse. The compute clusters are used to access and perform queries against one or more databases in one or more cloud storage resources. The method includes providing queries for the virtual warehouse to each of the plurality of compute clusters. Each of the plurality of compute clusters of the virtual warehouse receives a plurality of queries so that the computing load is spread across the different clusters. The method also includes dynamically adding compute clusters to and removing compute clusters from the virtual warehouse as needed based on a workload of the plurality of compute clusters.

Abstract: A request to replicate a first account maintained by a data platform is received. Based on the request, account data associated with the account is accessed. The account data comprises security configurations for the first account. In response to the request, the first account is replicated using the account data. A second account results from replicating the first account. The replicating of the first account comprises automatically replicating the security configurations for the first account to the second account. The replicating of the security configurations comprises replicating an identity management configuration of the first account; replicating an authorization configuration of the first account; and replicating an authentication configuration of the first account.

Abstract: Embodiments of the present disclosure may provide a database optimization system that can generate computational values through a database compiler and assignment data for execution of a query by a plurality of nodes of a database system. The computational values and assignment data can be generated by one or more machine learning schemes. The machine learning schemes can be trained on previous computational values and previous assignment data.

Abstract: A proxy revocation service provides a reliable service for performing revocation checks. The proxy revocation service queries public certificate authorities for the revocation status of a set of digital certificates and maintains a database of the revocation statuses. The proxy revocation service provides a singular endpoint that is Application Protocol Interface (API) accessible to web clients. Web clients communicate with the proxy revocation service through use of API message to perform revocation checks, rather than communicating with the public certificate authorities using an online certificate status protocol (OCSP). Use of the proxy revocation service provides both a reliable service for performing revocation checks as well as shifts the complexity away from the web clients.

Abstract: A method, apparatus, and system for join operations of a plurality of relations that are distributed over a plurality of storage locations over a network of computing components.

Abstract: Described herein are techniques for replicating external stages between deployments of e.g., a cloud data lake using a modified storage integration. The modified storage integration may be defined with multiple storage locations that it can point to, as well as a designation of an active storage location. The storage integration may also be defined with base file paths for each storage location as well as a relative file path which together may serve to synchronize data loading operations between deployments when e.g., a fail-over occurs from one deployment to another. The storage integration may be replicated from a first deployment to a second deployment, and when database replication occurs, an external stage may be replicated to the second deployment and bound to the replicated storage integration. Thus, a fail-over to the second deployment may result in a seamless transition of data loading processes to the second deployment.

Abstract: Techniques for triggering pipeline execution based on data change (transaction commit) are described. The pipelines can be used for data ingestion or other specified tasks. These tasks can be operational across account, organization, cloud region, and cloud provider boundaries. The tasks can be triggered by commit post-processing. Gates in the tasks can be set up to reference change data capture information. If the gate is satisfied, tasks can be executed to set up data pipelines.

Abstract: Disclosed herein are methods and systems for secure data comparison using data clean rooms. In an embodiment, a computer system generates a replica database based on a provider database, which stores a cross reference table that cross references a client dataset of a client database and a provider dataset of the provider database. The system receives, at the replica database, a table that is generated by the client database using the cross-reference table. The system transmits, from the replica database, the table to the provider database. The system receives, at the replica database, a results dataset that is generated by the provider database by applying a database statement to the provider database using the table generated by the client database. The system shares, from the replica database, the results dataset with the client database.

Abstract: Systems, methods, and devices for tracking a series of changes to database data are disclosed. A method includes executing a transaction to modify data in a micro-partition of a table of a database by generating a new micro-partition that embodies the transaction. The method includes associating transaction data with the new micro-partition, wherein the transaction data comprises a timestamp when the transaction was fully executed, and further includes associating modification data with the new micro-partition that comprises an indication of one or more rows of the table that were modified by the transaction. The method includes joining the transaction data with the modification data to generate joined data and querying the joined data to determine a listing of intermediate modifications made to the table between a first timestamp and a second timestamp.

Abstract: The subject technology receives a set of files corresponding to a library, the library comprising a set of functions included in the set of files. The subject technology parses the set of files. The subject technology identifies a set of functions in the set of files based on the parsing. The subject technology, for each function, registers the function as a user defined function (UDF) based on a set of input parameters utilized by the function and a type of parameter of each of the input parameters. The subject technology provides access to each registered function in a different application.