Abstract: Embodiments of the present disclosure provide techniques for deduplicating files during internal stage replication using a directory table of the replicated internal stage that is modified as a cache for storing and retrieving original file-level metadata for the replicated files. An initial list of candidate files for loading from the internal stage to a table of the target deployment is prepared based on the files listed in the internal stage, and refined using a directory table lookup. If there is any inconsistency between the files registered in the directory table and the files listed in the internal stage, the target deployment will inspect the user-defined file-level metadata to obtain original file-level metadata for each file that is present in the internal stage but not in the directory table. This information may be used during deduplication to ensure that no duplicate files are loaded.

Abstract: Provided herein are systems and methods for replicating share objects to remote deployments. An example method includes generating, at a second deployment in a data exchange, a replica of a share object included in a first deployment of the data exchange. The share object grants privileges to a first account to access a first database of the first deployment, wherein the first database comprises a database object. The method also includes retrieving, at the first deployment, share grant metadata from the share object of the first deployment, wherein the share grant metadata includes a reference to the database object and allows the first account to use the database object. The method also includes transmitting a message comprising the share grant metadata to the replica in the second deployment. The share grant metadata in the replica allows a second account to use the database object replicated in the second deployment.

Abstract: Embodiments of the present disclosure enable users of a data sharing system to build native applications that can be shared with other users of the data sharing system. The native applications can be published and discovered in the data sharing system like any other data listing, and consumers can install them in their local data sharing system account to serve their data processing needs. A provider may define an installation script for installing an application and create a share object to which the installation script may be attached. In response to an imported database being created in a consumer account based on the share object, a native application framework may automatically execute the installation script in the consumer account and may create a set of database roles to manage execution of the application in the consumer account.

Abstract: A method implementing a fault-tolerant data warehouse using availability zones includes allocating a plurality of processing units to a data warehouse, the processing units located in different availability zones, an availability zone comprising one or more data centers. The method further includes routing a query to a processing unit within the data warehouse, the query having a common session identifier with a query previously provided to the processing unit, the processing unit determined to be caching a data segment associated with a cloud storage resource independent of the plurality of processing units. The method further includes, as a result of monitoring a number of queries running at an input degree of parallelism, determining that the processing capacity of the processing units has reached a threshold; and changing a total number of processing units using the input degree of parallelism and the number of queries.

Abstract: Example resource management systems and methods are described. In one implementation, a resource manager is to be coupled to one or more virtual data warehouse instances that each comprise a plurality of computing resources. A resource manager receives a query directed to database data. The resource manager determines a plurality of tasks associated with the query. The resource manager distributes the plurality of tasks among the one or more virtual data warehouse instances to cause each computing resource of the plurality of computing resources of the one or more virtual data warehouse instances to process a respective task of the plurality of tasks. The resource manager receives results obtained from the processing of the plurality of tasks. The resource manager uses the results to respond to the query.

Abstract: Techniques for sharing query results in a multi-tenant database system are described. The query results can be shared between users of the same account or organization in the multi-tenant network-based database system using security tokens. A first user executes a query, and the results are stored in the network-based database system. The first user can invoke a function to create a security token to provide access to the stored query results to other users in the same account. The first user can share the security token with the other users, who can directly access the stored results in the network-based database system instead of having to download local copies of the query results.

Abstract: Embodiments of the present disclosure provide a region-specific events account that is used as a central place to store the events shared by consumers of shared applications in that region. Use of such an account helps reduce the operational burden of a provider of the applications as they do not need to determine which accounts have shared events for each different consumer. The provider only needs to login to the events account which is in the same region as the consumer and can query all shared events from different applications that are being used in that region. Also, since the provider can designate an events account in each region, the shared events could be directly ingested into the events account without traveling to a different region.

Abstract: Automated query retry includes selecting a first node of a plurality of nodes of an execution platform in response to a query. The first node is configured with a first version of a database platform. A first execution of the query is performed using the first version of the database platform at the first node. The method further includes detecting the first execution results in a failed execution. A second execution of the query is scheduled. The second execution uses at least one of the first version of the database platform or a second version of the database platform configured at a second node of the plurality of nodes. The second version is released after the first version. A cause of the failed execution of the query at the first node is determined based at least in part on a result of the second execution of the query.

Abstract: Data in a micro-partition of a table is stored in a compressed form. In response to a database query on the table comprising a filter, the portion of the data on which the filter operates is decompressed, without decompressing other portions of the data. Using the filter on the decompressed portion of the data, the portions of the data that are responsive to the filter are determined and decompressed. The responsive data is returned in response to the database query. When a query is run on a table that is compressed using dictionary compression, the uncompressed data may be returned along with the dictionary look-up values. The recipient of the data may use the dictionary look-up values for memoization, reducing the amount of computation required to process the returned data.

Abstract: The subject technology receives a query plan, the query plan comprising a set of query operations, the set of query operations including at least one aggregation and at least one join operation. The subject technology analyzes the query plan to identify an aggregation that is redundant. The subject technology removes the aggregation based at least in part on the analyzing. The subject technology determines at least one aggregation property corresponding to at least one query operation of the query plan. The subject technology inserts at least one adaptive aggregation operator in the query plan based at least in part on the at least one aggregation property, the at least one aggregation property comprising a set of aggregation properties. The subject technology provides a modified query plan based at least in part on the inserted at least one adaptive aggregation operator in the query plan.

Abstract: A database platform receives an object identifier from a client in association with a database session. The client is associated with a customer account of the database platform, and the database session is associated with the client. The customer account includes multiple disjoint account-level namespaces, each of which represents a distinct context for resolution of object identifiers, such that matching object identifiers in different account-level namespaces in the customer account do not collide with respect to object-identifier resolution. The database platform determines that the object identifier does not specify an account-level namespace, and responsively resolves the object identifier with reference to a current account-level namespace of the database session by identifying an object corresponding to the object identifier in the customer account.

Abstract: A method includes decoding, by at least one hardware processor, a notification of a changed database code of a database. A query is executed responsive to the notification. The query indicates a data processing command and a data object in the database. A regression in the changed database code is detected based on multiple regression testing operations applied to the data processing command and the data object. Analysis of the regression is performed to detect a rollout parameter of a plurality of rollout parameters as a root cause of the regression. The plurality of rollout parameters are associated with the changed database code. A determination is made on whether to perform a mitigation action for the regression based on the rollout parameter.

Abstract: Techniques for triggering pipeline execution based on data change (transaction commit) are described. The pipelines can be used for data ingestion or other specified tasks. These tasks can be operational across account, organization, cloud region, and cloud provider boundaries. The tasks can be triggered by commit post-processing. Gates in the tasks can be set up to reference change data capture information. If the gate is satisfied, tasks can be executed to set up data pipelines.

Abstract: The subject technology selects a particular zone among multiple zones based on a target skew to meet a global balancing of cluster instances. The subject technology deploys a particular type of cluster instance to the particular zone. The subject technology, for each zone from the multiple zones, determines a respective number of cluster instances. The subject technology identifies a second particular type of cluster instance to add based on a total number of the second particular type of cluster instance in the multiple zones and a second total number of the particular type of cluster instance in the multiple zones. The subject technology adds the second particular type of cluster instance to a second particular zone to meet the global balancing of cluster instances in the multiple zones.

Abstract: Provided herein are systems and methods to provide a way to share metrics regarding shared data access and accesses associated with data providers for different data listings of the data exchange. For example, the method may comprise detecting one or more client interactions with a set of data listings of a data exchange, the set of data listings associated with one or data providers. The method may further comprise collecting metrics corresponding to the one or more client interactions. In addition, the method may share metrics relevant to the one or more data providers with the one or more data providers.

Abstract: Example caching systems and methods are described. In one implementation, a method identifies multiple files used to process a query and distributes each of the multiple files to a particular execution node to execute the query. Each execution node determines whether the distributed file is stored in the execution node's cache. If the execution node determines that the file is stored in the cache, it processes the query using the cached file. If the file is not stored in the cache, the execution node retrieves the file from a remote storage device, stores the file in the execution node's cache, and processes the query using the file.

Abstract: A database export system exports data using a plurality of nodes that process the data to generate structured result files that are partitioned by an export parameter in an export request. The database export system distributes the data and merges the files to avoid small file creation and increase processing speed via parallelism. The database export system generates the result files of a specified maximum size in a final format, where the files are processed merged in a temporary file format. The parallel processing is optimized and constrained per the amount of processing nodes, available memory, requested final file sizes, and operation based ordering to complete data exports in a scalable multi-stage approach.

Abstract: Embodiments of the present disclosure describe systems, methods, and computer program products for redacting sensitive data within a database. An example method can include receiving a masking policy for a column of a database, the masking policy identifying a category of sensitive data, examining a column of a database to identify a category of sensitive data in a first location of the column, and, in response to a data query accessing the column, the first location of the column exceeding a threshold probability of comprising sensitive data, executing a redaction operation to redact the category of sensitive data from the first location of the column to generate redacted data for a response to the data query.

Abstract: A method for identity resolution and data enrichment is performed by at least one hardware processor and includes detecting at an account of a data provider, a shared data object that is shared by an account of a data consumer with the account of the data provider. An application executing at the account of the data consumer is enabled for an identity resolution process based on the detecting of the shared data object. A request for source data received from the application is detected at the account of the data provider. The source data is managed by the account of the data provider. The source data is communicated to the application executing at the account of the data consumer, based on a verification that the application is enabled for the identity resolution process. The identity resolution process is performed at the account of the data consumer using the source data.

Abstract: A method includes parsing, by at least one hardware processor, a query to determine query comments and query code associated with the query. A query execution plan is generated based on the query code. Query execution using the query code is performed at a first computing node associated with a query processing pipeline. A detection is made that the query comments are indicative of a software bug in the query code based on analysis of the query comments. The detection is performed at a second computing node associated with a query analysis pipeline. A notification of the software bug and a result of the query execution is output.