Abstract: A method implementing a fault-tolerant data warehouse using availability zones includes allocating a plurality of processing units to a data warehouse, the processing units located in different availability zones, an availability zone comprising one or more data centers. The method further includes routing a query to a processing unit within the data warehouse, the query having a common session identifier with a query previously provided to the processing unit, the processing unit determined to be caching a data segment associated with a cloud storage resource independent of the plurality of processing units. The method further includes, as a result of monitoring a number of queries running at an input degree of parallelism, determining that the processing capacity of the processing units has reached a threshold; and changing a total number of processing units using the input degree of parallelism and the number of queries.

Abstract: An anti-abuse system is provided for a data-platform. An anti-abuse scanner of the data-platform detects a creation of an application package by a provider of content to the data platform where the application package includes a set of files for deployment on the data platform. The anti-abuse scanner performs a review o the set of files to detect malicious content where the review is based on a set of analysis rules and generates a deployment decision for the application package based on a result of the review.

Abstract: Techniques for database replication include a method performed by at least one hardware processor of a resource management node. The method includes detecting a request that a database instance of a database stored at a first network deployment is available at a second network deployment. Responsive to the request, the second network deployment is provisioned with a plurality of data objects. A first data object of the plurality of data objects includes a database shell. The database shell is reconfigured as the database instance of the database based on pushing data of the database into the database shell. Access to the database instance at the second network deployment is authorized based on at least a second data object of the plurality of data objects.

Abstract: Sharing data in a data exchange across multiple cloud computing platforms and/or cloud computing platform regions is described. An example computer-implemented method can include creating a listing in a data exchange, the listing including a data set hosted by a first cloud computing entity. The data set can be shared with a second cloud computing entity. The method further includes receiving a request associated with a customer account of the second cloud computing entity to access the data set of the listing hosted by the first cloud computing entity and replicating at least a subset of the data set of the listing from the first cloud computing entity to a provider account at the second cloud computing entity to be accessible by the customer account at the second cloud computing entity.

Abstract: Systems, methods, and devices for retrying a query. A method includes receiving a query directed to database data and assigning execution of the query to one or more execution nodes of an execution platform, the one or more execution nodes configured to execute the query on a first version of a database platform. The method includes determining that execution of the query was unsuccessful. The method includes assigning a first retry execution of the query to the one or more execution nodes of the execution platform and determining whether a regression or an intermittent fault caused the execution of the query to be unsuccessful based at least in part on whether the first retry execution of the query was successful or unsuccessful.

Abstract: Systems, methods, and devices for batch ingestion of data into a table of a database. A method includes determining a notification indicating a presence of a user file received from a client account to be ingested into a database. The method includes identifying data in the user file and identifying a target table of the database to receive the data in the user file. The method includes generating an ingest task indicating the data and the target table. The method includes assigning the ingest task to an execution node of an execution platform, wherein the execution platform comprises a plurality of execution nodes operating independent of a plurality of shared storage devices collectively storing database data. The method includes registering metadata concerning the target table in a metadata store after the data has been fully committed to the target table by the execution node.

Abstract: Example resource management systems and methods are described. In one implementation, a system includes a memory and a processing device operatively coupled to the memory. The processing device is to receive a query referencing database data stored in a storage platform, determine a task associated with processing the received query, and create an execution node comprising cache resources and processing resources. Furthermore, a size of the cache resources of the execution node is determined upon creation of the execution node, based at least in part on the task, and processing resources of the execution node are determined upon creation of the execution node, based at least in part on the task. The execution node is included within a plurality of execution nodes to process the task associated with processing the received query.

Abstract: Techniques for continuous ingestion of files using custom file formats are described. A custom file format may include formats not natively supported by a data system. Unstructured files (e.g., images) may also be considered custom file formats. A custom file format may be set using a user defined table function and scanner options.

Abstract: Example resource management systems and methods are described. In one implementation, a resource manager is configured to manage data processing tasks associated with multiple data elements. An execution platform is coupled to the resource manager and includes multiple execution nodes configured to store data retrieved from multiple remote storage devices. Each execution node includes a cache and a processor, where the cache and processor are independent of the remote storage devices. A metadata manager is configured to access metadata associated with at least a portion of the multiple data elements.

Abstract: A system and method for pruning data based on metadata. The method may include receiving a query that includes a plurality of predicates and identifying one or more applicable files including database data satisfying at least one of the plurality of predicates. The identifying the one or more applicable files including reading metadata stored in a metadata store that is separate from the database data. The method further includes pruning inapplicable files including database data that does not satisfy at least one of the plurality of predicates to create a reduced set of files and reading the reduced set of files to execute the query.

Abstract: Techniques for active invalidation of cache entries are described. A first timestamp and a second timestamp are stored by a compute instance, the first timestamp corresponding to an initial time the computing instance read from a version update log in a database and the second timestamp corresponding to a last time the computing instance read from the version update log in the database. The compute instance can read records from the version update log that are associated with version timestamps greater than the second timestamp. The compute instance can invalidate a first set of cache entries with older versions stored in a cache based on reading the records, retaining a second set of cache entries, and then execute a query using the second set of cache entries.

Abstract: Systems, methods, and devices for sharing materialized views in multiple tenant database systems. A method includes defining a materialized view over a source table that is associated with a first account of a multiple tenant database. The method includes defining cross-account access rights to the materialized view to a second account such that that second account can read the materialized view without copying the materialized view. The method includes modifying the source table for the materialized view. The method includes identifying whether the materialized view is stale with respect to the source table by merging the materialized view and the source table.

Abstract: Interleaved infrastructure can interleave fragment execution during query compilation. For example, compilation can be paused, and a plan fragment is executed. The results of the plan fragment can be fed back into the compiler, which can then restart the compilation and modify the compilation based on the results of the plan fragment execution. The interleaved workflow provides a flexible, robust infrastructure, which can enable and improve a variety of database features and query optimizations.

Abstract: A method implementing a fault-tolerant data warehouse including allocating a plurality of processing units to a data warehouse, the processing units located in different availability zones, an availability zone comprising one or more data centers. The method further includes, as a result of monitoring a number of queries running at an input degree of parallelism on the plurality of processing units of the data warehouse, determining that the number of queries is serviceable by one fewer processing units. The method further includes routing a query from a first processing unit to a second processing unit within the data warehouse, the query having a common session identifier with another query previously provided to the second processing unit, the second processing unit determined to be caching a data segment associated with a cloud storage resource, usable by the query, and removing the first processing unit from the data warehouse.

Abstract: Systems, methods, and devices for tracking a series of changes to database data are disclosed. A method includes executing a transaction to modify data in a micro-partition of a table of a database by generating a new micro-partition that embodies the transaction. The method includes associating transaction data with the new micro-partition, wherein the transaction data comprises a timestamp when the transaction was fully executed, and further includes associating modification data with the new micro-partition that comprises an indication of one or more rows of the table that were modified by the transaction. The method includes joining the transaction data with the modification data to generate joined data and querying the joined data to determine a listing of intermediate modifications made to the table between a first timestamp and a second timestamp.

Abstract: Techniques for configuring a managed event table (MET) include detecting, by at least one hardware processor, a query for the MET. The query received at a first computing node of a network-based database system. The method includes retrieving via an ingestion function configured at the first computing node, reformatted data from a dedicated storage location of a first process into the MET. The reformatted data is based on log data associated with a second process. The first process and the second process are executing at a second computing node of the network-based database system. The method includes processing the query using the reformatted data in the MET.

Abstract: A method for a first set of processors and a second set of processors comprises, the first set of processors processing a set of queries, as a result of a change in utilization of the first set of processors, processing the set of queries using the second set of processors. The change in processors is independent of a change in storage resources, the storage resources shared by the first set of processors and the second set of processors.

Abstract: A database system can configure network devices, such as a primary database in a multi-tenant deployment and a secondary database in a private deployment, to send and receive sequence messages, such as input data indicative of a selection of a link. The database system can create a secure share area in the private deployment in response to receiving the input data indicative of the selection of the link. The database system can replicate the data from the multi-tenant deployment to the secure share area in the private deployment and share the replicated data from the secure share area to the secondary database hosted in the private deployment.

Abstract: A method and apparatus for managing a set of storage resources for a set of queries is described. In an exemplary embodiment, a method provisions processing resources of an execution platform and provisions storage resources of a storage platform. The execution platform uses the storage platform, which is shared with the execution platform, to process the set of queries. The method changes a number of the storage resources provisioned for the storage platform based on a storage capacity utilization by the set of queries of the storage resources. The method changes the number of the storage resources independently to a change of the processing resources in the execution platform. The method processes the set of queries using the changed number of the storage resources provisioned for the storage platform.

Abstract: System and method for efficiently querying external tables are described herein. In an embodiment, a database platform receives a query that is directed at least in part to external data in an external table stored on a data storage platform that is external to the database platform. The external table includes a plurality of partitions. The database platform identifies, from external-table metadata, a subset of the plurality of partitions of the external table as including data that potentially satisfies the query. The external-table metadata is stored by the database platform. The database platform identifies data that satisfies the query by scanning the identified subset of the partitions, and responds to the query at least in part with the identified data that satisfies the query.