Abstract: A system and method for pruning data based on metadata. The method may include receiving a query that includes a plurality of predicates and identifying one or more applicable files including database data satisfying at least one of the plurality of predicates. The identifying the one or more applicable files including reading metadata stored in a metadata store that is separate from the database data. The method further includes pruning inapplicable files including database data that does not satisfy at least one of the plurality of predicates to create a reduced set of files and reading the reduced set of files to execute the query.

Abstract: Example resource management systems and methods are described. In one implementation, a resource manager is configured to manage data processing tasks associated with multiple data elements. An execution platform is coupled to the resource manager and includes multiple execution nodes configured to store data retrieved from multiple remote storage devices. Each execution node includes a cache and a processor, where the cache and processor are independent of the remote storage devices. A metadata manager is configured to access metadata associated with at least a portion of the multiple data elements.

Abstract: Embodiments of the present disclosure describe systems, methods, and computer program products for redacting sensitive data within a database. An example method can include receiving a data query referencing unredacted data of a database, wherein the data query that is received comprises a value identifying a type of sensitive data to be redacted from the unredacted data, responsive to the data query, executing, by a processing device, a redaction operation to identify sensitive data that matches the type within the unredacted data of the database, and returning a redacted data set in which the sensitive data that matches the type is replaced or removed to the data query.

Abstract: The subject technology receives a query, the query including a statement for performing the query. The subject technology performs a lookup operation on a stored plan cache based on the query. The subject technology performs, in response to a cache match of the query to a stored query plan in the stored plan cache based on the lookup operation, a validation process of the stored query plan. The subject technology determines whether the stored query plan is valid based on the validation process. The subject technology performs, in response to determining that the stored query plan is valid, a program building process for the stored query plan to generate a final query plan. The subject technology sends the final query plan to an execution node for execution.

Abstract: The subject technology receives a query. The subject technology performs a lookup operation on a shadow cache. The subject technology performs a validation process on the particular query plan for determining a cache hit on the shadow cache. The subject technology performs a program building process to update the particular query plan with additional information for executing the particular query plan. The subject technology performs a compilation process to compile an updated program of the updated particular query plan. The subject technology determines that the updated particular query plan is cacheable based at least in part on the compilation process. The subject technology registers the updated particular query plan as a new entry in the shadow cache. The subject technology sends the updated program to an execution node for executing the updated particular query plan.

Abstract: Disclosed herein are systems and methods for query processing with restrictions in a database clean room. In an embodiment, a system receives a query directed to a combination of a first source dataset from a first database account of a distributed database and a second source dataset from a second database account of the distributed database. The system generates an approved statements table that contains database statement language that can be executed against the combination of the first and second source datasets. Based on determining that the approved statements table includes the query, the system executes the query to produce results data, and stores the results data in the first database account.

Abstract: A method provides deterministic row ordering for query result sets in a database system. A framework receives a query referencing a database table and generating a row index column in response. The framework generates the row index column that involves determining an order associated with source files underlying the referenced table and assigns each table row a unique index value based on the file order and row position within the files. The framework executes the received query to produce a result set ordered by the generated row index column. The framework determines row order based on analyzing underlying data file metadata and facilitates returning the result set with consistent, reliable ordering, enabling stable positional-based access semantics when querying distributed database tables.

Abstract: Systems, methods, and devices for sharing materialized views in multiple tenant database systems. A method includes defining a materialized view over a source table that is associated with a first account of a multiple tenant database. The method includes defining cross-account access rights to the materialized view to a second account such that that second account can read the materialized view without copying the materialized view. The method includes modifying the source table for the materialized view. The method includes identifying whether the materialized view is stale with respect to the source table by merging the materialized view and the source table.

Abstract: A method implementing a fault-tolerant data warehouse including allocating a plurality of processing units to a data warehouse, the processing units located in different availability zones, an availability zone comprising one or more data centers. The method further includes, as a result of monitoring a number of queries running at an input degree of parallelism on the plurality of processing units of the data warehouse, determining that the number of queries is serviceable by one fewer processing units. The method further includes routing a query from a first processing unit to a second processing unit within the data warehouse, the query having a common session identifier with another query previously provided to the second processing unit, the second processing unit determined to be caching a data segment associated with a cloud storage resource, usable by the query, and removing the first processing unit from the data warehouse.

Abstract: Systems, methods, and devices for tracking a series of changes to database data are disclosed. A method includes executing a transaction to modify data in a micro-partition of a table of a database by generating a new micro-partition that embodies the transaction. The method includes associating transaction data with the new micro-partition, wherein the transaction data comprises a timestamp when the transaction was fully executed, and further includes associating modification data with the new micro-partition that comprises an indication of one or more rows of the table that were modified by the transaction. The method includes joining the transaction data with the modification data to generate joined data and querying the joined data to determine a listing of intermediate modifications made to the table between a first timestamp and a second timestamp.

Abstract: A system may include a processing device and a memory storing instructions that, when executed by the processing device, causes the processing device to obtain a health check instruction that is specific to a name of a service that is associated with one or more endpoints, including performing a lookup with the name to obtain the health check instruction that is specific to the name. The processing device performs the one or more actions of the health check instruction to determine a health status of the one or more endpoints, and stores the health status of the one or more endpoints. In response to receiving a request to resolve the name from a client, the processing device returns the one or more endpoints based at least on the health status of the one or more endpoints.

Abstract: Methods of referencing row access policy (RAP) protected mapping tables in a RAP for a data table are disclosed herein. An example method of referencing a mapping table in a data table using nested RAP includes defining, by a processing device, a first access policy for the mapping table to control access by specific users or under specific conditions. The processing device further defines a second access policy attached to the data table referencing the mapping table. The processing device in response to a query, executes the second access policy of the data table to provide a response or operation of data associated with the data table and the mapping table. Executing the second access policy invokes executing the first access policy of the mapping table. The executing of both the second access policy of the data table and the first access policy of the mapping table are recorded.

Abstract: Techniques for fetching query result data using result batches includes retrieving query result information from a first computing node of a database system. The query result information includes result data associated with a multi-stage execution of a query. A plurality of result batches are generated based on the query result information. Each result batch of the plurality of result batches includes a retrieval function for a portion of the result data. The portion of the result data for each result batch of the plurality of result batches is retrieved based on execution of the retrieval function by at least a second computing node of the database system. The result data associated with the multi-stage execution of the query is output based on completion of the retrieval.

Abstract: Embodiments of the present disclosure provide a region-specific events account that is used as a central place to store the events shared by consumers of shared applications in that region. For each of a set of provider accounts in a first region, a global account object corresponding to the provider account is replicated to a second region to establish a replicated global account object in the second region. An account data persistence object (DPO) of a first provider account of the set of provider accounts is modified to designate the first provider account as an events account. The events account is a central location within the second region where execution information generated by execution of applications shared with consumer accounts in the second region is stored. The replicated global account object corresponding to the first provider account is updated to designate the first provider account as the events account.

Abstract: Provided herein are systems and methods for network intrusion event response and remediation. A file encryption key is generated based on a first private key of a user and a root key associated with a primary deployment of a database system. File data is encoded at a source account of the user using the file encryption key. The file data is replicated from the source account into a target account of the user to generate replicated file data. Notification of a detected network intrusion event is encoded for transmission to the user. A failover of the source account to the target account is performed based on the notification. The failover grants the user access to the replicated file data based at least on a second private key of the user.

Abstract: Provided herein are systems and methods for configuring integrity constraints (including a check constraint) and row violation logging using error tables. An example method includes decoding a query received at a network-based database system. The query includes a command to perform an operation on a base table. An integrity constraint associated with the base table is retrieved. The integrity constraint specifies a desired configuration for the base table. A verification of the integrity constraint is performed to detect erroneous data of the base table that violates the desired configuration. The erroneous data is input into an error table that is configured as a nested object of the base table. A notification that the erroneous data is available in the error table is generated and output.

Abstract: A data platform for managing an application as a first-class database object. The data object can include User Interface (UI) components. The data application can be shared by a provider account to a plurality of consumer accounts using a share object and based on grant commands. The consumer accounts can deploy and operate the UI component based on the share object.

Abstract: This disclosure provides methods and techniques of performing source network address translation (SNAT) at a packet sender (e.g., a client device) instead of a gateway device (e.g., a proxy device). The present disclosure performs a SNAT operation at the packet sender, relieving the gateway device from the SNAT operation to perform other duties (e.g., policy enforcement). An example method of network address translation includes modifying, by a processing device at a data packet origination device (e.g., a client device), network address information in an internet protocol (IP) header of a packet using a public IP address. The method further includes sending the packet to a public network based on the public IP address.

Abstract: External functions can provide users of a database system the ability to invoke external services while operating from inside the database system. The techniques described herein address may detect throttling by the external system and automatically adjust or modify a request rate accordingly. Also, the techniques described herein may increase the request rate to optimize the use of the resources provided by the external system. These techniques allow the database system to improve efficiency when working with external services, which may or may not be elastic or scalable. Moreover, these techniques can decrease query execution times and reduce costs.

Abstract: A method for a first set of processors and a second set of processors comprises, the first set of processors processing a set of queries, as a result of a change in utilization of the first set of processors, processing the set of queries using the second set of processors. The change in processors is independent of a change in storage resources, the storage resources shared by the first set of processors and the second set of processors.