Abstract: Techniques for continuous ingestion of files using custom file formats are described. A custom file format may include formats not natively supported by a data system. Unstructured files (e.g., images) may also be considered custom file formats. A custom file format may be set using a user defined table function and scanner options.

Abstract: Interleaved infrastructure can interleave fragment execution during query compilation. For example, compilation can be paused, and a plan fragment is executed. The results of the plan fragment can be fed back into the compiler, which can then restart the compilation and modify the compilation based on the results of the plan fragment execution. The interleaved workflow provides a flexible, robust infrastructure, which can enable and improve a variety of database features and query optimizations.

Abstract: A cluster is provided to execute at least one containerized service in a cloud data platform. One or more monitor agents deployed on a plurality of worker nodes in the cluster collect a set of metrics associated with the at least one containerized service. A controller in the cluster aggregates the set of metrics according to at least one aggregation type defined in one or more autoscaling policies associated with the at least one containerized service and evaluates the aggregated set of metrics against criteria based on the one or more autoscaling policies to generate one or more autoscaling decisions.

Abstract: The subject technology receives a query directed to a set of source tables, each source table organized into a set of micro-partitions. The subject technology determines a set of metadata, the set of metadata comprising table metadata, query metadata, and historical data related to the query. The subject technology predicts, using a machine learning model, an indicator of an amount of computing resources for executing the query based at least in part on the set of metadata. The subject technology generates a query plan for executing the query based at least in part on the predicted indicator of the amount of computing resources. The subject technology executes the query based at least in part on the query plan.

Abstract: A method to process UDFs includes performing, by at least one hardware processor of a database system, a resolution of application components to determine a plurality of resolved components of an application and a plurality of data types corresponding to the plurality of resolved components. The method includes instantiating a plurality of UDF clients. The plurality of UDF clients are associated with the plurality of data types. The method includes detecting a received database query specifies a data type of the plurality of data types. The method includes executing the received database query at a UDF client of the plurality of UDF clients.

Abstract: Row-level security (RLS) may provide fine-grained access control based on flexible, user-defined access policies to databases, tables, objects, and other data structures. A RLS policy may be an entity or object that defines rules for row access. A RLS policy may be decoupled or independent from any specific table. This allows more robust and flexible control. A RLS policy may then be attached to one or more tables. The RLS policy may include a Boolean-valued expression.

Abstract: Systems and methods for managing column hiding are provided. The systems and methods receive, from a client device, a query associated with a table. The systems and methods determine an access restriction associated with the client device. The systems and methods identify a column of the table that is restricted by the access restriction associated with the client device. In response to identifying the column of the table that is restricted by the access restriction associated with the client device, the systems and methods provide a result of the query that excludes data corresponding to the column.

Abstract: Embodiments of the present disclosure provide systems and methods for using secure schemas to address inconsistencies between standard RBAC rules and the use of inherited grants. A secure schema may be defined that transfers ownership of an object created in the secure schema to a role that owns the secure schema. An inherited grant may be attached to the secure schema, where the inherited grant specifies a permission on a first type of object in the secure schema and a grant of the permission to the role that owns the secure schema. When objects are created in the secure schema, ownership of each of the set of objects is transferred to the role that owns the secure schema to authorize the role that owns the secure schema to manage grants to the set of objects on the secure schema.

Abstract: In an embodiment, a data platform creates an application in a data-provider account. The application includes one or more APIs corresponding to one or more underlying code blocks. The data platform shares provider data with the application in the data-provider account, and also installs, in a data-consumer account, an application instance of the application. The application instance includes one or more APIs corresponding to the one or more APIs in the application in the data-provider account. The data platform shares consumer data with the application instance in the data-consumer account, and invokes one or more of the APIs of the application instance to execute respective associated underlying code blocks, which are not visible to the data-consumer account. The data platform also saves output of the one or more respective associated underlying code blocks locally within the data-consumer account.

Abstract: Various example embodiments described herein provide for systems, methods, devices, instructions, and the like for optimizing a query plan for executing a query by pushing down top-k information in the query plan with respect to a group-by grouping sets operator or a window function operator, which can be used within a data platform (e.g., database) environment. In particular, various example embodiments enable information from a top-k operator of a query plan to be pushed down the query plan and enable pushing the information to a select operator (e.g., aggregation operator) positioned below the top-k operator that implements the group-by grouping sets operator or the window function operator.

Abstract: A data platform is provided that builds a container image filesystem from a container image. The data platform executes an application that accesses the container image filesystem and the data platform records directory and file access activity of the container image filesystem by the application during execution. The data platform dynamically rebuilds the container image filesystem using the directory and file access activity, and mounts the container image filesystem as rebuilt during a subsequent execution of the application.

Abstract: The subject technology stores an original file master key identifier (FMKID) in metadata associated with an encrypted file, the encrypted file including data. The subject technology determines that the encrypted file is to be re-encrypted based at least on a period of time since the original FMKID was stored. The subject technology re-encrypts the data using a new encryption key associated with a new FMKID. The subject technology generates a mapping from the original FMKID to the new FMKID. The subject technology stores the mapping in a metadata database, the mapping being associated with a re-encrypted file, the re-encrypted file including the re-encrypted data.

Abstract: Techniques for configuring a managed event table (MET) include detecting, by at least one hardware processor, a query for the MET. The query received at a first computing node of a network-based database system. The method includes retrieving via an ingestion function configured at the first computing node, reformatted data from a dedicated storage location of a first process into the MET. The reformatted data is based on log data associated with a second process. The first process and the second process are executing at a second computing node of the network-based database system. The method includes processing the query using the reformatted data in the MET.

Abstract: Monitoring health metrics of computing devices in a data system can be implemented at different levels. At a first level, local background services can be run locally on the computing devices to monitor a set of health metrics on the respective computing devices. At a second level, a central health marker service can monitor a set of health metrics for the computing devices.

Abstract: Different database deployments, or other data system deployments, may want to communicate with each other without sacrificing security or control. To this end, embodiments of the present disclosure may provide secure message exchange techniques for a source and/or target deployment. Configurable rule sets may be stored in the deployments; the rule sets may define what messages may be communicated between deployments. The deployments may implement a selective filtering scheme in one or more stages based on the rule sets to filter outgoing and/or incoming messages.

Abstract: Embodiments of the present disclosure may provide a data clean room allowing secure data analysis across multiple accounts, without the use of third parties. Each account may be associated with a different company or party. The data clean room may provide security functions to safeguard sensitive information. For example, the data clean room may restrict access to data in other accounts. The data clean room may also restrict which data may be used in the analysis and may restrict the output. The overlap data may be anonymized to prevent sensitive information from being revealed.

Abstract: A method for data sharing between cloud deployments is disclosed. A provider in a virtual private cloud (VPC) deployment generates a listing of data for sharing with a consumer in a public multi-tenant deployment. After creating a public account and transmitting a link associated with the listing to the consumer, the method responds to the consumer's selection of the link by automatically creating a secure shared area in the VPC deployment. The method includes replicating both a database and share object associated with the listing into the secure shared area as respective replicas. The replicated objects are then shared with the consumer account as data share objects, enabling the consumer to launch them in a consumer mounted database. The method concludes by enabling the consumer to perform database operations on the provider's data within the VPC deployment, ensuring secure and efficient data sharing between the cloud environments.

Abstract: The subject technology receives a first query. The subject technology determines a set of rows that are modified based on performing the first DML operation on the first table, the set of rows including a first set of updated rows. The subject technology generates a first delta file based on the determined set of rows. The subject technology generates a first updated rows file based on a first set of updated rows. The subject technology determines a second set of rows of the first table that have not been modified by performing the first DML operation on the first table. The subject technology generates a first residual file based on the second set of rows. The subject technology performs a validation process to validate the first DML operation using at least the first delta file, the first residual file, and a matching CoW file.

Abstract: Described herein are techniques for data quality monitoring in a network-based data system. A data metric function used to evaluate data quality can be stored, where the data metric function is defined as schema level object. The data metric function can be attached to a table associated with an account and is evaluated on data associates with the table to generate evaluation results. The evaluation results can be stored in an account-specific central database, from which access is provided to the evaluation results to a user for the account.

Abstract: Provided herein are systems, methods, and computer-storage media for managing data object access in a catalog service account. The method includes detecting, at a catalog service account, by a catalog manager (CM), a user request to access a data object. The user request includes identification information of the user and a principal role associated with the user. The CM retrieves one or more catalog roles from a first catalog of the catalog service account. The one or more catalog roles correspond to the principal role. The CM determines a set of hierarchical data objects based on the user request. The set of hierarchical data objects comprise the data object. The CM performs a determination of whether access privileges of the set of hierarchical data objects and access privileges of the one or more catalog roles allow access to the data object. The CM grants access to the data object based on the determination.