Abstract: A system and method for pruning data based on metadata. The method may include receiving a query that includes a plurality of predicates and identifying one or more applicable files including database data satisfying at least one of the plurality of predicates. The identifying the one or more applicable files including reading metadata stored in a metadata store that is separate from the database data. The method further includes pruning inapplicable files including database data that does not satisfy at least one of the plurality of predicates to create a reduced set of files and reading the reduced set of files to execute the query.

Abstract: Disclosed herein are embodiments of systems and methods for synchronizing file-catalog table with a file stage. In an embodiment, a data platform receives a notification of a modification to one or more files in a file stage. The file stage includes data storage having a storage location. The data platform updates, based on receiving the notification of the modification, a first file-catalog table for the file stage based on the modification. The first file-catalog table includes a row for each file in the file stage, as well as a column for each of one or more metadata properties of the one or more files in the file stage.

Abstract: Functional wrappers are scripts and related software that provide a way to group and invoke functionality comprising semantic intent in a legacy application. These functional wrappers allow programmatic access to functionality in legacy applications in contemporary software architectures without risk of porting errors. Additional functionality to provide features with presently expected, but not available at the time the legacy applications were developed, such as functional wrappers to provide parallelism and scaling, are disclosed. Finally, automatic generation of the functional wrappers are also disclosed.

Abstract: Example resource management systems and methods are described. In one implementation, a resource manager is configured to manage data processing tasks associated with multiple data elements. An execution platform is coupled to the resource manager and includes multiple execution nodes configured to store data retrieved from multiple remote storage devices. Each execution node includes a cache and a processor, where the cache and processor are independent of the remote storage devices. A metadata manager is configured to access metadata associated with at least a portion of the multiple data elements.

Abstract: A system to repeatedly execute a test run of a workload using resources of a cloud environment to determine whether there is a performance difference in the test run. The system to, in response to determining that there is no performance difference, identify one or more sets of decreased resources of the cloud environment. The system to re-execute the test run using the one or more sets of decreased resources of the cloud environment to determine whether there is a performance difference in the test run that is attributed to the one or more sets of decreased resources of the cloud environment. The system to determine minimum resources of the cloud environment to repeatedly execute the test run using the minimum resources without existence of a performance difference in response to re-executing the test run using the one or more sets of decreased resources of the cloud environment.

Abstract: Embodiments of the present disclosure describe systems, methods, and computer program products for redacting sensitive data within a database. An example method can include examining a first column of a plurality of columns of a database to identify a first category of sensitive data in a first location of the first column and a second category of sensitive data in a second location of the first column, receiving a masking policy for the first column, the masking policy identifying the first category of sensitive data, and, in response to a data query accessing the first column, executing a redaction operation to redact the first category of sensitive data from the first location of the first column to generate redacted data for a response to the data query.

Abstract: A data platform for developing and deploying a data application. The data platform receives from a first user the data application and provider granted privileges including a consumer usage privilege and a consumer access to data privilege. The data platform authorizes the second user to access the data platform based on one or more consumer account privileges included in a set of account privileges. The data platform authorizes the second user to execute the data application based on the consumer usage privilege. During execution, the data platform authorizes the data application to access the provider database object based on the consumer access to data privilege, and authorizes the data application to access the consumer database object based on a provider access to data privilege provided by the second user.

Abstract: The subject technology receives by a database system, raw input data from a source table provided by an external environment, the source table comprising multiple rows and multiple columns, the raw input data comprising values in a first format, the values comprising input features corresponding to datasets included in the raw input data for machine learning models, the external environment comprising an external system from the database system and is accessed by different users. The subject technology generates cell data for a second table based on the values from the source table. The subject technology performs a database operation to generate the second table including table metadata, column metadata, and the generated cell data.

Abstract: A hardware database privacy device is communicatively coupled to a private database system. The hardware database privacy device receives a request from a client device to perform a query of the private database system and identifies a level of differential privacy corresponding to the request. The identified level of differential privacy includes privacy parameters (?,?) indicating the degree of information released about the private database system. The hardware database privacy device identifies a set of operations to be performed on the set of data that corresponds to the requested query. After the set of data is accessed, the set of operations is modified based on the identified level of differential privacy such that a performance of the modified set of operations produces a result set that is (?,?)-differentially private.

Abstract: A data dictionary generation system automatically populates and updates a data dictionary for listings offering shared data. The data listing distribution component distributes the data dictionaries to various remote deployments in a data exchange by using a global messaging framework and replication method. For example, the data listing distribution component replicates a data dictionary generated for the listing and its shared data from a source deployment to one or more destination deployments associated with various geographic regions. The data listing distribution component distributes the listing to the various remote deployments to allow for the listing, including its shared data and data dictionary, to be accessed by users within the geographic region associated with the remote deployment.

Abstract: Described herein are techniques for improving transfer of metadata from a metadata database to a database stored in a data system, such as a data warehouse. The metadata may be written into the metadata database with a version stamp, which is monotonic increasing register value, and a partition identifier, which can be generated using attribute values of the metadata. A plurality of readers can scan the metadata database based on version stamp and partition identifier values to export the metadata to a cloud storage location. From the cloud storage location, the exported data can be auto ingested into the database, which includes a journal and snapshot table.

Abstract: A cloud extension agent can be provided on a customer premise for interfacing, via an outbound secure connection, cloud based services.

Abstract: Systems, methods, and devices for implementing secure views for zero-copy data sharing in a multi-tenant database system are disclosed. A method includes generating a share object in a first account comprising a share role. The method includes associating view privileges for the share object such that an underlying detail of the share object comprises a secure view definition. The method includes granting, to a second account, cross-account access rights to the share role or share object in the first account. The method includes receiving a request from the second account to access data or services of the first account and providing a response to the second account based on the data or services of the first account. The method is such that the underlying detail of the share object that comprises the secure view definition is hidden from the second account and visible to the first account.

Abstract: Provided herein are systems and methods for a database object (e.g., materialized table) configuration including using the database object for incremental maintenance of query results. For example, a method includes creating a materialized table using a table definition. The table definition includes a query on one or more base tables and a lag duration value. The lag duration value indicates a maximum time period that a result of a prior refresh of the query can lag behind a current time instance. A delta set for the one or more base tables is determined. The delta set includes at least one change applied to the one or more base tables since the prior refresh of the query. The delta set is merged with the materialized table to perform an incremental update of the materialized table.

Abstract: Provided herein are systems and methods for resilience testing using a DEW. A system includes at least one hardware processor coupled to a memory and configured to decode a workflow to obtain a plurality of workflow steps. Each workflow step of the plurality of workflow steps includes a runtime engine identifier and a script identifier. A script for a workflow step of the plurality of workflow steps is retrieved based on the script identifier. The script specifies a plurality of tasks. One or more containers are configured with a runtime engine based on the runtime engine identifier. The script executes against the one or more containers using the runtime engine, to perform the plurality of tasks.

Abstract: A data platform creates an application in a data-provider account, where the application includes one or more application programming interfaces (APIs) corresponding to one or more underlying code blocks. The data platform shares homomorphically encrypted provider data with the application in the data-provider account. The data platform installs, in a data-consumer account, an application instance of the application. The data platform shares homomorphically encrypted consumer data with the application instance in the data-consumer account. The data platform invokes one or more of the APIs of the application instance to execute respective associated underlying code blocks, which are not visible to the data-consumer account, and which operate on the shared homomorphically encrypted provider data and the shared homomorphically encrypted consumer data. The data platform saves homomorphically encrypted output of the one or more respective associated underlying code blocks locally within the data-consumer account.

Abstract: The subject technology receives a transaction for performing an operation on a distributed database, the transaction associated with an account. The subject technology identifies a tenant corresponding to the account associated with the transaction. The subject technology retrieves a set of encryption keys based at least in part on the account and the tenant. The subject technology determines a derived encryption key using a cryptographic hash function applied to a hybrid tenant master encryption key and a local random generated identifier. The subject technology encrypts a record value and a key value associated with transaction using the derived encryption key. The subject technology determines a tree structure associated with the tenant. The subject technology determines a non-leaf node using a tenant prefix of the tenant. The subject technology inserts the encrypted record value, and the encrypted key value at a leaf node below the non-leaf node of the tree structure.

Abstract: A method for table data processing includes retrieving, by at least one hardware processor, first table data associated with a first table version from a plurality of partitions of a storage device. The method further includes detecting, by the at least one hardware processor, an executed transaction causing a modification to the first table data resulting in a second table data associated with a second table version. The second table data is modified to include partition metadata. The partition metadata includes table version information with a mapping of the first table version and the second table version to a plurality of transactions associated with generating the first table data and the second table data.

Abstract: Methods, systems, and computer programs are presented for a packages policy object to enable customers to create and apply packages policies provided by third-party package managers. A user-defined function (UDF) is received by a cloud data platform. The UDF includes code related to at least one operation to be performed. The cloud data platform receives a package policy including at least one allowlist and at least one blocklist and compares the at least one allowlist and the at least one blocklist to the at least one package specification. The cloud data platform computes a difference set based on the comparison. The difference set includes metadata associated with one or more packages to remove. The cloud data platform determines whether the UDF is permitted based at least in part on the difference set.

Abstract: A method for sharing data in a multi-tenant database includes generating a share object in a first account comprising a share role. The method includes associating one or more access rights with the share role, wherein the one or more access rights indicate which objects in the first account are accessible based on the share object. The method includes granting, to a second account, cross-account access rights to the share role or share object in the first account. The method includes receiving a request from the second account to access data or services of the first account. The method further includes providing a response to the second account based on the data or services of the first account.