Abstract: An in-database application package and application instance for a data platform. The data platform creates an application instance of an application package having a versioned schema, creates one or more system roles for the application instance, creates a user role and an administrator role for the application instance, creates one or more objects of the application instance based on a versioned schema, and grants one or more use privileges to the one or more roles. Application instances of the application package are upgraded or patched on the data platform based on application package versions. To ensure a proper upgrade or patch, the data platform tracks versions of executing objects of application instances in a call context.

Abstract: A data platform is provided that builds a container image filesystem from a container image. The data platform executes an application that accesses the container image filesystem and the data platform records directory and file access activity of the container image filesystem by the application during execution. The data platform dynamically rebuilds the container image filesystem using the directory and file access activity, and mounts the container image filesystem as rebuilt during a subsequent execution of the application.

Abstract: Example caching systems and methods are described. In one implementation, a method receives a set of queries to be processed by a set of virtual warehouses. The method distributes the set of queries to the set of virtual warehouses to be executed and creates, during the processing of the set of queries by the set of virtual warehouses, a new virtual warehouse, wherein cache resources associated with the new virtual warehouse are populated with data files associated with the set of queries at the time the virtual warehouse is created and the cache resources vary among the processors, wherein a first subset of the processors comprises minimal cache resources and a second subset of processors comprises cache resources providing faster input-output operations. The method redistributes the set of queries across the set of virtual warehouses.

Abstract: Example caching systems and methods are described. In one implementation, a method receives a query, at an execution platform, directed to data stored across a plurality of shared storage devices, the execution platform comprising one or more execution nodes, an execution node comprising a plurality of processors. The method processes the query using the one or more execution nodes of the execution platform, and in response to a determination of a change in a number of execution nodes of the execution platform, wherein the change is creating a new execution node, wherein a first subset of the plurality of processors comprises a minimal cache and a second subset of the plurality of processors comprises a cache providing faster input-output operations, reassigns processing of the query, among the changed number of execution nodes of the execution platform.

Abstract: The subject technology receives a first query, the first query comprising a first set of statements, the first set of statements including at least a first statement for performing a first Data Manipulation Language (DML) operation on a first table, the first table included in a source file. The subject technology executes the first query, the executing including determining whether to perform a copy-on-write (CoW) process. subject technology performs a CoW fallback process. The subject technology performs the copy-on-write process for the first query based on a result of the CoW fallback process.

Abstract: Provided herein are systems and methods for replicating share objects to remote deployments. An example method includes creating a share object for a first deployment of a data sharing platform. The share object includes share object metadata that describes access privileges in the first deployment. The method also includes creating, in a second deployment of the data sharing platform, a replica of the share object. The replica determines access privileges in the second deployment. The method also includes receiving, at the first deployment, a refresh message from the second deployment. The method also includes, in response to receiving the refresh message, transmitting the share object metadata from the first deployment to the second deployment and updating, by a processing device, the replica based on the share object metadata.

Abstract: Provided herein are systems, methods, and computer-storage media for managing data object access in a catalog service account. The method includes detecting, at a catalog service account, by a catalog manager (CM), a user request to access a data object. The user request includes identification information of the user and a principal role associated with the user. The CM retrieves one or more catalog roles from a first catalog of the catalog service account. The one or more catalog roles correspond to the principal role. The CM determines a set of hierarchical data objects based on the user request. The set of hierarchical data objects comprise the data object. The CM performs a determination of whether access privileges of the set of hierarchical data objects and access privileges of the one or more catalog roles allow access to the data object. The CM grants access to the data object based on the determination.

Abstract: The subject technology provides embodiments for supporting a unified table which may be a managed table or an unmanaged table. Managed tables are those where the subject technology manages the metastore/catalog for the table, whereas unmanaged tables are tables where an external catalog controls the table and the subject technology integrates with that catalog to work with the table, but does not assume control of the table.

Abstract: Example methods include providing a data platform for participants to assign roles of various rights to access data objects on the data platform. A first participant acting in a role as an owner may create a code entity on the data platform to interact with the data objects. A second participant acting in a role as an administrator may define a security boundary over the code entity created by the first participant. A third participant acting as a caller may request to interact with the data objects using the code entity. A processing device of the database system provides the third participant, access to the code entity created by the first participant based on the security boundary defined by the second participant.

Abstract: A method includes detecting, by at least one hardware processor, a change request for a table of a database, the table comprising a plurality of micro-partitions. A transaction associated with the change request is executed at a first timestamp. The transaction causes replacement of a first micro-partition of the plurality of micro-partitions with a second micro-partition. A change tracking column is generated in the second micro-partition. The change tracking column comprises metadata for the transaction. A delta for the table between the first timestamp and a second timestamp is generated using the metadata in the change tracking column. The delta indicates changes made to one or more rows of the table between the first time stamp and the second timestamp.

Abstract: A system and method of using a stateless node to process data of a catalog object. The method includes accessing a catalog object comprising metadata associated with a dataset. The method includes distributing, by one or more processors, a task to a stateless node to cause the stateless node to process the dataset without storing information indicative of a particular state of the stateless node.

Abstract: Example resource management systems and methods are described. The method includes providing a plurality of processors and a plurality of cache memories in association with a virtual data warehouse management system, wherein each of the plurality of processors is associated with a stateless node. The method includes monitoring a performance of at least one processor of the plurality of processors to process database data. The method includes detecting a failure associated with the stateless node to process the database data responsive to monitoring the performance of the at least one processor of the plurality of processors to process the database data. The method includes replacing the stateless node with a different node without recreating a particular state.

Abstract: Embodiments of the present disclosure related to sharing applications within a data sharing platform. An example method includes replicating a database from a provider account of a data sharing platform to a consumer account of the data sharing platform. The method further includes executing an installation script within the consumer account to install an application in the consumer account of the data sharing platform responsive to the replicating. The method further includes creating, by a processing device, a set of database roles to manage execution of the application in the consumer account, wherein one or more of the set of database roles determine access privileges granted to the application for accessing the database inside the consumer account. The application is to perform a data processing service relative to a data asset in the database.

Abstract: Embodiments of the present disclosure provide techniques for mountless querying of listing data. A processing device obtains a query that includes a universal listing identifier of a database, wherein the universal listing identifier is different from an identifier for the database. The processing device activates, at runtime, at least one role for accessing the database and shared objects based on the universal listing identifier. The processing device generates, based on the universal listing identifier and the at least one activated role, an in-memory placeholder object associated with the database. The processing device provides access to data of the database based on the in-memory placeholder object and the query.

Abstract: As described herein, a N-Gram index may be created and the search may be conducted using the index, which will lead to faster search results. The N-Gram index may also include partial N-Gram components to capture more relevant data. Moreover, as described herein, the search may also take into account recent log data that has not yet been indexed. Techniques for building an index store using log data and efficiently searching the index store and log data to process search requests are described herein.

Abstract: The subject technology receives a first set of statements to create an image repository, the first set of statements comprising a set of Structured Query Language (SQL) statements. The subject technology creates the image repository, the creating including generating a uniform resource locator (URL), the URL associated with a location for storing data. The subject technology performs a domain name service (DNS) mapping of the URL to an image registry service. The subject technology receives a first command to tag a container image to an alias corresponding to the URL. The subject technology assigns the alias to the container image. The subject technology receives a second command to push the tagged container image to the image repository. The subject technology sends the tagged container image to the location to store the tagged container image.

Abstract: Example resource management systems and methods are described. In one implementation, a resource manager is to be coupled to one or more virtual data warehouse instances that each comprise a plurality of computing resources. A resource manager receives a query directed to database data. The resource manager determines a task associated with the query. The resource manager distributes the task to a first virtual data warehouse instance of the plurality of virtual data warehouse instances. The resource manager determines that the first virtual data warehouse instance of the plurality of virtual data warehouse instances no longer accesses the database data. The resource manager reconfigures the first virtual data warehouse instance to delete a communication between the first virtual data warehouse instance and the database.

Abstract: Provided herein are systems and methods for discrete workload processing using a file processing service. An example method includes retrieving a manifest file from a work queue. The manifest file includes metadata associated with a plurality of workloads. A plurality of processing configurations corresponding to the plurality of workloads is generated. A processing configuration of the plurality of processing configurations is associated with scheduling execution of one or more tasks for a workload of the plurality of workloads. A processing pipeline definition of the manifest file is generated. The processing pipeline definition includes the plurality of processing configurations. The processing pipeline definition is registered with a pipeline definition registry of a network-based database system to generate a definition registration.

Abstract: Replication of data is disclosed. A method includes replicating the data stored in a primary deployment hosted by a first cloud storage provider such that the data is further stored in a secondary deployment hosted by a second cloud storage provider. The method includes determining that the primary deployment transitioned from an available state to an unavailable state. The method includes executing one or more transactions on the data at the secondary deployment to cause a change to the data in response to determining that the primary deployment is unavailable.

Abstract: A data platform for managing an application as a first-class database object. The data platform includes at least one processor and a memory storing instructions that cause the at least one processor to perform operations including detecting a data request from a browser for a data object located on the data platform, executing a stored procedure, the stored procedure containing instructions that cause the at least one processor to perform additional operations including instantiating a User Defined Function (UDF) server, an application engine, and the application within a security context of the data platform based on a security policy determined by an owner of the data object. The data platform then communicates with the browser using the application engine as a proxy server.