Abstract: Data in a micro-partition of a table is stored in a compressed form. In response to a database query on the table comprising a filter, the portion of the data on which the filter operates is decompressed, without decompressing other portions of the data. Using the filter on the decompressed portion of the data, the portions of the data that are responsive to the filter are determined and decompressed. The responsive data is returned in response to the database query. When a query is run on a table that is compressed using dictionary compression, the uncompressed data may be returned along with the dictionary look-up values. The recipient of the data may use the dictionary look-up values for memoization, reducing the amount of computation required to process the returned data.

Abstract: The subject technology receives a command to commit a table in a different table format on an external volume. The subject technology generates a first snapshot of the table on internal storage. The subject technology generates a first list of metadata files on the internal storage. The subject technology generates, based on the first list of metadata files, a first set of metadata files on the internal storage. The subject technology generates a second snapshot of the table on the external volume. The subject technology generates a second list of metadata files on the external volume. The subject technology generates, based on the second list of metadata files, a second set of metadata files on the external volume. The subject technology generates a first set of data files in a different file format on the external volume.

Abstract: Embodiments of the present disclosure may provide a data protection system that performs identification of errors from queries on a database. The data protection system can further identify corrupted data from additional errors, are difficult to detect, and occur between layers of data in the database system. The data protection system can perform corrections of the error data by rebuilding database data or removing the corrupted data.

Abstract: The subject technology receives a command to commit a table in a different table format on an external volume. The subject technology generates a first snapshot of the table on internal storage. The subject technology generates a first list of metadata files on the internal storage. The subject technology generates, based on the first list of metadata files, a first set of metadata files on the internal storage. The subject technology generates a second snapshot of the table on the external volume. The subject technology generates a second list of metadata files on the external volume. The subject technology generates, based on the second list of metadata files, a second set of metadata files on the external volume. The subject technology generates a first set of data files in a different file format on the external volume.

Abstract: A pruning index is generated for a source table organized into a set of batch units. The source table comprises a column of semi-structured data. The pruning index comprises a set of filters that index distinct values in each column of the source table. Rather than reassembling an entire tree structure of the semi-structured data prior to indexing, the generating of the pruning index comprises traversing a reassembly hook object that represents a first portion of the semi-structured data that is subcolumnarized and traversing a residual object that represents a second portion of the semi-structured data that is not subcolumnarized. The reassembly hook object is traversed to identify values corresponding to the first portion of the semi-structured data and the residual object is traversed to identify values corresponding to the second portion. The pruning index is stored with an association with the source table.

Abstract: Systems and methods for caching file system collections atomically. The systems and methods perform operations comprising: receiving a request to access a collection of data comprising a plurality of files stored on a storage device; in response to receiving the request, transferring the collection of data from the storage device to a first subfolder in a cache associated with the storage device; generating a lock file comprising a reference to the first subfolder in the cache; and atomically controlling access to the collection of data in the first subfolder and removal of the collection of data in the first subfolder from the cache via the lock file.

Abstract: A pruning index is generated for a source table organized into a set of batch units. The source table comprises a column of semi-structured data. The pruning index comprises a set of filters that index distinct values in each column of the source table. Rather than reassembling an entire tree structure of the semi-structured data prior to indexing, the generating of the pruning index comprises traversing a reassembly hook object that represents a first portion of the semi-structured data that is subcolumnarized and traversing a residual object that represents a second portion of the semi-structured data that is not subcolumnarized. The reassembly hook object is traversed to identify values corresponding to the first portion of the semi-structured data and the residual object is traversed to identify values corresponding to the second portion. The pruning index is stored with an association with the source table.

Abstract: Embodiments of the present disclosure may provide a data protection system that performs identification of errors from queries on a database. The data protection system can further identify corrupted data from additional errors, are difficult to detect, and occur between layers of data in the database system. The data protection system can perform corrections of the error data by rebuilding database data or removing the corrupted data.

Abstract: Multiple execution threads process a query directed to a database organized into a plurality of files. In processing the query, a first thread downloads a file from the plurality of files. The file comprises a set of blocks. A parallel scan of the set of blocks is performed by at least the first thread and a second thread to identify data that matches the query. A response to the query is provided based in part on the parallel scan of the set of blocks.

Abstract: A pruning index is generated for a source table organized into a set of batch units. The source table comprises a column of semi-structured data. The pruning index comprises a set of filters that index distinct values in each column of the source table. Rather than reassembling an entire tree structure of the semi-structured data prior to indexing, the generating of the pruning index comprises traversing a reassembly hook object that represents a first portion of the semi-structured data that is subcolumnarized and traversing a residual object that represents a second portion of the semi-structured data that is not subcolumnarized. The reassembly hook object is traversed to identify values corresponding to the first portion of the semi-structured data and the residual object is traversed to identify values corresponding to the second portion. The pruning index is stored with an association with the source table.

Abstract: A pruning index is generated for a source table organized into a set of batch units. The source table comprises a column of semi-structured data. The pruning index comprises a set of filters that index distinct values in each column of the source table. Rather than reassembling an entire tree structure of the semi-structured data prior to indexing, the generating of the pruning index comprises traversing a reassembly hook object that represents a first portion of the semi-structured data that is subcolumnarized and traversing a residual object that represents a second portion of the semi-structured data that is not subcolumnarized. The reassembly hook object is traversed to identify values corresponding to the first portion of the semi-structured data and the residual object is traversed to identify values corresponding to the second portion. The pruning index is stored with an association with the source table.

Abstract: In one aspect, a computer-implemented method includes detecting, by a server includes one or more processors, a request to perform a hash join operation on a data structure stored in a data storage device, forming a hash lookup dictionary based on lookup results in a hash table, storing the hash lookup dictionary in a cache, and probing, during a probing phase of the hash join operation, the cache.

Abstract: Data in a micro-partition of a table is stored in a compressed form. In response to a database query on the table comprising a filter, the portion of the data on which the filter operates is decompressed, without decompressing other portions of the data. Using the filter on the decompressed portion of the data, the portions of the data that are responsive to the filter are determined and decompressed. The responsive data is returned in response to the database query. When a query is run on a table that is compressed using dictionary compression, the uncompressed data may be returned along with the dictionary look-up values. The recipient of the data may use the dictionary look-up values for memoization, reducing the amount of computation required to process the returned data.

Abstract: A storage device uses non-volatile memory devices for caching. The storage device operates in a mode referred to herein as write-back mode. In write-back mode, a storage device responds to a request to write data by persistently writing the data to a cache in a non-volatile memory device and acknowledges to the requestor that the data is written persistently in the storage device. The acknowledgement is sent without necessarily having written the data that was requested to be written to primary storage. Instead, the data is written to primary storage later.

Abstract: Techniques are provided for using an intermediate cache to provide some of the items involved in a scan operation, while other items involved in the scan operation are provided from primary storage. Techniques are also provided for determining whether to service an I/O request for an item with a copy of the item that resides in the intermediate cache based on factors such as a) an identity of the user for whom the I/O request was submitted, b) an identity of a service that submitted the I/O request, c) an indication of a consumer group to which the I/O request maps, d) whether the I/O request is associated with an offloaded filter provided by the database server to the storage system, or e) whether the intermediate cache is overloaded. Techniques are also provided for determining whether to store items in an intermediate cache in response to the items being retrieved, based on logical characteristics associated with the requests that retrieve the items.

Abstract: Approaches, techniques, and mechanisms are disclosed for improved caching in database systems that deal with multiple data access patterns, such as in database systems that interface with both OLTP and Data Warehouse clients. A cache is deployed between a database server and a storage system that stores data units. Some of the data units accessed by the database server are buffered within the cache. The data units may be associated with data access patterns, such as a random data access pattern or a scan data access pattern, in accordance with which the database server is or appears to be accessing the data units. A processor selects when to cache data units accessed by the database server, based at least on the associated data access patterns. Recent access counts may also be stored for the data units, and may further be utilized to select when to cache data units.

Abstract: Techniques herein are for chaining nonvolatile storage devices to achieve high availability. A method involves a storage server receiving a write request to store data blocks in a first nonvolatile memory device. The storage server comprises a plurality of nonvolatile memory devices that cache data blocks stored on primary storage. The plurality of nonvolatile memory devices comprises the first nonvolatile memory device. The storage server maintains a cache index of data blocks that reside in the plurality of nonvolatile memory devices. Based on one or more criteria, the storage server reroutes the write request to a second nonvolatile memory device of the plurality of nonvolatile memory devices and stores an identifier of the second nonvolatile memory device in the cache index.

Abstract: A method and system for fast file initialization is provided. An initialization request to create or extend a file is received. The initialization request comprises or identifies file template metadata. A set of allocation units are allocated, the set of allocation units comprising at least one allocation unit for the file on a primary storage medium without initializing at least a portion of the file on the primary storage medium. The file template metadata is stored in a cache. The cache resides in at least one of volatile memory and persistent flash storage. A second request is received corresponding to a particular allocation unit of the set of allocation units. Particular file template metadata associated with the particular allocation unit is obtained. In response to the second request, at least a portion of a new allocation unit is generated.

Abstract: Flash memory on a flash memory device is virtualized using compression that is native to the flash memory device. Through compression, the flash memory device is used to logically store more data in a virtual address space that is larger than the physical address space of the flash memory device. Physical storage capacity of a flash memory device may prevent further storage of data even when the virtual address space is not fully populated. Because compressibility may vary, the extent to which the virtual address space may be populated before physical storage capacity is reached varies. The approaches for virtual memory described herein rely on the memory device client to monitor when this point is reached. In addition, the memory device client is responsible for freeing space as needed to accommodate subsequent requests to store data in the flash memory.

Abstract: A method and system for fast file initialization is provided. An initialization request to create or extend a file is received. The initialization request comprises or identifies file template metadata. A set of allocation units are allocated, the set of allocation units comprising at least one allocation unit for the file on a primary storage medium without initializing at least a portion of the file on the primary storage medium. The file template metadata is stored in a cache. The cache resides in at least one of volatile memory and persistent flash storage. A second request is received corresponding to a particular allocation unit of the set of allocation units. Particular file template metadata associated with the particular allocation unit is obtained. In response to the second request, at least a portion of a new allocation unit is generated.