Abstract: A method, apparatus, and system for a multi-instance database scan server is provided to process data scan operations for segments within in-memory stores, such as in-memory columnar (IMC) stores. The scan server uses a client/server based model, wherein each database instance in the DBMS is associated with a client-side scan server and a server-side scan server. The client-side scan server can reside in the data and/or index layer of the DBMS, receiving requests for scan operations on in-memory segments. After receiving a data scan request from an IMC client, the client-side scan server can create several independently executable scanning tasks to process the request. The scanning tasks are each assigned to a particular server-side scan server for processing based on one or more factors, which may include a load balancing component. Additional factors to be considered may include instance affinity and/or NUMA (Non-Uniform Memory Access) affinity.

Abstract: Techniques are provided for granular load and refresh of columnar data. In an embodiment, a particular data object that contains particular data formatted different from column-major format is maintained, the particular data including first data and second data. First and second data objects contain the first and second data, respectively, organized in the column-major format. In response to changes being committed to the first data in the particular data object, invalidating one or more rows of the first data object. In response to a number of invalidated rows of the first data object exceeding a threshold, automatically performing a refresh operation on the first data object independent of any refresh operation on the second data object.

Abstract: Techniques are provided for maintaining data persistently in one format, but making that data available to a database server in more than one format. For example, one of the formats in which the data is made available for query processing is based on the on-disk format, while another of the formats in which the data is made available for query processing is independent of the on-disk format. Data that is in the format that is independent of the disk format may be maintained exclusively in volatile memory to reduce the overhead associated with keeping the data in sync with the on-disk format copies of the data. Selection of data to be maintained in the volatile memory may be based on various factors. Once selected the data may also be compressed to save space in the volatile memory. The compression level may depend on one or more factors that are evaluated for the selected data.

Abstract: Techniques are described herein for maintaining a directory map that contains a data chunk-to-server mapping. Directory maps, maintained within the volatile memory of each server node in the set of multiple server nodes, include a node hierarchy represented as a tree structure in which each node represents a range of an on-disk address range. Each child node, within the node hierarchy, represents a sub-range of the on-disk address range represented by the parent node of the child node. Once a data chunk is loaded into volatile memory of a particular server, mapping information relating the particular on-disk starting address of the data chunk loaded into volatile memory and the pointer address to the location in volatile memory is loaded into one or more node in the directory map.

Abstract: Techniques are described for materializing pre-computed results of expressions. In an embodiment, a set of one or more column units are stored in volatile or non-volatile memory. Each column unit corresponds to a column that belongs to an on-disk table within a database managed by a database server instance and includes data items from the corresponding column. A set of one or more virtual column units, and data that associates the set of one or more column units with the set of one or more virtual column units, are also stored in memory. The set of one or more virtual column units includes a particular virtual column unit storing results that are derived by evaluating an expression on at least one column of the on-disk table.

Abstract: Techniques are described herein for automatically selecting the compression techniques to be used on tabular data. A compression analyzer gives users high-level control over the selection process without requiring the user to know details about the specific compression techniques that are available to the compression analyzer. Users are able to specify, for a given set of data, a “balance point” along the spectrum between “maximum performance” and “maximum compression”. The point thus selected is used by the compression analyzer in a variety of ways. For example, in one embodiment, the compression analyzer uses the user-specified balance point to determine which of the available compression techniques qualify as “candidate techniques” for the given set of data. The compression analyzer selects the compression technique to use on a set of data by actually testing the candidate compression techniques against samples from the set of data.

Abstract: Techniques are herein described for loading a portion of a database object into volatile memory without blocking database manipulation language transactions. The techniques involve invalidating data items loaded from blocks affected by a transaction, referred to as a straddling transaction that started before the load time and committed after the load time. Identifying these straddling transactions involves reviewing one or more transaction lists associated with the set of data items loaded in memory. The transaction list may be read in reverse temporal order of commit to identify a transaction meeting the criteria of starting before the load start, not committing before the load time, and affecting a data item loaded in memory.

Abstract: Techniques are described for maintaining coherency of a portion of a database object populated in the volatile memories of multiple nodes in a database cluster. The techniques involve maintaining a local invalidation bitmap for chunks of data stored in memory in each particular node in the cluster by tracking locks granted by a lock manager. During a pre-loading operation, each given node requests a set of shared locks associated with the chunks of data to be store in the given node's memory. When a request to release one of these shared locks occurs, the in-memory copy of those data items may be invalidated in the node releasing its shared lock.

Abstract: Techniques are provided for maintaining data persistently in one format, but making that data available to a database server in more than one format. For example, one of the formats in which the data is made available for query processing is based on the on-disk format, while another of the formats in which the data is made available for query processing is independent of the on-disk format. Data that is in the format that is independent of the disk format may be maintained exclusively in volatile memory to reduce the overhead associated with keeping the data in sync with the on-disk format copies of the data.

Abstract: Techniques are provided for managing in-memory space and objects. In one embodiment, a set of in-memory objects are maintained within an area in volatile memory that is accessible to a database server. An in-memory object in this context includes a set of one or more in-memory segments where each respective in-memory segment includes a set of in-memory extents and each respective in-memory extent is a contiguous chunk of memory from the area in volatile memory that is accessible to the database server. The area in volatile memory is managed as a set of stripes, where each stripe is a contiguous chunk of in-memory extents. Stripe control blocks are used to locate free in-memory extents for allocation and registration with an in-memory segment.

Abstract: Techniques are provided for performing parallel processing on in-memory objects within a database system. In one embodiment, a plurality of in-memory chunks are maintained on a plurality of non-uniform memory access (NUMA) nodes. In response to receiving a query, a set of clusters is determined for the plurality of in-memory chunks. Each respective cluster in the set of clusters corresponds to a particular NUMA node of the plurality of NUMA nodes and includes a set of one or more in-memory chunks from the plurality of in-memory chunks. For each respective cluster in the set of clusters, a query coordinator assigns, to the respective cluster, a set of one or more processes associated with the particular NUMA node that corresponds to the respective cluster.

Abstract: Techniques for compressing leaf blocks of an index are provided. The decision whether (and/or how much) to compress leaf blocks of an index is performed on a per block basis. Thus, for example, some leaf blocks of the index may be compressed while other leaf blocks of the index may be uncompressed. As another example, different leaf blocks of an index may be compressed a different amount. In one technique, a set of index entries is identified for storage in a leaf block. Multiple estimates of potential space savings are calculated for compressing the set of index entries, one estimate for each different variation of a compression technique. The compression technique variation that is associated with the most space savings is selected and applied to the set of index entries. Thus, some techniques prevent negative compression altogether. Additionally, users require no knowledge of the underlying data to perform optimal compression.

Abstract: Techniques are provided for maintaining data persistently in one format, but making that data available to a database server in more than one format. For example, one of the formats in which the data is made available for query processing is based on the on-disk format, while another of the formats in which the data is made available for query processing is independent of the on-disk format. Data that is in the format that is independent of the disk format may be maintained exclusively in volatile memory to reduce the overhead associated with keeping the data in sync with the on-disk format copies of the data.

Abstract: A method and apparatus for efficiently processing data in various formats in a single instruction multiple data (“SIMD”) architecture is presented. Specifically, a method to unpack a fixed-width bit values in a bit stream to a fixed width byte stream in a SIMD architecture is presented. A method to unpack variable-length byte packed values in a byte stream in a SIMD architecture is presented. A method to decompress a run length encoded compressed bit-vector in a SIMD architecture is presented. A method to return the offset of each bit set to one in a bit-vector in a SIMD architecture is presented. A method to fetch bits from a bit-vector at specified offsets relative to a base in a SIMD architecture is presented. A method to compare values stored in two SIMD registers is presented.

Abstract: Techniques are provided for maintaining data persistently in one format, but making that data available to a database server in more than one format. For example, one of the formats in which the data is made available for query processing is based on the on-disk format, while another of the formats in which the data is made available for query processing is independent of the on-disk format. Data that is in the format that is independent of the disk format may be maintained exclusively in volatile memory to reduce the overhead associated with keeping the data in sync with the on-disk format copies of the data.

Abstract: Techniques are provided for using bitmaps to indicate which items, in a set of items, are invalid. The bitmaps include an “active” bitmap and one or more “temporal clones”. The active bitmap indicates which items in the set are currently valid. The temporal clones are outdated versions of the active bitmap that indicate which items in the set were invalid at previously points in time. Temporal clones may not be very different from each other. Therefore, temporal clones may be efficiently compressed. For example, a bitmap may be selected as a “base bitmap”, and one or more other bitmaps are encoded using delta encoding. Run length encoding may then be applied to further compress the bitmap information. These bitmaps may then be used to determine which items are valid relative to past-version requests.

Abstract: Techniques are provided for more efficiently using the bandwidth of the I/O path between a CPU and volatile memory during the performance of database operation. Relational data from a relational table is stored in volatile memory as column vectors, where each column vector contains values for a particular column of the table. A binary-comparable format may be used to represent each value within a column vector, regardless of the data type associated with the column. The column vectors may be compressed and/or encoded while in volatile memory, and decompressed/decoded on-the-fly within the CPU. Alternatively, the CPU may be designed to perform operations directly on the compressed and/or encoded column vector data. In addition, techniques are described that enable the CPU to perform vector processing operations on the column vector values.

Abstract: To prioritize repopulation of in-memory compression units (IMCU), a database server compresses, into an IMCU, a plurality of data units from a database table. In response to changes to any of the plurality of data units within the database table, the database server performs the steps of: (a) invalidating corresponding data units in the IMCU; (b) incrementing an invalidity counter of the IMCU that reflects how many data units within the IMCU have been invalidated; (c) receiving a data request that targets one or more of the plurality of data units of the database table; (d) in response to receiving the data request, incrementing an access counter of the IMCU; and (e) determining a priority for repopulating the IMCU based, at least in part, on the invalidity counter and the access counter.

Abstract: Techniques are provided for maintaining data persistently in one format, but making that data available to a database server in more than one format. For example, one of the formats in which the data is made available for query processing is based on the on-disk format, while another of the formats in which the data is made available for query processing is independent of the on-disk format. Data that is in the format that is independent of the disk format may be maintained exclusively in volatile memory to reduce the overhead associated with keeping the data in sync with the on-disk format copies of the data.

Abstract: Techniques are provided for automatic verification of query results obtained from data formatted different from row-major format. Techniques are also provided for automatic triage of query results obtained from data formatted different from row-major format. In an embodiment, the query results may be obtained from querying data formatted in column-major format. In an embodiment, when the query results are determined to be incorrect, a cause of the error in the query results can be automatically identified.