Abstract: Systems and methods for integrating the lifecycle of nonvolatile memory blocks with the transactional guarantees of a database are disclosed. One method includes creating a first fragment of a first database column in a volatile memory system, receiving a first pointer to a first block of a second non-volatile memory (NVM) system, the first block associated with the first fragment of the first database column, and populating the first block of the second non-volatile memory system using the first pointer. The method further includes committing the first block of the second non-volatile memory system and associating a first block identifier of the first block of the second non-volatile memory system with the first fragment of the first database column, while blocking creation of a database savepoint, and then unblocking creation of the database savepoint. A block key associated with a first fragment may comprise various identifiers such as block, column, table and partition identifiers.

Abstract: A system includes a volatile random access memory storing first header data and a first data block, the first header data including descriptive information and a first pointer to the first data block, and a non-volatile random access memory. The system includes determination of a memory size associated with the first header data and the first data block, allocation of a first memory block of the non-volatile random access memory based on the determined memory size, determination of an address of the non-volatile random access memory associated with the allocated first memory block, and writing of the descriptive information and a binary copy of the first data block at the address of the non-volatile random access memory.

Abstract: Disclosed herein are system, method, and computer program product embodiments for partition pruning using globally encoded range partition information. An embodiment operates by partitioning a data table into a plurality of data partitions, determining a plurality of sub-partitions within the data partitions, and determining a global range table based at least in part on value ranges corresponding to the sub-partitions. Further, the global range table may be used to determine bit vectors for the data partitions and query predicates. In some examples, the bit vectors may be used to determine whether to prune data partitions during execution of a query over a large data volume.

Abstract: Disclosed herein are system, method, and computer program product embodiments for partition pruning via non-homogenous synopsis information. An embodiment operates by maintaining synopsis information for a data partition in accordance with a first synopsis strategy, monitoring performance of the synopsis information within a partition pruning system, determining that the performance of the synopsis information is insufficient, and updating the synopsis information in accordance with a second synopsis strategy better suited for the attributes of the data partition. In some embodiments, a first data partition of a partitioned data table may employ a first synopsis strategy and a second data partition of the partitioned data table may employ a second synopsis strategy.

Abstract: The subject matter disclosed herein provides methods for reformatting a page due to a rollover. An in-memory array holding a column of data can be maintained. One or more pages can be maintained. Each page can have one or more rows for storing the column of data. The column of data in the in-memory array can be monitored for a change. A rollover can be performed on at least one of the pages based on the change. The rollover can reformat the at least one page by rewriting metadata associated with the at least one page. Related apparatus, systems, techniques, and articles are also described.

Abstract: A system and method, including determining, by a processor, a data type for each column of a database table; determining, by the processor and based on the determined data type for each column of the database table, an indication of a size of the database table; calculating, by the processor and based on the determined indication of the size of the database table, a start nbit size for a nbit compression process to be used on the database table; specifying, by the processor, the calculated start nbit size for the nbit compression process; and compressing the database table by executing the nbit data compression process using the specified start nbit size.

Abstract: Computer-implemented systems and methods for categorizing tables of a database into groups are provided. A set of queries used to access a database are processed to generate a log that identifies for each of the queries (i) a database operation associated with the query, (ii) a table of the database accessed via the query, and (iii) a column or columns of the table used by the query to access the table. Database operation criteria for categorizing database tables into groups are accessed. The criteria are based on a total number of times that a table is accessed by the set of queries and a number of times that the table is accessed using a column or group of columns. An algorithm that evaluates data of the log against the criteria is applied. The algorithm is configured to automatically categorize tables identified in the log into groups based on the evaluation.

Abstract: Disclosed herein are system and method embodiments for generating a paged inverted index. An embodiment is generated by storing a first data structure and the second data structure in a plurality of pages, where the plurality of pages are stored in the one or more memories. The first data structure is stored in the plurality of pages and includes a plurality of value identifiers, where a value identifier corresponds to an offset. The second data structure stored in the plurality of pages includes a plurality of row positions, wherein a row position is at a location that corresponds to the offset in the first data structure and identifies a position of row in a table that stores data associated with the value ID.

Abstract: An oldest version timestamp is received from a first database. A current commit timestamp is received from a second database configured to asynchronously replicate the first database. The current global commit timestamp corresponds to a commit transaction executed in the first database and subsequently replicated in the second database. A first test query is then issued to the second database including a request for a current timestamp of the second database. A second test query is issued to the primary database requesting records associated with a target update version that has a target timestamp equal to the current snapshot timestamp. A determination is then made that a first return associated with the first test query and a second return associated with the second test query are identical indicating a successful validation. Related apparatus, systems, techniques and articles are also described.

Abstract: System and methods for persisting in-memory runtime data within a database system include receiving a query at a database system that includes a unified table container and determining whether the query includes a data definition statement. In response to determining that the query includes a data definition statement, in-memory runtime data is persisted to a persistent volume that is configured to be accessed by the database system. The in-memory runtime data includes one or more first data fields and one or more second data fields that are different from the one or more first data fields. Furthermore, the unified table container includes a persistent runtime data descriptor that is associated with the one or more first data fields and persists the first one or more first data fields when persisting the in-memory runtime data. Related apparatus, systems, techniques and articles are also described.

Abstract: Disclosed herein are system, method, and computer program product embodiments for determining a constraint data statistics object is consistent. An embodiment operates by receiving a query for a database table. The embodiment determines a constraint data statistic object associated with a partition of the database table. The embodiment determines the constraint data statistics object is consistent with a data state of the partition of the database table based on consistency metadata of the constraint data statistics object. The embodiment processes the query for the database table based on the determining the constraint data statistics object is consistent with the data state of the partition of the database table.

Abstract: Disclosed herein are system, method, and computer program product embodiments for performing dynamic partition pruning using data statistic objects as data integrity constraints. An embodiment operates by partitioning a database table into a plurality of partitions based on a partition criterion. The embodiment creates a data statistics object for a partition in the plurality of partitions. The embodiment receives a query for the database table. The embodiment determines the data statistics object is consistent with data in the partition. The embodiment processes the query for the partition based on the data statistics object.

Abstract: Disclosed herein are system, method, and computer program product embodiments for using a data statistic as a dynamic data integrity constraint. An embodiment operates by defining a data statistic for a column or a set of columns of a partition of a plurality of partitions of a database table. The embodiment creates a constraint data statistics object based on the data statistic. The embodiment receives a query for the database table. The embodiment determines the constraint data statistics object is consistent with a data state of the partition. The embodiment derives an implied constraint based on the constraint data statistics object. The embodiment processes the query for the partition based on the implied constraint.

Abstract: As part of a database system comprising a combination of on-disk storage and in-memory storage, a plurality of records that comprise a table are stored in a plurality of fragments that include at least a delta fragment and a main fragment retained in the on-disk storage. Each fragment has visibility data structures to enable multi-version concurrency control. Each fragment can be compressed using dictionary compression and n-bits compression. The fragments are loaded into main system memory in the in-memory storage from the on-disk storage if they are accessed for read operations or write operations and are not already in memory. A plurality of lockless read and write operations are concurrently performed, while providing snapshot isolation, on the at least one of the plurality of fragments while the at least one of the plurality of fragments is in the main system memory.

Abstract: A compression manager may store, within an index vector, a plurality of value identifiers (IDs), each value ID representing a value within a database. A page generator may designate a number of the value IDs as defining a page within the index vector, so that the index vector includes a plurality of pages, each page including the number of value IDs. The page generator may store the index vector in a secondary memory of a main memory database. An iterator may access a requested value ID, and a page loader may load a corresponding page of the index vector that contains the requested value ID into the main memory database.

Abstract: The present disclosure involves systems, software, and computer implemented methods for using declarative rules for optimized access to data. One example method includes receiving a structured query language (SQL) query, the SQL query associated with at least one database table and at least one predicate. A determination is made as to whether the SQL query is associated with at least one declarative rule based on the at least one database table and the at least one predicate. In response to determining that the SQL query is associated with at least one declarative rule, the at least one associated declarative rule is applied, during optimization of the SQL query. The at least one declarative rule defines a transformation to the SQL query to be performed during query optimization. An updated SQL query is generated based on the applied at least one associated declarative rule and the updated SQL query is provided.

Abstract: A system includes creation of a first fragment of a first database column in a volatile memory system, reception of a first pointer to a first block of a second non-volatile memory system, the first block associated with the first fragment of the first database column, population of the first block of the second non-volatile memory system using the first pointer, while blocking creation of a database savepoint, commitment of the first block of the second non-volatile memory and association of a first block identifier of the first block of the second non-volatile memory with the first fragment of the first database column, and unblocking creation of the database savepoint.

Abstract: A request for an entry of a dictionary having multiple dictionary blocks may be received, where the dictionary stores string values associated with corresponding value identifiers (IDs) for dictionary compression. One of the dictionary blocks may be selectively loaded into memory, and the dictionary block that has been loaded into memory may be searched. A value ID directory may be constructed in memory, where the value ID directory includes last value IDs for the dictionary blocks, and each of the last value IDs is mapped to an index of one of the dictionary blocks that includes a string value for that last value ID. A separator directory may also be constructed in memory, where the separator directory includes separators for the dictionary blocks, and each of the separators is mapped to an index of one of the dictionary blocks that includes a string value for that separator.

Abstract: As part of a columnar in-memory database, value identifiers are inserted into a backing array in-memory until such time that it is determined that such backing array does not have sufficient capacity. A new backing array is then generated that includes the value identifiers in the old backing array and which has sufficient capacity. The old backing array can be flushed from memory when there are no active operations using such backing array. Such an arrangement allows for readers and non-structural writers to operate concurrently.

Abstract: A computer implemented method for manipulating data comprises receiving log transaction information from a primary database system at a secondary database system and parsing the log transaction information to identify data objects that have been modified on the primary database system. The secondary database system determines whether the data objects are present in main memory on the secondary database system. If the data objects are not in the main memory, they are copied from persistence data volumes to the main memory. The data objects are modified by replaying the log transactions on the data objects in the main memory. Then, the data objects in the persistence data volumes are modified. Related apparatus, systems, techniques and articles are also described.