Abstract: Disclosed herein are system, method, and computer-program product embodiments for generating a paged and in-memory representation of a database object. An embodiment operates by maintaining in-memory and paged form primitives unique to the database object or a substructure thereof in a database such that the in-memory and paged form primitives are capable of providing the in-memory and paged representations of the database objects, respectively. Thereafter, a load configuration for the database object is determined. Based on the load configuration, the in-memory and/or paged representations of the database object are generated using the in-memory form primitive or the paged form primitive unique to the database object, respectively. Subsequently, the in-memory and/or paged representations of the database object are stored in the database.

Abstract: Systems, apparatus, techniques, articles, and methods include a unified table container for storing data and metadata in pages organized in one or more page chains. A determination is made as to whether the query includes a data definition statement. In response to determining that the query includes a data definition statement, in-memory runtime data is persisted to a persistent volume that is configured to be accessed by the database system. The in-memory runtime data includes one or more first data fields and one or more second data fields that are different from the one or more first data fields. Furthermore, the unified table container includes a persistent runtime data descriptor that is associated with the one or more first data fields and persists the first one or more first data fields when persisting the in-memory runtime data.

Abstract: Implementations of the present disclosure include providing, at each node in a set of nodes of a database system, a table partition of a plurality of table partitions, the plurality of table partitions being provided by partitioning a table using a primary key, providing, at each node in the set of nodes of the database system, a secondary index partition of a plurality of secondary index partitions, each secondary index partition including a replicate table of at least a portion of the table, the plurality of secondary index partitions being provided by partitioning the table using one or more secondary keys, and for at least one operation executed on a table partition, executing a replication protocol to replicate the at least one operation on a secondary index partition that corresponds to the table partition.

Abstract: A computer implemented method for updating a secondary database includes receiving transaction log information from a primary database system at a secondary database system. The transaction log information comprises a first marker identifying a beginning of an operation comprising a plurality of sub-operations and a second marker identifying an end of the operation. The transaction log information is sequentially parsed, a first marker is identified, and a lock state is created after replaying the first marker to ensure proper serialization of the plurality of sub-operations. The lock state is terminated after replaying the second marker. Related apparatus, systems, techniques and articles are also described.

Abstract: Provided is a system and method for improving memory management in a database. In one example, the method may include receiving a request to store a data object within a database, determining a category type associated with the data object from among a plurality of category types based on an attribute of the data object, and storing the data object via a memory pool corresponding to the determined category from among a plurality of memory pools corresponding to the plurality of respective categories, where the storing comprises allocating a first category type of data object to a first memory pool locked to main memory and allocating a second category type of data object to a second memory pool that is swapped out to disk over time. The locked memory pool can ensure that more important data items remain available even when they are the least recently used.

Abstract: The subject matter disclosed herein provides methods for the forced ordering of a dictionary based on row identifier values. A plurality of concurrent transactions can be handled in an in-memory database. At least one of the transactions can include at least one write operation to a dictionary. Each write operation can be assigned a row identifier (ID). At least one of the write operations can be written to the dictionary out of sequence. The sequence can be based on the row ID. Each row ID in the dictionary can be mapped to a corresponding value identifier in the dictionary. The dictionary can position the value identifiers so that the corresponding row IDs are in a sorted sequential order based on the row ID. Related apparatus, systems, techniques, and articles are also described.

Abstract: Computer-implemented systems and methods for categorizing tables of a database into groups are provided. A set of queries used to access a database are processed to generate a log that identifies for each of the queries (i) a database operation associated with the query, (ii) a table of the database accessed via the query, and (iii) a column or columns of the table used by the query to access the table. Database operation criteria for categorizing database tables into groups are accessed. The criteria are based on a total number of times that a table is accessed by the set of queries and a number of times that the table is accessed using a column or group of columns. An algorithm that evaluates data of the log against the criteria is applied. The algorithm is configured to automatically categorize tables identified in the log into groups based on the evaluation.

Abstract: Cracking page-loadable columns for in-memory data management is described herein. An embodiment operates by accessing a column according to a received query, determining that the received query requires a non-critical data structure associated with the column, and rebuilding the non-critical data structure from data associated with the column.

Abstract: A system and method, including determining, by a processor, a data type for each column of a database table; determining, by the processor and based on the determined data type for each column of the database table, an indication of a size of the database table; calculating, by the processor and based on the determined indication of the size of the database table, a start nbit size for a nbit compression process to be used on the database table; specifying, by the processor, the calculated start nbit size for the nbit compression process; and compressing the database table by executing the nbit data compression process using the specified start nbit size.

Abstract: A computer implemented method for manipulating data comprises receiving log transaction information from a primary database system at a secondary database system and parsing the log transaction information to identify data objects that have been modified on the primary database system. The secondary database system determines whether the data objects are present in main memory on the secondary database system. If the data objects are not in the main memory, they are copied from persistence data volumes to the main memory. The data objects are modified by replaying the log transactions on the data objects in the main memory. Then, the data objects in the persistence data volumes are modified. Related apparatus, systems, techniques and articles are also described.

Abstract: In one aspect, there is provided a method. The method may include accessing a multi-version concurrency control block providing row state for a block of rows in a table of a database, the multi-version concurrency control block including a header portion and a data portion, the header portion including a type indicator indicating whether all of the rows of the block are visible to a plurality of threads at a database management system or invisible to the plurality of threads at the database management system. Related systems, methods, and articles of manufacture are also disclosed.

Abstract: A system includes a volatile random access memory storing first header data and a first data block, the first header data including descriptive information and a first pointer to the first data block, and a non-volatile random access memory. The system includes determination of a memory size associated with the first header data and the first data block, allocation of a first memory block of the non-volatile random access memory based on the determined memory size, determination of an address of the non-volatile random access memory associated with the allocated first memory block, and writing of the descriptive information and a binary copy of the first data block at the address of the non-volatile random access memory.

Abstract: The present disclosure involves systems, software, and computer implemented methods for using declarative rules for optimized access to data. One example method includes receiving a structured query language (SQL) query, the SQL query associated with at least one database table and at least one predicate. A determination is made as to whether the SQL query is associated with at least one declarative rule based on the at least one database table and the at least one predicate. In response to determining that the SQL query is associated with at least one declarative rule, the at least one associated declarative rule is applied, during optimization of the SQL query. The at least one declarative rule defines a transformation to the SQL query to be performed during query optimization. An updated SQL query is generated based on the applied at least one associated declarative rule and the updated SQL query is provided.

Abstract: Disclosed herein are system, method, and computer program product embodiments for providing a memory optimization system for inverted indexes. An embodiment operates by determining a value identifier corresponding to a value to be searched for in a database based on a query. From a lookup table, a prefix of an offset corresponding to a location in a compressed posting list that identifies row locations where the value is found in the database is retrieved. From a directory page, a suffix of the offset corresponding to the location is retrieved. The offset is generated based on the prefix and the suffix. From the posting list, row identifiers corresponding to the generated offset are retrieved. The retrieved row identifiers are returned responsive to the query.

Abstract: A system and method, including determining, by a processor, a data type for each column of a database table; determining, by the processor and based on the determined data type for each column of the database table, an indication of a size of the database table; calculating, by the processor and based on the determined indication of the size of the database table, a start nbit size for a nbit compression process to be used on the database table; specifying, by the processor, the calculated start nbit size for the nbit compression process; and compressing the database table by executing the nbit data compression process using the specified start nbit size.

Abstract: An oldest version timestamp is received from a first database. A current commit timestamp is received from a second database configured to asynchronously replicate the first database. The current global commit timestamp corresponds to a commit transaction executed in the first database and subsequently replicated in the second database. A first test query is then issued to the second database including a request for a current timestamp of the second database. A second test query is issued to the primary database requesting records associated with a target update version that has a target timestamp equal to the current snapshot timestamp. A determination is then made that a first return associated with the first test query and a second return associated with the second test query are identical indicating a successful validation. Related apparatus, systems, techniques and articles are also described.

Abstract: First page data from one or more pages stored on a persistent storage can be loaded into a page buffer in a main system memory of one or more computing systems, and second page data that includes first dictionary values of a first dictionary for a first database column can be loaded and pinned into the page buffer in the main system memory. First contents of a first dictionary block containing first re-direction references to the first dictionary can be copied from the loaded first page data into a first in-memory array, and the first page data can be deallocated from the page buffer. The first re-direction references can direct to the first dictionary values of the first dictionary in the pinned second page data.

Abstract: A method may include iterating through at least a portion of a table stored in a database by identifying a timestamp block associated with a range of rows forming the portion of the table. In response to determining that the timestamp block is a sparse timestamp block storing timestamps associated with only some of the rows in the first range of rows, generating a bitmap. Each of the binary values in the bitmap may correspond one of the rows in the range of rows. Moreover, each of the binary values in the bitmap may indicate whether the timestamp block includes a timestamp associated with a corresponding row from the range of rows. Iterating through the range of rows may include accessing, based on the bitmap, the timestamp block to read or write timestamps of transaction affecting the rows in the range of rows.

Abstract: Provided is a database system and method in which storage is partitioned in an unbalanced format for faster access. In one example, the method may include one or more of receiving a request to store a data record, identifying a partition from among a plurality of partitions of a database based on a shard identifier in the request, automatically determining a unique range of data identifiers designated to the partition from the plurality of partitions, respectively, based on an unbalanced partitioning, determining whether the data identifier is available within the unique range of data identifiers of the identified partition, and storing the data record at the identified partition in response to determining the data identifier is available. The unbalanced partitioning according to various embodiments reduces the partitions that need to be checked during a data insert/access operation of the database.

Abstract: A method may include performing a transaction on a row in a table stored in a database. In response to performing the transaction, a first timestamp block associated with a range of rows including the row may be retrieved. In response to determining that the first timestamp block is a sparse timestamp block that has reached maximum capacity, a second timestamp block may be allocated for storing the timestamp associated with the transaction. The sparse timestamp block may be allocated with sufficient storage space for storing some but not all of the timestamps associated with the range of rows. By contrast, the second timestamp block may be a dense timestamp block allocated with sufficient storage space for storing all of the timestamps associated with the range of rows. The timestamp associated with the transaction may be stored in the second timestamp block.