Abstract: System, method, and various embodiments for providing a metadata based memory management system are described herein. An embodiment operates by determining metadata of a table of a database indicating whether data of the table is to be stored in memory or disk storage, wherein the table comprises a plurality of columns and a plurality of rows. A first column of the table is identified. A hierarchy of the metadata for the table including a column value, partition value, and table value is determined, wherein the column value is prioritized over the partition value, and wherein the partition value is prioritized over the table value. The data of the table stored in the column is loaded from the disk storage into the memory based on the hierarchy. A query is processed based on the data loaded into the memory.

Abstract: Implementations of the present disclosure include providing, at each node in a set of nodes of a database system, a table partition of a plurality of table partitions, the plurality of table partitions being provided by partitioning a table using a primary key, providing, at each node in the set of nodes of the database system, a secondary index partition of a plurality of secondary index partitions, each secondary index partition including a replicate table of at least a portion of the table, the plurality of secondary index partitions being provided by partitioning the table using one or more secondary keys, and for at least one operation executed on a table partition, executing a replication protocol to replicate the at least one operation on a secondary index partition that corresponds to the table partition.

Abstract: Techniques and solutions are described for implementing virtual table schemas. A virtual table schema includes a logical pointer that is used to target a table that includes data and is defined according to the virtual table schema. Values assigned to the logical pointer can be used to target tables at different locations, such as in a federated database system or in a cache of a database management system of a central computing system. When data associated with the virtual table is requested, or prior to receiving the request, the data can be stored in a table in the cache. The logical pointer can be updated to reference the cache. If the cache is full, the table can be removed from the cache, and the logical pointer can be updated to reference a table at the federated database system.

Abstract: Disclosed herein are system, method, and computer-program product embodiments for generating a paged and in-memory representation of a database object. An embodiment operates by maintaining in-memory and paged form primitives unique to the database object or a substructure thereof in a database such that the in-memory and paged form primitives are capable of providing the in-memory and paged representations of the database objects, respectively. Thereafter, a load configuration for the database object is determined. Based on the load configuration, the in-memory and/or paged representations of the database object are generated using the in-memory form primitive or the paged form primitive unique to the database object, respectively. Subsequently, the in-memory and/or paged representations of the database object are stored in the database.

Abstract: In one aspect, there is provided a method. The method may include accessing a multi-version concurrency control block providing row state for a block of rows in a table of a database, the multi-version concurrency control block including a header portion and a data portion, the header portion including a type indicator indicating whether all of the rows of the block are visible to a plurality of threads at a database management system or invisible to the plurality of threads at the database management system. Related systems, methods, and articles of manufacture are also disclosed.

Abstract: Disclosed herein are system, method, and computer program product embodiments for partition pruning via non-homogenous synopsis information. An embodiment operates by maintaining synopsis information for a data partition in accordance with a first synopsis strategy, monitoring performance of the synopsis information within a partition pruning system, determining that the performance of the synopsis information is insufficient, and updating the synopsis information in accordance with a second synopsis strategy better suited for the attributes of the data partition. In some embodiments, a first data partition of a partitioned data table may employ a first synopsis strategy and a second data partition of the partitioned data table may employ a second synopsis strategy.

Abstract: Disclosed herein are system, method, and computer program product embodiments for partition pruning using globally encoded range partition information. An embodiment operates by partitioning a data table into a plurality of data partitions, determining a plurality of sub-partitions within the data partitions, and determining a global range table based at least in part on value ranges corresponding to the sub-partitions. Further, the global range table may be used to determine bit vectors for the data partitions and query predicates. In some examples, the bit vectors may be used to determine whether to prune data partitions during execution of a query over a large data volume.

Abstract: Disclosed herein are system, method, and computer program product embodiments for performing fair prefetching. An embodiment operates by splitting a data vector into a first subrange and a second subrange. The embodiment performs a first chance prefetch operation on the first subrange based on a fixed number of pages, thereby loading a set of pages of the first subrange into a main memory. The embodiment performs the first chance prefetch operation on the second subrange based on the fixed number of pages, thereby loading a first set of pages of the second subrange into the main memory. The embodiment performs a second chance prefetch operation on the second subrange based on the performing the first chance prefetch operation on the second subrange, thereby loading a second set of pages of the second subrange into the main memory. The embodiment then executes the query.

Abstract: Disclosed herein are system, method, and computer-program product embodiments for generating a paged and in-memory representation of a database object. An embodiment operates by maintaining in-memory and paged form primitives unique to the database object or a substructure thereof in a database such that the in-memory and paged form primitives are capable of providing the in-memory and paged representations of the database objects, respectively. Thereafter, a load configuration for the database object is determined. Based on the load configuration, the in-memory and/or paged representations of the database object are generated using the in-memory form primitive or the paged form primitive unique to the database object, respectively. Subsequently, the in-memory and/or paged representations of the database object are stored in the database.

Abstract: System, method, and various embodiments for providing a metadata based memory management system are described herein. An embodiment operates by determining metadata of a table of a database indicating whether data of the table is to be stored in memory or disk storage, wherein the table comprises a plurality of columns and a plurality of rows. A first column of the table is identified. A hierarchy of the metadata for the table including a column value, partition value, and table value is determined, wherein the column value is prioritized over the partition value, and wherein the partition value is prioritized over the table value. The data of the table stored in the column is loaded from the disk storage into the memory based on the hierarchy. A query is processed based on the data loaded into the memory.

Abstract: A method may include iterating through at least a portion of a table stored in a database by identifying a timestamp block associated with a range of rows forming the portion of the table. In response to determining that the timestamp block is a sparse timestamp block storing timestamps associated with only some of the rows in the first range of rows, generating a bitmap. Each of the binary values in the bitmap may correspond one of the rows in the range of rows. Moreover, each of the binary values in the bitmap may indicate whether the timestamp block includes a timestamp associated with a corresponding row from the range of rows. Iterating through the range of rows may include accessing, based on the bitmap, the timestamp block to read or write timestamps of transaction affecting the rows in the range of rows.

Abstract: A method may include performing a transaction on a row in a table stored in a database. In response to performing the transaction, a first timestamp block associated with a range of rows including the row may be retrieved. In response to determining that the first timestamp block is a sparse timestamp block that has reached maximum capacity, a second timestamp block may be allocated for storing the timestamp associated with the transaction. The sparse timestamp block may be allocated with sufficient storage space for storing some but not all of the timestamps associated with the range of rows. By contrast, the second timestamp block may be a dense timestamp block allocated with sufficient storage space for storing all of the timestamps associated with the range of rows. The timestamp associated with the transaction may be stored in the second timestamp block.

Abstract: A method, a system and a computer program product for implicitly partitioning data. An index in a plurality of indexes is selected. The index corresponds to a plurality of ranges of data values stored in a plurality of database slices of a database. A partitioning structure for the selected index is generated. The partitioning structure includes a plurality of hierarchically arranged nodes. Each node corresponds to a range of data values in the plurality of ranges of data values stored and at least one database slice storing corresponding range of data values. Another range of data values is received for storage in the database. Using the partitioning structure, a node in the partitioning structure corresponding to the received range of data values and a corresponding database slice for storing the received range of data values are identified. The received range of data values is stored in the identified database slice.

Abstract: Techniques and solutions are described for implementing virtual table schemas. A virtual table schema includes a logical pointer that is used to target a table that includes data and is defined according to the virtual table schema. Values assigned to the logical pointer can be used to target tables at different locations, such as in a federated database system or in a cache of a database management system of a central computing system. When data associated with the virtual table is requested, or prior to receiving the request, the data can be stored in a table in the cache. The logical pointer can be updated to reference the cache. If the cache is full, the table can be removed from the cache, and the logical pointer can be updated to reference a table at the federated database system.

Abstract: Techniques and solutions are described for implementing virtual table schemas. A virtual table schema includes a logical pointer that is used to target a table that includes data and is defined according to the virtual table schema. Values assigned to the logical pointer can be used to target tables at different locations, such as tables located in a remote database system, replica tables containing data replicated from a table in a remote database system, or cached table data. Replica tables targeted by a logical pointer of a virtual table schema can have various locations, such as being located in main memory of a database system, being located in a dynamic storage system, or being located in a data lake. Replica tables can be transferred between locations based on various criteria.

Abstract: Techniques and solutions are described for implementing virtual table schemas. A virtual table schema includes a logical pointer that is used to target a table that includes data and is defined according to the virtual table schema. Values assigned to the logical pointer can be used to target tables at different locations, such as in-memory storage or in dynamic storage of a database management system. The dynamic storage includes an in-memory buffer cache and persistent storage. The logical pointer can be updated to point to the in-memory storage or the dynamic storage, including based on use statistics. Similarly, a table having data for the virtual table can be transitioned between the in-memory buffer cache and the persistent storage, including based on use statistics for the virtual table. A value for the logical pointer can be updated when the table is transferred between the in-memory buffer cache and the dynamic storage.

Abstract: Disclosed herein are system, method, and computer program product embodiments for providing a memory optimization system for inverted indexes. An embodiment operates by determining a value identifier corresponding to a value to be searched for in a database based on a query. From a lookup table, a prefix of an offset corresponding to a location in a compressed posting list that identifies row locations where the value is found in the database is retrieved. From a directory page, a suffix of the offset corresponding to the location is retrieved. The offset is generated based on the prefix and the suffix. From the posting list, row identifiers corresponding to the generated offset are retrieved. The retrieved row identifiers are returned responsive to the query.

Abstract: Persisting data in logical tiers of an in-memory database based on aging rules of the database system. The database system can be configured to age data by moving the data from a first logical tier supporting high performance data access to a second logical tier supporting lower performance data access, thereby improving the operational efficiency of the database system.

Abstract: Provided is a system and method for improving memory management in a database. In one example, the method may include receiving a request to store a data object within a database, determining a category type associated with the data object from among a plurality of category types based on an attribute of the data object, and storing the data object via a memory pool corresponding to the determined category from among a plurality of memory pools corresponding to the plurality of respective categories, where the storing comprises allocating a first category type of data object to a first memory pool locked to main memory and allocating a second category type of data object to a second memory pool that is swapped out to disk over time. The locked memory pool can ensure that more important data items remain available even when they are the least recently used.

Abstract: A method, a system and a computer program product for implicitly partitioning data. An index in a plurality of indexes is selected. The index corresponds to a plurality of ranges of data values stored in a plurality of database slices of a database. A partitioning structure for the selected index is generated. The partitioning structure includes a plurality of hierarchically arranged nodes. Each node corresponds to a range of data values in the plurality of ranges of data values stored and at least one database slice storing corresponding range of data values. Another range of data values is received for storage in the database. Using the partitioning structure, a node in the partitioning structure corresponding to the received range of data values and a corresponding database slice for storing the received range of data values are identified. The received range of data values is stored in the identified database slice.