Abstract: Methods for local page writes via pre-staging buffers for resilient buffer pool extensions are performed by computing systems. Compute nodes in database systems insert, update, and query data pages maintained in storage nodes. Data pages cached locally by compute node buffer pools are provided to buffer pool extensions on local disks as pre-copies via staging buffers that store data pages prior to local disk storage. Encryption of data pages occurs at the staging buffers, which allows a less restrictive update latching during the copy process, with page metadata being updated in buffer pool extensions page tables with in-progress states indicating it is not yet written to local disk. When stage buffers are filled, data pages are written to buffer pool extensions and metadata is updated in page tables to indicate available/valid states. Data pages in staging buffers can be read and updated prior to writing to the local disk.

Abstract: Methods for local page writes via pre-staging buffers for resilient buffer pool extensions are performed by computing systems. Compute nodes in database systems insert, update, and query data pages maintained in storage nodes. Data pages cached locally by compute node buffer pools are provided to buffer pool extensions on local disks as pre-copies via staging buffers that store data pages prior to local disk storage. Encryption of data pages occurs at the staging buffers, which allows a less restrictive update latching during the copy process, with page metadata being updated in buffer pool extensions page tables with in-progress states indicating it is not yet written to local disk. When stage buffers are filled, data pages are written to buffer pool extensions and metadata is updated in page tables to indicate available/valid states. Data pages in staging buffers can be read and updated prior to writing to the local disk.

Abstract: Methods for local page writes via pre-staging buffers for resilient buffer pool extensions are performed by computing systems. Compute nodes in database systems insert, update, and query data pages maintained in storage nodes. Data pages cached locally by compute node buffer pools are provided to buffer pool extensions on local disks as pre-copies via staging buffers that store data pages prior to local disk storage. Encryption of data pages occurs at the staging buffers, which allows a less restrictive update latching during the copy process, with page metadata being updated in buffer pool extensions page tables with in-progress states indicating it is not yet written to local disk. When stage buffers are filled, data pages are written to buffer pool extensions and metadata is updated in page tables to indicate available/valid states. Data pages in staging buffers can be read and updated prior to writing to the local disk.

Abstract: Disclosed herein are system, method, and computer program product embodiments for storing a large object (LOB) in a database. An embodiment operates by identifying the LOB to be stored in an on-disk store of the database and determining the size of the LOB. Thereafter, an index vector for the on-disk store to include an identifier corresponding to the LOB is created. Additionally, a dictionary for the on-disk store to include a copy of the identifier and corresponding LOB data is also created. The LOB data corresponds to the LOB itself or a LOB location inside of the on-disk store based on the LOB's size.

Abstract: Disclosed herein are system, method, and computer program product embodiments for converting column loadable data in a database into a page loadable format. When a column oriented in-memory database needs to perform an operation on a table, the affected columns must be loaded fully into memory. In some cases, a database may partition tables, allowing for column fragments corresponding to those partitions to be loaded into memory. These column fragments may exceed available memory, such that it is beneficial to convert the column fragment into a page loadable format. Approaches are discussed herein for performing the conversion without the need to update global metadata during the process, allowing for multiple parallel column fragments to be updated, allowing data manipulation language (DML) operations to continue on the column during the conversion process, and reducing the impact of blocked parallel savepoint operations during the conversion process by chunking the conversion process.

Abstract: System, method, and various embodiments for providing a metadata converter and memory management system are described herein. An embodiment operates by determining that first metadata corresponding to a table of a database comprising load preferences for a column level for a plurality of columns of the table, wherein the load preferences include either column load or page load. It is determined that the database is enabled with both load preferences for a table level and load preferences partition level, in addition to load preferences for the column level. Values for the load preferences are automatically assigned for both the table level and the partition level in second metadata, and wherein the second metadata preserves the load preferences for the column level of the first metadata. A query against the table based on load preferences from the second metadata.

Abstract: System, method, and various embodiments for providing a metadata based memory management system are described herein. An embodiment operates by determining metadata of a table of a database indicating whether data of the table is to be stored in memory or disk storage, wherein the table comprises a plurality of columns and a plurality of rows. A first column of the table is identified. A hierarchy of the metadata for the table including a column value, partition value, and table value is determined, wherein the column value is prioritized over the partition value, and wherein the partition value is prioritized over the table value. The data of the table stored in the column is loaded from the disk storage into the memory based on the hierarchy. A query is processed based on the data loaded into the memory.

Abstract: A computer implemented method for manipulating data comprises receiving log transaction information from a primary database system at a secondary database system and parsing the log transaction information to identify data objects that have been modified on the primary database system. The secondary database system determines whether the data objects are present in main memory on the secondary database system and whether the log transaction corresponds to a data manipulation language (DML) query or a data definition language (DDL) query. If the data objects are not in the main memory and the query is a DML query, they are copied from persistence data volumes to the main memory. The data objects in main memory are modified by replaying the log transactions on the data objects in the main memory. Then, the data objects in the persistence data volumes are modified. Related apparatus, systems, techniques and articles are also described.

Abstract: System, method, and various embodiments for providing a metadata converter and memory management system are described herein. An embodiment operates by determining that first metadata corresponding to a table of a database comprising load preferences for a column level for a plurality of columns of the table, wherein the load preferences include either column load or page load. It is determined that the database is enabled with both load preferences for a table level and load preferences partition level, in addition to load preferences for the column level. Values for the load preferences are automatically assigned for both the table level and the partition level in second metadata, and wherein the second metadata preserves the load preferences for the column level of the first metadata. A query against the table based on load preferences from the second metadata.

Abstract: System, method, and various embodiments for providing a metadata based memory management system are described herein. An embodiment operates by determining metadata of a table of a database indicating whether data of the table is to be stored in memory or disk storage, wherein the table comprises a plurality of columns and a plurality of rows. A first column of the table is identified. A hierarchy of the metadata for the table including a column value, partition value, and table value is determined, wherein the column value is prioritized over the partition value, and wherein the partition value is prioritized over the table value. The data of the table stored in the column is loaded from the disk storage into the memory based on the hierarchy. A query is processed based on the data loaded into the memory.

Abstract: Disclosed herein are system, method, and computer program product embodiments for identifying and loading a relevant page of a dictionary into temporary memory. An embodiment operates by receiving a query to be executed. The query includes a value for executing the query. The server queries a dictionary to retrieve a value ID. The server executes a binary search on a helper vector of the dictionary based on the value. The helper vector includes a last value for each page of a dictionary. The server identifies a page of the dictionary including the value. The server loads the page into temporary memory and retrieves the value ID of the value from the page. The server executes the query on a column using the value ID.

Abstract: Disclosed herein are system, method, and computer program product embodiments for converting column loadable data in a database into a page loadable format. When a column oriented in-memory database needs to perform an operation on a table, the affected columns must be loaded fully into memory. In some cases, a database may partition tables, allowing for column fragments corresponding to those partitions to be loaded into memory. These column fragments may exceed available memory, such that it is beneficial to convert the column fragment into a page loadable format. Approaches are discussed herein for performing the conversion without the need to update global metadata during the process, allowing for multiple parallel column fragments to be updated, allowing data manipulation language (DML) operations to continue on the column during the conversion process, and reducing the impact of blocked parallel savepoint operations during the conversion process by chunking the conversion process.

Abstract: Disclosed herein are system, method, and computer program product embodiments for storing a large object (LOB) in a database. An embodiment operates by identifying the LOB to be stored in an on-disk store of the database and determining the size of the LOB. Thereafter, an index vector for the on-disk store to include an identifier corresponding to the LOB is created. Additionally, a dictionary for the on-disk store to include a copy of the identifier and corresponding LOB data is also created. The LOB data corresponds to the LOB itself or a LOB location inside of the on-disk store based on the LOB's size.

Abstract: Systems, apparatus, techniques, articles, and methods include a unified table container for storing data and metadata in pages organized in one or more page chains. A determination is made as to whether the query includes a data definition statement. In response to determining that the query includes a data definition statement, in-memory runtime data is persisted to a persistent volume that is configured to be accessed by the database system. The in-memory runtime data includes one or more first data fields and one or more second data fields that are different from the one or more first data fields. Furthermore, the unified table container includes a persistent runtime data descriptor that is associated with the one or more first data fields and persists the first one or more first data fields when persisting the in-memory runtime data.

Abstract: Objects across multiple database manipulation language (DML) redo log records are cached, during continuous replay of redo log records on a secondary system, for the same table partition to enable reuse of such cached objects. Later, these cached objects can be reused as they are accessed during the sequential processing of DML redo records for a specific table partition.

Abstract: A computer implemented method for updating a secondary database includes receiving transaction log information from a primary database system at a secondary database system. The transaction log information comprises a first marker identifying a beginning of an operation comprising a plurality of sub-operations and a second marker identifying an end of the operation. The transaction log information is sequentially parsed, a first marker is identified, and a lock state is created after replaying the first marker to ensure proper serialization of the plurality of sub-operations. The lock state is terminated after replaying the second marker. Related apparatus, systems, techniques and articles are also described.

Abstract: A computer implemented method for initializing a secondary database system includes receiving table state information from a primary database system at a secondary database system. The table state information includes information identifying which tables had an open operation during a savepoint event. Metadata associated with the tables is parsed to identify table state information from the metadata and a lock is created for each table identified as having an open operation during the savepoint event. Afterwards, log transaction information is sequentially parsed. Related apparatus, systems, techniques and articles are also described.

Abstract: System and methods for persisting in-memory runtime data within a database system include receiving a query at a database system that includes a unified table container and determining whether the query includes a data definition statement. In response to determining that the query includes a data definition statement, in-memory runtime data is persisted to a persistent volume that is configured to be accessed by the database system. The in-memory runtime data includes one or more first data fields and one or more second data fields that are different from the one or more first data fields. Furthermore, the unified table container includes a persistent runtime data descriptor that is associated with the one or more first data fields and persists the first one or more first data fields when persisting the in-memory runtime data. Related apparatus, systems, techniques and articles are also described.

Abstract: Objects across multiple database manipulation language (DML) redo log records are cached, during continuous replay of redo log records on a secondary system, for the same table partition to enable reuse of such cached objects. Later, these cached objects can be reused as they are accessed during the sequential processing of DML redo records for a specific table partition.

Abstract: A computer implemented method for initializing a secondary database system includes receiving table state information from a primary database system at a secondary database system. The table state information includes information identifying which tables had an open operation during a savepoint event. Metadata associated with the tables is parsed to identify table state information from the metadata and a lock is created for each table identified as having an open operation during the savepoint event. Afterwards, log transaction information is sequentially parsed. Related apparatus, systems, techniques and articles are also described.