Abstract: As part of a columnar in-memory database, value identifiers are inserted into a backing array in-memory until such time that it is determined that such backing array does not have sufficient capacity. A new backing array is then generated that includes the value identifiers in the old backing array and which has sufficient capacity. The old backing array can be flushed from memory when there are no active operations using such backing array. Such an arrangement allows for readers and non-structural writers to operate concurrently.

Abstract: Disclosed herein are innovations in memory management and data recovery for systems that operate using storage class memory (SCM), such as non-volatile RAM (NVRAM). The disclosed innovations have particular application to production database systems, where reducing database downtime in the event of a system crash is highly desirable. Embodiments of the disclosed technology can address a variety of problems that exist during a system crash. For example, embodiments of the disclosed technology can be used to address the loss of the physical memory mapping and/or the loss of the CPU cache that typically occurs in the event of a system crash. Furthermore, embodiments of the disclosed technology can be used to prevent data inconsistency and/or memory leak problems that may arise in the event of a system crash.

Abstract: A system for memory allocation and deallocation with a multi-level memory map is provided. In some implementations, the system performs operations comprising allocating a memory map for addressing a plurality of memory locations in a heap, the memory map comprising a root node, one or more second-level nodes, and a plurality of third-level nodes. The plurality of third-level nodes can comprise third entries for pointing to the memory locations and/or the one or more second-level nodes can comprise a plurality of second entries corresponding to the plurality of third-level nodes. The operations can further include determining a location to store data within the heap and/or tracking the location by placing a pointer within a third-level node of the plurality of third-level nodes and incrementing a counter corresponding to the third-level node. Related systems, methods, and articles of manufacture are also described.

Abstract: A system for memory allocation and deallocation with a multi-level memory map is provided. In some implementations, the system performs operations comprising allocating a memory map for addressing a plurality of memory locations in a heap, the memory map comprising a root node, one or more second-level nodes, and a plurality of third-level nodes. The plurality of third-level nodes can comprise third entries for pointing to the memory locations and/or the one or more second-level nodes can comprise a plurality of second entries corresponding to the plurality of third-level nodes. The operations can further include determining a location to store data within the heap and/or tracking the location by placing a pointer within a third-level node of the plurality of third-level nodes and incrementing a counter corresponding to the third-level node. Related systems, methods, and articles of manufacture are also described.

Abstract: A generic state machine concept can be used to decouple specific state machine functionality from any specific consensus protocol. A generic state machine can prepare a local log record and generate a local log index in response to a data update received at a computing node of a plurality of computing nodes in a distributed data storage application. The generic state machine can pass replication information to one or more other computing nodes in the plurality of computing nodes in accordance with the consensus protocol and can mutate a state of the generic state machine upon the plurality of computing nodes achieving a quorum.

Abstract: A system for allocating memory (e.g., heap) in multi-core processors is provided. In some implementations, the system performs operations comprising receiving, at a shared cache having a plurality of segments, a first data allocation including a plurality of data blocks, and allocating at least a first and second data block from the first allocation. First and second segments in the shared cache can each comprise a plurality of data slots (e.g., of equal length). Allocating the first and second data blocks can include storing the first data block in a data slot of the first segment and storing the second data block in a data slot of the second segment. The plurality of data slots which do not contain data may contain padding, and/or the data slots to which the first and second data blocks are allocated are not adjacent. Related systems, methods, and articles of manufacture are also described.

Abstract: Techniques for handling large string values in a string dictionary within an in-memory database system are presented. The string dictionary can be a paged-column main dictionary. For example, a value block of a dictionary may be loaded into memory, where the dictionary stores values for dictionary compression. The value block may include a first portion of a large string value and one or more logical pointers to one or more large string pages containing a remainder of the large string value. The large string value may be materialized into a contiguous location in memory, and a pointer to the materialized large string value may be created.

Abstract: A request for an entry of a dictionary having multiple dictionary blocks may be received, where the dictionary stores string values associated with corresponding value identifiers (IDs) for dictionary compression. One of the dictionary blocks may be selectively loaded into memory, and the dictionary block that has been loaded into memory may be searched. A value ID directory may be constructed in memory, where the value ID directory includes last value IDs for the dictionary blocks, and each of the last value IDs is mapped to an index of one of the dictionary blocks that includes a string value for that last value ID. A separator directory may also be constructed in memory, where the separator directory includes separators for the dictionary blocks, and each of the separators is mapped to an index of one of the dictionary blocks that includes a string value for that separator.

Abstract: Disclosed herein are system, method, and computer program product embodiments for inverted indexing. An embodiment operates by receiving a request to add a new document identifier (doc ID) to an inverted index. A memory block corresponding to where the doc ID is to be added is determined or located, and the available space remaining in the memory block is determined. If the memory block has available space, the doc ID is added to the memory block.

Abstract: Loading of table metadata into memory of an in-memory database is initiated. The table metadata is persisted across pages in a page chain. Thereafter, a plurality of metadata objects are materialized into memory that each include an object handle pinning an underlying persisted page in the page chain. The objects are populated with data from the underlying persisted pages. Subsequently, for one to many object relationships, a vector of object handles is generated that comprises a plurality of transient handles that each point to a different instance of a respective transient object. Alternatively, for one to one object relationships or many to one object relationships, an object handle to point to a respective linked object is generated. Related apparatus, systems, techniques and articles are also described.

Abstract: Disclosed herein are system, method, and computer program product embodiments for determining row visibility states. An embodiment operates by initializing a result bitmap for rows based on visible row state values of row state bitmaps corresponding to the rows, wherein values of the result bitmap indicate whether the row are visible or not visible; determining one or more rows whose corresponding visible row state values indicate that the one or more rows are not visible; and determining whether to update the result bitmap to indicate that the one or more rows are visible based on one or more versioned row state values, wherein the one or more versioned row state values indicate whether the corresponding rows are not visible or possibly visible.

Abstract: Disclosed herein are system, method, and computer program product embodiments for performing garbage collection on a multi-version concurrency control information in the database management system. An embodiment operates by determining, using multi-version concurrency control (MVCC) information, when a row manipulated by a write transaction is visible to a plurality of readers accessing a table that includes the row. The MVCC information for the row includes at least a creation timestamp, a destruction timestamp and a row state. Once the row is visible to the plurality of readers, garbage collecting at least the creation timestamp or the destruction timestamp in the MVCC information. After the creation timestamp or destruction timestamp have been garbage collected, the plurality of readers use the row state to determine accessibility of the row in the table.

Abstract: A central processing unit (CPU) forming part of a computing device, initiates execution of code associated with each of a plurality of objects used by a worker thread. The CPU has an associated cache that is split into a plurality of slices. It is determined, by a cache slice allocation algorithm for each object, whether any of the slices will be exclusive to or shared by the object. Thereafter, for each object, any slices determined to be exclusive to the object are activated such that the object exclusively uses such slices and any slices determined to be shared by the object are activated such that the object shares or is configured to share such slices.

Abstract: Each of a plurality of accesses by a multithreaded program to shared data structures stored within a database is monitored. The accesses are implemented by varying application programming interface (API) methods. Thereafter, it is determined, based on pre-defined synchronization safeguards, whether each of the accesses is valid or invalid based on the utilized corresponding API method. Those accesses to the shared data structures that were determined to be valid are allowed to proceed while those accesses to the shared data structures that were determined to be invalid are prevented from proceeding.

Abstract: The subject matter disclosed herein provides methods for determining the page on which a desired row position value is located. A table having a plurality of rows can be accessed. The rows can be distributed across one or more pages in an in-memory database. Each row can be associated with a unique row position value. Each page can be associated with a capacity that is representative of an amount of data stored on the page. A capacity index having a plurality of entries can be created to record changes in capacity between pages. Neighboring entries in the capacity index can have a different capacity. Each entry in the capacity index can correspond to a page. A page directory can be created based on the capacity index. The page directory can indicate all possible row position values associated with each page. Related apparatus, systems, techniques, and articles are also described.

Abstract: The subject matter disclosed herein provides methods for the dual storage of data using an in-memory array and an on-disk page structure. An in-memory array holding a column of data can be maintained. One or more pages can be maintained. Each of the one or more pages can have one or more rows for storing the column of data. Random access can be provided to a subset of the one or more rows by at least loading the subset of rows from the one or more pages to the in-memory array without loading all of the rows from the one or more pages. Related apparatus, systems, techniques, and articles are also described.

Abstract: A plurality of free list data structures are maintained in a multi-processor computing system that each correspond to one processor of the multi-processor computing system and that each comprise an ordered queue of processor-specific items. Thereafter, a number of processor-specific items allocated to each free list data structure is calculated. Processor-specific items allocated to a first of the free list data structures are moved to a second of the free list data structures when the number of calculated processor-specific items in the first free data structure exceeds a first threshold. In addition, processor-specific items allocated to the second of the free list data structures are moved to the first of the free list data structures when the number of calculated processor-specific items in the first free data structure is below a second threshold. Related apparatus, systems, techniques and articles are also described.

Abstract: A system includes a first node including a first processor and a first memory, and a second node including a second processor and a second memory. A worker thread of the first processor calls an allocation API to allocate a portion of the first memory, and calls a first allocator of a first memory manager associated with the first node to allocate a specified size of the first memory. The first memory manager calls an operating system function to reserve a memory segment of the specified size, sets a first Non-Uniform Memory Access policy of the first node to preferred, binds the reserved memory segment to the first node, adds the memory segment to a global freelist, and returns an address pointer of the reserved memory segment to the worker thread.

Abstract: Lockless allocation of a block from a page is initiated by computing a needed block size from an original block header value. Thereafter, a currently used block size and a capacity of the page are determined. A new block size is then computed based on the currently used block size and the needed block size and a remaining capacity of the page. A new block header value is subsequently generated that includes the determined new block size and the determined remaining capacity of the page The original block header value is then replaced with the new block header value. Related apparatus, systems, techniques and articles are also described.

Abstract: A database operation is initiated in an in-memory database that stores persistent data in a secondary storage and has at least one associated log. The database operation involves data having a size above a pre-defined threshold. Thereafter, the data is split into a plurality of chunks and the database operation is executed by parallelizing each chunk over a plurality of hierarchically arranged threads. Related systems, apparatus, methods and computer program products are also described.