Abstract: Disclosed herein is a method of managing data in a database management system comprising a first, master storage, node and a second node for performing operations on data stored in the database. The method comprises configuring the second node to perform a set of steps. The second node is configured to write data indicative of one or more operations on data to a set of log files. The second node segments the set of log files into a set of log file chunks For each of the log file chunks, the second node may transmit data from the database associated with a log file chunk to the first node and thereafter transmit the log file chunk to the first node. The segmenting and transmitting of the set of log files and their associated data may be performed asynchronously to the updating of the set of log files.

Abstract: A database management system configured to perform a computer-implemented method is provided. The method involving configuring a first set of nodes to determine an instantaneous state of the database management system, generate system configuration data, and provide the system configuration data to a second set of nodes of the database management system, wherein the configuration data is replicated amongst the first set of nodes according to a consensus-based replication protocol so as to maintain the system configuration data. The method also involves configuring the second set of nodes to receive the system configuration data and implement a state machine that is arranged to modify a configuration of the second set of nodes according to the system configuration data, wherein the second set of nodes are arranged to replicate user data according to a non-consensus-based replication protocol.

Abstract: Disclosed herein is a method of managing data in a database management system comprising a first, master storage, node and a second node for performing operations on data stored in the database. The method comprises configuring the second node to perform a set of steps. The second node is configured to write data indicative of one or more operations on data to a set of log files. The second node segments the set of log files into a set of log file chunks For each of the log file chunks, the second node may transmit data from the database associated with a log file chunk to the first node and thereafter transmit the log file chunk to the first node. The segmenting and transmitting of the set of log files and their associated data may be performed asynchronously to the updating of the set of log files.

Abstract: A database is partitioned into a plurality of sorted runs, wherein each sorted run includes a set of sorted segments of data records. One of the sorted runs preferably includes more than half of the records of the database, and the other sorted runs are progressively smaller. A query is processed by searching each of the sorted runs. Writes are effected by appending a new sorted run to the database. Sorted merges are used to combine the smaller sorted runs. Deletions are effected by marking the deleted record in the sorted run. Modifications are effected by deleting the original record and writing the modified record to the database. The larger sorted runs are only re-sorted or merged when the sorted run includes a substantial number of deletions. Two merge queues are maintained to enable rapid merges of the smaller sorted runs while a merger of larger sorted runs are occurring.

Abstract: A set of six atomic primitives are provided for a database management system that may be used in combination to provide all of the common features and functions of a clustered database, including data failover, auto-healing, and elastic scaling. These six atomic primitives include CREATE, DROP, DETACH, ATTACH, COPY, and PROMOTE. Of particular note, it is shown that by maintaining appropriate metadata, including the status of each instance of each partition in the cluster, the versatility and reliability of this set of primitives is sufficient to implement each of the aforementioned data failover, auto-healing, and elastic scaling functions with high efficiency using a minimal number of these primitives. Each primitive is atomic (such that the cluster clearly in one state or another) and online (a workload of reads and writes is uninterrupted while the primitive runs), and each primitive is scoped to a single partition of data, thereby enabling parallel processing.

Abstract: A set of six atomic primitives are provided for a database management system that may be used in combination to provide all of the common features and functions of a clustered database, including data failover, auto-healing, and elastic scaling. These six atomic primitives include CREATE, DROP, DETACH, ATTACH, COPY, and PROMOTE. Of particular note, it is shown that by maintaining appropriate metadata, including the status of each instance of each partition in the cluster, the versatility and reliability of this set of primitives is sufficient to implement each of the aforementioned data failover, auto-healing, and elastic scaling functions with high efficiency using a minimal number of these primitives. Each primitive is atomic (such that the cluster clearly in one state or another) and online (a workload of reads and writes is uninterrupted while the primitive runs), and each primitive is scoped to a single partition of data, thereby enabling parallel processing.

Abstract: A database is partitioned into a plurality of sorted runs, wherein each sorted run includes a set of sorted segments of data records. One of the sorted runs preferably includes more than half of the records of the database, and the other sorted runs are progressively smaller. A query is processed by searching each of the sorted runs. Writes are effected by appending a new sorted run to the database. Sorted merges are used to combine the smaller sorted runs. Deletions are effected by marking the deleted record in the sorted run. Modifications are effected by deleting the original record and writing the modified record to the database. The larger sorted runs are only re-sorted or merged when the sorted run includes a substantial number of deletions. Two merge queues are maintained to enable rapid merges of the smaller sorted runs while a merger of larger sorted runs are occurring.

Abstract: A system and method for executing query plans preprocesses a user's queries to identify and extract select parameters of the query, creates a skeletal query with reference to the extracted parameters, creates and compiles an executable version of the skeletal query, then executes the compiled version of the skeletal query using the particular parameters in the current query. The compiled version of the parameterized skeletal query is stored, and when another query is submitted that matches the skeletal query (but with potentially different parameters), the previously compiled skeleton query is executed with the parameters of this latter query.

Abstract: A database durability implementation records only committed transactions in a log file. A pair of log files and a pair of snapshot files are maintained. When a snapshot of the database is completed, the ‘current’ log becomes the ‘prior’ log and the other log becomes the ‘current’ log. After the next snapshot is completed, the prior log and prior snapshot may be deleted. Transactions that are not committed are not recorded in the current log, thereby avoiding the need to undo aborted transactions. If a given change is reflected in a completed snapshot, it does not appear in either of the logs; if the change is not yet reflected in a completed snapshot, it is guaranteed to be stored in one of the logs. During recovery, the system assesses both snapshots. The most recent of the completed snapshots is used, and the corresponding log(s) is (are) applied.

Abstract: Methods, systems, and computer program products are provided for deriving and updating document similarity indices for a plurality of documents. The number of maintained similarities can be controlled to conserve CPU and storage resources.

Abstract: Methods, systems, and computer program products are provided for deriving and updating document similarity indices for a plurality of documents. The number of maintained similarities can be controlled to conserve CPU and storage resources.

Abstract: A database durability implementation records only committed transactions in a log file. A pair of log files and a pair of snapshot files are maintained. When a snapshot of the database is completed, the ‘current’ log becomes the ‘prior’ log and the other log becomes the ‘current’ log. After the next snapshot is completed, the prior log and prior snapshot may be deleted. Transactions that are not committed are not recorded in the current log, thereby avoiding the need to undo aborted transactions. If a given change is reflected in a completed snapshot, it does not appear in either of the logs; if the change is not yet reflected in a completed snapshot, it is guaranteed to be stored in one of the logs. During recovery, the system assesses both snapshots. The most recent of the completed snapshots is used, and the corresponding log(s) is (are) applied.

Abstract: A system and method for executing query plans preprocesses a user's queries to identify and extract select parameters of the query, creates a skeletal query with reference to the extracted parameters, creates and compiles an executable version of the skeletal query, then executes the compiled version of the skeletal query using the particular parameters in the current query. The compiled version of the parameterized skeletal query is stored, and when another query is submitted that matches the skeletal query (but with potentially different parameters), the previously compiled skeleton query is executed with the parameters of this latter query.

Abstract: The present invention extends to methods, systems, and computer program products for deriving document similarity indices. Embodiments of the invention include scalable and efficient mechanisms for deriving and updating a document similarity index for a plurality of documents. The number of maintained similarities can be controlled to conserve CPU and storage resources.

Abstract: Embodiments are directed to efficiently determining which cache entries are to be evicted from memory and to incorporating a probability of reuse estimation in a cache entry eviction determination. A computer system with multiple different caches accesses a cache entry. The computer system determines an entry cost value for the accessed cache entry. The entry cost value indicates an amount of time the computer system is slowed down by to load the cache entry into cache memory. The computer system determines an opportunity cost value for the computing system caches. The opportunity cost value indicates an amount of time by which the computer system is slowed down while performing other operations that could have used the cache entry's cache memory space. Upon determining that the entry cost value is lower than the opportunity cost value, the computer system probabilistically evicts the cache entry from cache memory.

Abstract: Embodiments are directed to efficiently determining which cache entries are to be evicted from memory and to incorporating a probability of reuse estimation in a cache entry eviction determination. A computer system with multiple different caches accesses a cache entry. The computer system determines an entry cost value for the accessed cache entry. The entry cost value indicates an amount of time the computer system is slowed down by to load the cache entry into cache memory. The computer system determines an opportunity cost value for the computing system caches. The opportunity cost value indicates an amount of time by which the computer system is slowed down while performing other operations that could have used the cache entry's cache memory space. Upon determining that the entry cost value is lower than the opportunity cost value, the computer system probabilistically evicts the cache entry from cache memory.

Abstract: The present invention extends to methods, systems, and computer program products for deriving document similarity indices. Embodiments of the invention include scalable and efficient mechanisms for deriving and updating a document similarity index for a plurality of documents. The number of maintained similarities can be controlled to conserve CPU and storage resources.