Abstract: Techniques are described herein for achieving physically dispersed item-dispersal patterns in queue tables that are used as circular buffers in a FIFO system. The techniques achieve the physically dispersed item-dispersal patterns while retaining FIFO semantics, efficient data ingest, efficient data access, and low latency for both fixed and variable-size ingest data. Because consecutively inserted rows are written to physically dispersed locations on persistent storage, lock contention between concurrent ingest sessions is significantly reduced. Once a physically dispersed item-dispersal pattern is established for a first set of inserted rows, subsequent rows are added by overwriting the existing rows. The physical dispersal of the initial rows may be accomplished by setting values in a slot number column such that updates that are made in a sequence that is based on the slot number column exhibit a desired physical dispersal pattern.

Abstract: A ledgered repository of persistent data objects is replicated on a network of persistent storage systems (PSSs) by transactions recorded across multiple blockchains. The blockchains are replicated on each of the PSSs. Using multiple blockchains enables greater parallelism; however, use of the multiple blockchains requires using measures that ensure that transactions distributed across multiple blockchains are applied in way that ensures a level of transactional consistency. Furthermore, the measures are efficient, thereby reducing overhead of maintaining a level of transactional consistency and increasing throughput of applying the transactions using multiple blockchains.

Abstract: A ledgered repository of persistent data objects is replicated on a network of persistent storage systems (PSSs) by transactions recorded across multiple blockchains. The blockchains are replicated on each of the PSSs. Using multiple blockchains enables greater parallelism; however, use of the multiple blockchains requires using measures that ensure that transactions distributed across multiple blockchains are applied in way that ensures a level of transactional consistency. Furthermore, the measures are efficient, thereby reducing overhead of maintaining a level of transactional consistency and increasing throughput of applying the transactions using multiple blockchains.

Abstract: Herein are high throughput techniques for incorporating cryptographic blockchains into relational data. In an embodiment, a computer indicates, in a database dictionary in a database, that a relational table is for blockchain storage. The relational table contains application column(s). In response to that indication, the relational table is associated with system column(s) that are unmodifiable by administrators and clients of the database. The system column(s) include a cryptographic hash column. A request to store a particular value in a particular application column is received from a client. In response to receiving the request, a cryptographic hash value is calculated for a new row for the relational table. In the relational table, the computer stores the new row that contains: the particular value in the particular application column, and the cryptographic hash value in the cryptographic hash column.

Abstract: Distributed ledgered data is stored within a distributed persistent storage system comprising multiple persistent storage systems as distributed ledgered participants. In various embodiments, the distributed ledgered data is maintained using the native capabilities of a persistent storage system. The distributed ledgered data is replicated as persistent data objects in a “ledgered repository of objects” that are replicated at each of the persistent storage systems. Changes at one persistent storage system are recorded within a block in a distributed blockchain that is distributed across each of the other distributed ledgered participants. The other distributed ledgered participants read the changes from the blockchain and apply the changes to the respective replicas at each of the other distributed ledgered participants. Hence, this approach is referred to as blockchain apply. Blockchain apply may be used to replicate the repository objects of various forms of PSSs.

Abstract: Update-anywhere replication of queuing operations on a replicated message queue is performed. A dequeue ready time (“ready time”) is associated by each participating persistent storage server with a queue message to be dequeued. Unless a queue message is already locked by a distributed dequeue transaction, a participating leader PSS initiates a distributed dequeue transaction for the queue message once the ready time for the queue message is reached, subject to certain conditions. An initiator PSS is in effect designated for a queue message; the initiator PSS associates a desired ready time for the queue message. The designated PSS is referred to herein as the primary leader PSS and the ready time the primary PSS associates with queue message is referred as the primary ready time. The other participating leader PSSs are backup leader PSSs which serve as backups for dequeuing a queue message. Each backup leader PSS associates a later “backup ready time” with the queue message.

Abstract: Update-anywhere replication of queuing operations on a replicated message queue is performed. A dequeue ready time (“ready time”) is associated by each participating persistent storage server with a queue message to be dequeued. Unless a queue message is already locked by a distributed dequeue transaction, a participating leader PSS initiates a distributed dequeue transaction for the queue message once the ready time for the queue message is reached, subject to certain conditions. An initiator PSS is in effect designated for a queue message; the initiator PSS associates a desired ready time for the queue message. The designated PSS is referred to herein as the primary leader PSS and the ready time the primary PSS associates with queue message is referred as the primary ready time. The other participating leader PSSs are backup leader PSSs which serve as backups for dequeuing a queue message. Each backup leader PSS associates a later “backup ready time” with the queue message.

Abstract: A ledgered repository of persistent data objects is replicated on a network of persistent storage systems (PSSs) by transactions recorded across multiple blockchains. The blockchains are replicated on each of the PSSs. Using multiple blockchains enables greater parallelism; however, use of the multiple blockchains requires using measures that ensure that transactions distributed across multiple blockchains are applied in way that ensures a level of transactional consistency. Furthermore, the measures are efficient, thereby reducing overhead of maintaining a level of transactional consistency and increasing throughput of applying the transactions using multiple blockchains.

Abstract: A method is provided for space-efficient bookkeeping using bit-level locking. Each row of a bookkeeping table stores binary data comprising a plurality of bits. Each bit of the plurality of bits corresponds to a particular message and a particular consumer of the message, and indicates whether the particular message has been dequeued by the particular consumer. The bookkeeping table is stored on-disk and data corresponding to the bookkeeping table is stored in volatile memory. When a message is dequeued the in-memory bookkeeping data is updated, and the bookkeeping table is updated based on the in-memory bookkeeping data when the dequeue is committed.

Abstract: Herein are high throughput techniques for incorporating cryptographic blockchains into relational data. In an embodiment, a computer indicates, in a database dictionary in a database, that a relational table is for blockchain storage. The relational table contains application column(s). In response to that indication, the relational table is associated with system column(s) that are unmodifiable by administrators and clients of the database. The system column(s) include a cryptographic hash column. A request to store a particular value in a particular application column is received from a client. In response to receiving the request, a cryptographic hash value is calculated for a new row for the relational table. In the relational table, the computer stores the new row that contains: the particular value in the particular application column, and the cryptographic hash value in the cryptographic hash column.

Abstract: Distributed ledgered data is stored within a distributed persistent storage system comprising multiple persistent storage systems as distributed ledgered participants. In various embodiments, the distributed ledgered data is maintained using the native capabilities of a persistent storage system. The distributed ledgered data is replicated as persistent data objects in a “ledgered repository of objects” that are replicated at each of the persistent storage systems. Changes at one persistent storage system are recorded within a block in a distributed blockchain that is distributed across each of the other distributed ledgered participants. The other distributed ledgered participants read the changes from the blockchain and apply the changes to the respective replicas at each of the other distributed ledgered participants. Hence, this approach is referred to as blockchain apply. Blockchain apply may be used to replicate the repository objects of various forms of PSSs.

Abstract: A method and apparatus for message cache sizing are provided. A plurality of messages from one or more enqueuers are enqueued in a queue comprising at least one shard. Each shard includes one or more subshards. A message cache configured to store a plurality of cached subshards is maintained in memory. The plurality of messages is dequeued from the message cache by one or more dequeuers. An estimated enqueue rate is determined for each shard. An estimated dequeue rate is determined for each dequeuer-shard pair. Estimated restore overhead data is determined over a range of candidate message cache sizes based on the estimated enqueue rate for each shard and the estimated dequeue rate for each dequeuer-shard pair. A preferred message cache size is determined based on the estimated restore overhead data.

Abstract: A method is provided for space-efficient bookkeeping using bit-level locking. Each row of a bookkeeping table stores binary data comprising a plurality of bits. Each bit of the plurality of bits corresponds to a particular message and a particular consumer of the message, and indicates whether the particular message has been dequeued by the particular consumer. The bookkeeping table is stored on-disk and data corresponding to the bookkeeping table is stored in volatile memory. When a message is dequeued the in-memory bookkeeping data is updated, and the bookkeeping table is updated based on the in-memory bookkeeping data when the dequeue is committed.

Abstract: Techniques are described herein for achieving physically dispersed item-dispersal patterns in queue tables that are used as circular buffers in a FIFO system. The techniques achieve the physically dispersed item-dispersal patterns while retaining FIFO semantics, efficient data ingest, efficient data access, and low latency for both fixed and variable-size ingest data. Because consecutively inserted rows are written to physically dispersed locations on persistent storage, lock contention between concurrent ingest sessions is significantly reduced. Once a physically dispersed item-dispersal pattern is established for a first set of inserted rows, subsequent rows are added by overwriting the existing rows. The physical dispersal of the initial rows may be accomplished by setting values in a slot number column such that updates that are made in a sequence that is based on the slot number column exhibit a desired physical dispersal pattern.

Abstract: A method and apparatus for message cache management for message queues is provided. A plurality of messages from a plurality of enqueuers are enqueued in a queue comprising one or more shards, each shard comprising one or more subshards. A message cache is maintained in memory. Enqueuing a message includes enqueuing the message in a current subshard of a particular shard, which includes storing the message in a cached subshard corresponding to the current subshard of the particular shard. For each dequeuer-shard pair, a dequeue rate is determined. Estimated access time data is generated that includes an earliest estimated access time for each of a plurality of subshards based on the dequeuer-shard pair dequeue rates. A set of subshards is determined for storing as cached subshards in the message cache based on the earliest estimated access times for the plurality of subshards.

Abstract: A method and apparatus for message cache sizing are provided. A plurality of messages from one or more enqueuers are enqueued in a queue comprising at least one shard. Each shard includes one or more subshards. A message cache configured to store a plurality of cached subshards is maintained in memory. The plurality of messages is dequeued from the message cache by one or more dequeuers. An estimated enqueue rate is determined for each shard. An estimated dequeue rate is determined for each dequeuer-shard pair. Estimated restore overhead data is determined over a range of candidate message cache sizes based on the estimated enqueue rate for each shard and the estimated dequeue rate for each dequeuer-shard pair. A preferred message cache size is determined based on the estimated restore overhead data.

Abstract: A method and apparatus for message cache management for message queues is provided. A plurality of messages from a plurality of enqueuers are enqueued in a queue comprising one or more shards, each shard comprising one or more subshards. A message cache is maintained in memory. Enqueuing a message includes enqueuing the message in a current subshard of a particular shard, which includes storing the message in a cached subshard corresponding to the current subshard of the particular shard. For each dequeuer-shard pair, a dequeue rate is determined. Estimated access time data is generated that includes an earliest estimated access time for each of a plurality of subshards based on the dequeuer-shard pair dequeue rates. A set of subshards is determined for storing as cached subshards in the message cache based on the earliest estimated access times for the plurality of subshards.

Abstract: An in-database sharded queue for a shared-disk database is provided. First messages from at least one first enqueuer are stored on a first queue shard. The first enqueuer/s execute in a first instance of a shared-disk database system. The first shard includes a first plurality of subshards. Second messages from at least one second enqueuer are stored on a second queue shard. The second enqueuer/s execute in a second instance of the shared-disk database system. The second shard includes a second plurality of subshards. In volatile memory of the respective instances, a first message cache and a second message cache is maintained for the first shard and the second shard. The respective message caches include at least one buffer associated with the plurality of subshards.

Abstract: Techniques for replicating data in database systems are described. In an example embodiment, a set of changes is received at a destination database, where the set of changes has been applied at a source database and is being replicated from the source database to the destination database. The set of changes is analyzed and it is determined that the set of changes includes two or more of: a subset of row-level changes, a subset of statement-level changes, and a subset of procedure-level changes. A set of dependencies is determined at least between the changes that are included in the subsets of changes. The changes, in the subsets of changes, are assigned to two or more processing elements. The set of changes is applied to the destination database by executing the two or more processing elements in parallel to each other and based on the set of dependencies.

Abstract: Handling memory pressure in an in-database sharded queue is described. Messages from a plurality of enqueuers are stored in a plurality of shards of a sharded queue. Messages from a first enqueuer are stored in a first shard. A queue table corresponding to the sharded queue is maintained. In volatile memory, a plurality of message caches is maintained, each message cache corresponding to a shard of the plurality of shards. Memory pressure is detected based on memory usage of the volatile memory. To store a specific message from the enqueuer, the specific message is stored in rows of the queue table that are assigned to the first shard. When memory pressure is not detected, the specific message is stored in a first message cache corresponding to the first shard. Subscribers of the sharded queue are caused to dequeue messages from the plurality of shards.