Abstract: A funnel locking technique improves scalability of callback handling while mediating grace-period-start requests in a sleepable read-copy update (SRCU) environment. Per-CPU data structures maintain lists of SRCU callbacks. A management data structure maintains a current-grace-period record tracking a current SRCU grace period and a future-grace-period record tracking a future SRCU grace period needed by the callbacks. A combining tree mediates grace-period-start requests concurrently vying for an opportunity to update the future-grace-period record. The grace-period-start requests are issued on behalf of the callbacks to a combining tree leaf level and propagate toward a combining tree root node. The future-grace-period record is updated in response to grace-period-start requests reaching the root node.

Abstract: An updater needing an expedited RCU grace period may initiate a leaf-to-root traversal of a funnel lock embodied as a hierarchical tree of nodes. For each accessed node, the updater may check an indicator to determine if another updater needing the same expedited grace period has visited the node. If true, the updater may add itself to a waitqueue of updaters waiting for the expedited RCU grace period. If false, the updater may set the indicator to indicate it has visited the node, and then continue to a next node. If the updater reaches the root node with no indication that any other updater needing the expedited RCU grace period has visited the nodes accessed by the updater, the updater may, while holding a mutex lock, start a new expedited RCU grace period and at the end thereof wake up other updaters waiting on the expedited RCU grace period.

Abstract: A technique for determining if a CPU stalling an RCU grace period has interrupts enabled. Per-CPU state information is maintained for the CPU, including an RCU grace-period number snapshot and an interrupt work-request indicator. If a current RCU grace period has endured for a predetermined time period, it is determined if there is a pending interrupt work request for the CPU. If not, an interrupt work request is generated and the interrupt work-request indicator is updated accordingly. In response to an RCU CPU stall-warning condition, it is determined if the interrupt work request was handled. An RCU CPU stall-warning message reports an interrupt-handling status of the CPU according to the interrupt work request handling determination. If the interrupt work request was not handled, the RCU CPU stall-warning message reports how many RCU grace periods elapsed while the interrupt work request was pending.

Abstract: A technique for enabling real-time CPU-bound in-kernel workloads to run infinite loops while keeping read-copy update (RCU) grace periods finite. In an embodiment, a per-CPU indicator may be set to indicate that a CPU running the CPU-bound in-kernel workload has not reported an RCU quiescent state within a defined time. A function may be invoked from within the workload that causes an RCU quiescent state to be reported on behalf of the CPU if the per-CPU indicator is set. If the RCU quiescent state is not reported within a longer defined time, the CPU may be rescheduled.

Abstract: A technique for enabling real-time CPU-bound in-kernel workloads to run infinite loops while keeping read-copy update (RCU) grace periods finite. In an embodiment, a per-CPU indicator may be set to indicate that a CPU running the CPU-bound in-kernel workload has not reported an RCU quiescent state within a defined time. A function may be invoked from within the workload that causes an RCU quiescent state to be reported on behalf of the CPU if the per-CPU indicator is set. If the RCU quiescent state is not reported within a longer defined time, the CPU may be rescheduled.

Abstract: A technique for shielding real-time workloads from operating system (OS) jitter due to expedited read-copy update (RCU) grace periods. In accordance with the disclosed technique, a kernel parameter is set to indicate that expedited RCU grace periods are to be suppressed. The kernel parameter is checked to see if it is set. A normal non-expedited RCU grace period is invoked in response to attempted invocation of an expedited RCU grace period if the check reveals that the kernel parameter is set.

Abstract: A technique for enabling real-time CPU-bound in-kernel workloads to run infinite loops while keeping read-copy update (RCU) grace periods finite. In an embodiment, a per-CPU indicator may be set to indicate that a CPU running the CPU-bound in-kernel workload has not reported an RCU quiescent state within a defined time. A function may be invoked from within the workload that causes an RCU quiescent state to be reported on behalf of the CPU if the per-CPU indicator is set. If the RCU quiescent state is not reported within a longer defined time, the CPU may be rescheduled.

Abstract: A technique for enabling real-time CPU-bound in-kernel workloads to run infinite loops while keeping read-copy update (RCU) grace periods finite. In an embodiment, a per-CPU indicator may be set to indicate that a CPU running the CPU-bound in-kernel workload has not reported an RCU quiescent state within a defined time. A function may be invoked from within the workload that causes an RCU quiescent state to be reported on behalf of the CPU if the per-CPU indicator is set. If the RCU quiescent state is not reported within a longer defined time, the CPU may be rescheduled.

Abstract: A technique for shielding real-time workloads from operating system (OS) jitter due to expedited read-copy update (RCU) grace periods. In accordance with the disclosed technique, a kernel parameter is set to indicate that expedited RCU grace periods are to be suppressed. The kernel parameter is checked to see if it is set. A normal non-expedited RCU grace period is invoked in response to attempted invocation of an expedited RCU grace period if the check reveals that the kernel parameter is set.

Abstract: A technique for short-circuiting normal read-copy update (RCU) grace period computations in the presence of expedited RCU grace periods. Both normal and expedited RCU grace period processing may be periodically performing to respectively report normal and expedited quiescent states on behalf of CPUs in a set of CPUs until all of the CPUs have respectively reported normal or expedited quiescent states so that the normal and expedited grace periods may be respectively ended. The expedited grace periods are of shorter duration than the normal grace periods. Responsive to a condition indicating that the normal RCU grace period processing can be short-circuited by the expedited RCU grace period processing, the expedited RCU grace period processing may report both expedited quiescent states and normal quiescent states on behalf of the same CPUs in the set of CPUs.

Abstract: A technique for enabling real-time CPU-bound in-kernel workloads to run infinite loops while keeping read-copy update (RCU) grace periods finite. In an embodiment, a per-CPU indicator may be set to indicate that a CPU running the CPU-bound in-kernel workload has not reported an RCU quiescent state within a defined time. A function may be invoked from within the workload that causes an RCU quiescent state to be reported on behalf of the CPU if the per-CPU indicator is set. If the RCU quiescent state is not reported within a longer defined time, the CPU may be rescheduled.

Abstract: A technique for enabling real-time CPU-bound in-kernel workloads to run infinite loops while keeping read-copy update (RCU) grace periods finite. In an embodiment, a per-CPU indicator may be set to indicate that a CPU running the CPU-bound in-kernel workload has not reported an RCU quiescent state within a defined time. A function may be invoked from within the workload that causes an RCU quiescent state to be reported on behalf of the CPU if the per-CPU indicator is set. If the RCU quiescent state is not reported within a longer defined time, the CPU may be rescheduled.

Abstract: A technique for atomically moving a linked data element may include providing an atomic-move wrapper around the data element, along with an existence header whose status may be permanent, outgoing or incoming to indicate whether the data element is not in transition, or if in transition is either outgoing or incoming. The existence header may reference an existence group having a state field that changes state using a single store operation. A first state may indicate that the data element exists if its existence header is outgoing, and does not exist if its existence header is incoming. A second state may indicate that the data element exists if its existence header is incoming, and does not exist if its existence header is outgoing. Following the state change, the existence group and any atomic-move wrapper containing an outgoing existence header and data element may be freed following an RCU grace period.

Abstract: An optimized skiplist iterator technique for RCU-protected skiplists. A skiplist iterator operation may attempt to validate a cached pointer hint that references a skiplist element of an RCU-protected skiplist, the skiplist element having an associated first key. If the pointer hint is validated, the pointer hint may be dereferenced to access the skiplist element. A pointer in the skiplist element may be dereferenced to advance to a next skiplist element of the RCU-protected skiplist, the next skiplist element having an associated second key that is larger than the first key. If the pointer hint is not validated, a value-based skiplist iterator operation may be performed that includes traversing the RCU-protected skiplist using the first key to find the next skiplist element having the second key. A new pointer hint that references the next skiplist element may be cached for use in a next invocation of the skiplist iterator operation.

Abstract: Read-copy update (RCU) grace period initialization and CPU hotplugging are synchronized without a sleeplock. Periodic RCU grace period initialization includes, for each node of a combining tree, copying a first bitmask indicating online/offline status of a set of CPUs to a second bitmask indicating RCU quiescent state status of the CPUs. Periodic CPU hotplug operations include indicating CPU online/offline status to leaf nodes associated with the CPUs. This status is indicated without manipulating any bits in the first bitmask of the leaf nodes. Prior to each RCU grace period initialization, RCU grace period pre-initialization is performed at each leaf node. The RCU grace period pre-initialization includes (1) updating the first bitmask to account for the CPU hotplug operations, and (2) if this results in the first bitmask transitioning between fully cleared and not-fully cleared states, the state change is conditionally propagated to a higher level node.

Abstract: Read-copy update (RCU) grace period initialization and CPU hotplugging are synchronized without a sleeplock. Periodic RCU grace period initialization includes, for each node of a combining tree, copying a first bitmask indicating online/offline status of a set of CPUs to a second bitmask indicating RCU quiescent state status of the CPUs. Periodic CPU hotplug operations include indicating CPU online/offline status to leaf nodes associated with the CPUs. This status is indicated without manipulating any bits in the first bitmask of the leaf nodes. Prior to each RCU grace period initialization, RCU grace period pre-initialization is performed at each leaf node. The RCU grace period pre-initialization includes (1) updating the first bitmask to account for the CPU hotplug operations, and (2) if this results in the first bitmask transitioning between fully cleared and not-fully cleared states, the state change is conditionally propagated to a higher level node.

Abstract: An updater needing an expedited RCU grace period may initiate a leaf-to-root traversal of a funnel lock embodied as a hierarchical tree of nodes. For each accessed node, the updater may check an indicator to determine if another updater needing the same expedited grace period has visited the node. If true, the updater may add itself to a waitqueue of updaters waiting for the expedited RCU grace period. If false, the updater may set the indicator to indicate it has visited the node, and then continue to a next node. If the updater reaches the root node with no indication that any other updater needing the expedited RCU grace period has visited the nodes accessed by the updater, the updater may, while holding a mutex lock, start a new expedited RCU grace period and at the end thereof wake up other updaters waiting on the expedited RCU grace period.

Abstract: A data element of a linked data structure is atomically moved without delaying lockless readers. A status-indicating entity is allocated, associated with the data element, and indicates validity of the data element with respect to the first linked data structure. A copy element, or a pointer thereto, is created from the data element. The status-indicating entity is associated with the copy element and indicates no validity of the copy element with respect to a second linked data structure. The copy element is linked to the second linked data structure. The status-indicating entity is atomically updated to indicate no validity of the data element with respect to the first linked data structure and validity of the copy element with respect to the second linked data structure. The data element is deleted and the status-indicating entity is disassociated from the copy element. Both structures may be deallocated in a deferred reader-friendly manner.

Abstract: A data element of a linked data structure is atomically moved without delaying lockless readers. A status-indicating entity is allocated, associated with the data element, and indicates validity of the data element with respect to the first linked data structure. A copy element, or a pointer thereto, is created from the data element. The status-indicating entity is associated with the copy element and indicates no validity of the copy element with respect to a second linked data structure. The copy element is linked to the second linked data structure. The status-indicating entity is atomically updated to indicate no validity of the data element with respect to the first linked data structure and validity of the copy element with respect to the second linked data structure. The data element is deleted and the status-indicating entity is disassociated from the copy element. Both structures may be deallocated in a deferred reader-friendly manner.

Abstract: A technique for short-circuiting normal read-copy update (RCU) grace period computations in the presence of expedited RCU grace periods. The technique may include determining during normal RCU grace period processing whether at least one expedited RCU grace period elapsed during a normal RCU grace period. If so, the normal RCU grace period is ended. If not, the normal RCU grace period processing is continued. Expedited RCU grace periods may be implemented by expedited RCU grace period processing that periodically awakens a kernel thread that implements the normal RCU grace period processing. The expedited RCU grace period processing may conditionally throttle wakeups to the kernel thread based on CPU utilization.