Abstract: Database server instances in a database server cluster broadcast, to other instances in the cluster, information concerning certain problem events. Because each server instance is aware of problems that other server instances are experiencing, each server instance is enabled to make more intelligent decisions regarding the actions that it should perform in response to the problems that the server instance is experiencing. Instead of terminating itself, a server instance might opt to wait for a longer amount of time for an operation to complete. The server instance may do so due to the server instance having received information that indicates that other server instances are experiencing similar problems. Whenever the information received from other server instances makes it appear that a problem is unlikely to be solved in the cluster as a whole by terminating a server instance, that server instance may continue to wait instead of terminating itself.

Abstract: A method and apparatus for managing locks in a database system is provided. A master node grants a lock on a first resource and a group of resources that includes the first resource to a first requester node. The requester node receives a mapping corresponding to the group of resources that may indicate that a lock already exists for a second resource in the group. If the requester node desires a lock on a resource located in the group, the requester node grants itself the lock without notifying the master node. A second requester node requests a lock for a particular resource in the group of resources. The first requester node grants the lock on the particular resource and updates the mapping to indicate that a different node holds a lock for the particular resource.

Abstract: A method and apparatus is provided for determining the most probable cause of a problem observed in a complex multi-host system. The approach relies on a probabilistic model to represent causes and effects in a complex computing system. However, complex systems include a multitude of independently operating components that can cause temporary anomalous states. To reduce the resources required to perform root cause analysis on each transient failure, as well as to raise the confidence in the most probable cause of a failure that is identified by the model, inputs to the probabilistic model are aggregated over a sliding window of values from the recent past.

Abstract: A method and apparatus for managing shared resources in a clustered database management system is provided. In an embodiment, multiple master nodes exist in a database management system. A master node receives a lock request from a second node. The lock request is a request for a lock on a shared resource. The master node grants the lock request to the second node. While the second node holds the lock, the second node causes the master node to modify the shared resource.

Abstract: Database server instances in a database server cluster broadcast, to other instances in the cluster, information concerning certain problem events. Because each server instance is aware of problems that other server instances are experiencing, each server instance is enabled to make more intelligent decisions regarding the actions that it should perform in response to the problems that the server instance is experiencing. Instead of terminating itself, a server instance might opt to wait for a longer amount of time for an operation to complete. The server instance may do so due to the server instance having received information that indicates that other server instances are experiencing similar problems. Whenever the information received from other server instances makes it appear that a problem is unlikely to be solved in the cluster as a whole by terminating a server instance, that server instance may continue to wait instead of terminating itself.

Abstract: Described herein are techniques for using super masters to coordinate accesses by nodes in a database server cluster to data stored in a database. In many cases, the number of inter-nodal messages is reduced, thereby saving the overheads attendant to coordinating accesses in such a system.

Abstract: Techniques for adaptive trace logging include, in one embodiment, obtaining input data on trace logging behavior and computing resources used by trace logging. Based on the obtained input data, an adaptive trace logging module automatically takes action at runtime to reduce the amount of computing resources consumed by tracing logging. For example, the action taken may include decreasing a trace logging level of an executing software program to reduce the number of trace logging messages added to a trace log. In another embodiment, the techniques include detecting a condition of an executing software program that warrants a change to a trace logging level of the executing program. The adaptive trace logging module automatically changes the trace logging level of the executing program as-needed for the detected condition. For example, the adaptive trace logging module may increase the trace logging level of an executing program upon detecting a deadlock or other abnormal condition of the executing program.

Abstract: Systems, methods, and other embodiments associated with selective tag-based file backup and recovery are described. One example method includes selectively tagging a file for inclusion in a snapshot-based backup image by associating a tag with the file. The associating may include encoding file metadata with a tag. The method may include selectively adding a file to the backup image upon determining that the file has experienced a write event and that the file is associated with a tag. The method may also include receiving a request to provide a recovery file from the backup image and selectively providing the recovery file upon determining that the recovery file is associated with a recovery tag specified in the request.

Abstract: A system, method, and computer program product are described for distinguishing between a computing system that is hung in a hang state and systems that are in an idle or otherwise non-hang state which do not need intervention before regaining the ability to adequately process work. According to some approaches, heuristics are employed to perform hang and idle system detection and validation. Data representative of systems resources are analyzed and transformed in order to identify systems that are in a hang state.

Abstract: Techniques are provided for responding to the termination of a node by selecting another node, and assigning to the selected node the affinity relationships that existed between the terminated node and one or more objects. The resources that belong to the objects involved in the affinity relationships are remastered to the selected node. The selected node then performs recovery of the resources that had been opened by the terminated node and/or serves as a failover node to execute the transactions that had been executing on the terminated node.

Abstract: Described herein are techniques for dynamically monitoring process activeness of processes running on a computing node. Problems affecting processes to serve their designated functions on the computing node can be relatively quickly detected and dealt with, thereby making restoring process activeness on the computing node much more quickly than otherwise.

Abstract: A method and apparatus for selecting one or more instances to perform database recovery is provided. According to one aspect, one or more performance metrics are determined for each instance on each node in a cluster. Based on these performance metrics, which may be weighted to give one or more metrics more significance relative to the other metrics, an overall score is determined for each instance on each surviving node when a node in the cluster fails. One or more instances with the highest of these scores are selected. The selected instances are assigned the task of performing the database recovery process. Because instances best able to handle to database recovery process are selected, the database recovery process consumes less time, thereby making data accessible in less time after a node fails.

Abstract: Described herein are techniques for dynamically monitoring and managing resource usages of processes running on a node in a multi-node database system. High resource usages of processes can be proactively detected and alleviated, thereby making such a node to perform significantly better than otherwise.

Abstract: Described herein are techniques for adaptively managing timers that are used in various layers of a node. In many cases, the number of timers that occur in the system is reduced by proactively and reactively adjusting values of the timers based on conditions affecting the system, thereby making such a system to perform significantly better and more resiliently than otherwise.

Abstract: Described herein are techniques for dynamically monitoring and rebalancing priority levels of processes running on a computing node. Runaway processes and starved processes can be proactively detected and prevented, thereby making such a node to perform significantly better and more responsively than otherwise.

Abstract: Described herein are techniques for using super masters to coordinate accesses by nodes in a database server cluster to data stored in a database. In many cases, the number of inter-nodal messages is reduced, thereby saving the overheads attendant to coordinating accesses in such a system.

Abstract: Described herein are techniques for time limited lock ownership. In one embodiment, in response to receiving a request for a lock on a shared resource, the lock is granted and a lock lease period associated with the lock is established. Then, in response to determining that the lock lease period has expired, one or more lock lease expiration procedures are performed. In many cases, the time limited lock ownership may prevent system hanging, timely detect system deadlocks, and/or improve overall performance of the database.

Abstract: Techniques are described for storing portions of files on different storage devices based on the devices' respective storage class. For example, different extents from a large file within a system, application, or database may be stored on different storage devices. The storage devices may be part of a single storage system that includes some storage devices that have different performance capabilities. The decision of where, within the storage classes, to store data may be based on the access patterns of the data. Access patterns regarding file extents are tracked using respective access bits, which are set when a memory block associated with an extent is accessed via an I/O operation. In one embodiment, an access bitmap is modified to account for cached blocks of file extents.

Abstract: Techniques are described herein for synchronizing cluster time. According to one technique, a master node is appointed in a cluster. Other “slave” nodes periodically synchronize their clocks with the master node. To synchronize its clock with the master node, a slave node sends a timestamped message to the master node, which also timestamps the message and sends the message back to the slave node, which then timestamps the message again. Based on the timestamps, the slave node is able to determine the difference between the master node's clock's time and slave node's clock's time, compensating for the message travel time between master node and slave node. Depending on various circumstances, and based on the determined difference, the slave node adjusts its clock so that the time indicated by the slave node's clock at least begins to approach more closely the time indicated by the master node's clock.

Abstract: In a multi-node database system, a list of data blocks to perform recovery operations on in the event of failure of a given node is dynamically maintained on another node. List is available when the given node fails, and allows the database to be made available more quickly.