Abstract: Computational methods and systems that troubleshoot and localize failures in data storage of a multitenant application are executed in a multitenant database management system that comprises a control plane of control plane nodes and a data plane of independent and isolated data plane nodes. Each control plane node receives data requests and data queries. Each data plane node maintains data storage for tenants of the multitenant application and executes data storage and data query operations requested by the control plane nodes. Methods and system determine whether data plane nodes are network connected and whether the data cores of network connected data plane nodes are available to ingest data and respond to data queries. Data cores of disconnected data plane nodes and unavailable data cores are isolated from data ingestion and data queries. Remedial measures are executed to restore disconnected data plane nodes and unavailable data cores.

Abstract: Computational methods and systems that troubleshoot and localize failures in data storage of a multitenant application are executed in a multitenant database management system that comprises a control plane of control plane nodes and a data plane of independent and isolated data plane nodes. Each control plane node receives data requests and data queries. Each data plane node maintains data storage for tenants of the multitenant application and executes data storage and data query operations requested by the control plane nodes. Methods and system determine whether data plane nodes are network connected and whether the data cores of network connected data plane nodes are available to ingest data and respond to data queries. Data cores of disconnected data plane nodes and unavailable data cores are isolated from data ingestion and data queries. Remedial measures are executed to restore disconnected data plane nodes and unavailable data cores.

Abstract: A method and apparatus are provided for performing cross-host root cause diagnosis within a complex multi-host environment. In a multi-host environment, sometimes system failures on one host may cause problems at another host within the same environment. A probabilistic model is used to represent failures that can occur within each host in the environment. The cause and effect relationships among these failures together with measurement values are used to generate a probability that each potential failure occurred in each host. When a problem is observed on one host without detecting a corresponding root cause within the same host, a cross-host failure diagnosis is performed. The probabilistic models for other hosts in the environment are used to determine the most likely cause of the failure.

Abstract: A method and apparatus are provided for determining the probability that one or more problems have occurred within a complex multi-host system. A probabilistic model representing the cause/effect relationships among potential system problems identifies the probability that a problem occurred in the system based at least on system measure states that are input into the probabilistic model. System measure states may be determined based on an aggregation of system measurement values taken periodically. Aggregating system measurement values may be performed over system measurement values that were taken during a recent time interval. A rolling count aggregation function may be used for this purpose. A rolling count function counts the number of system measurement values taken within the recent time interval that lie within a particular range of values. A system measure state may be determined based on whether the rolling count exceeds a threshold associated with the system measure.

Abstract: A method and apparatus are provided for determining that problems have occurred within a complex multi-host system and for identifying for each problem, sequences of causes and effects called a fault cause path, starting with a root cause. A probabilistic model representing the cause/effect relationships among potential system problems identifies the probability that a problem occurred in the system. Such failure probabilities may be determined based on aggregating, over a recent time interval, probability of failure values determined by the probabilistic model. Each fault cause path may have an associated probability of accuracy value reflecting the expected accuracy of the fault cause path relative to other fault cause paths. When more than one fault cause path is identified, the number and order of the fault cause paths may be ranked and displayed based on their probability of accuracy value.

Abstract: A method and apparatus are provided for performing cross-host root cause diagnosis within a complex multi-host environment. In a multi-host environment, sometimes system failures on one host may cause problems at another host within the same environment. A probabilistic model is used to represent failures that can occur within each host in the environment. The cause and effect relationships among these failures together with measurement values are used to generate a probability that each potential failure occurred in each host. When a problem is observed on one host without detecting a corresponding root cause within the same host, a cross-host failure diagnosis is performed. The probabilistic models for other hosts in the environment are used to determine the most likely cause of the failure.

Abstract: A method and apparatus are provided for determining that problems have occurred within a complex multi-host system and for identifying for each problem, sequences of causes and effects called a fault cause path, starting with a root cause. A probabilistic model representing the cause/effect relationships among potential system problems identifies the probability that a problem occurred in the system. Such failure probabilities may be determined based on aggregating, over a recent time interval, probability of failure values determined by the probabilistic model. Each fault cause path may have an associated probability of accuracy value reflecting the expected accuracy of the fault cause path relative to other fault cause paths. When more than one fault cause path is identified, the number and order of the fault cause paths may be ranked and displayed based on their probability of accuracy value.

Abstract: A method and apparatus are provided for determining the probability that one or more problems have occurred within a complex multi-host system. A probabilistic model representing the cause/effect relationships among potential system problems identifies the probability that a problem occurred in the system based at least on system measure states that are input into the probabilistic model. System measure states may be determined based on an aggregation of system measurement values taken periodically. Aggregating system measurement values may be performed over system measurement values that were taken during a recent time interval. A rolling count aggregation function may be used for this purpose. A rolling count function counts the number of system measurement values taken within the recent time interval that lie within a particular range of values. A system measure state may be determined based on whether the rolling count exceeds a threshold associated with the system measure.

Abstract: A method and apparatus is provided for determining the most probable cause of a problem observed in a complex multi-host system. The approach relies on a probabilistic model to represent causes and effects in a complex computing system. However, complex systems include a multitude of independently operating components that can cause temporary anomalous states. To reduce the resources required to perform root cause analysis on each transient failure, as well as to raise the confidence in the most probable cause of a failure that is identified by the model, inputs to the probabilistic model are aggregated over a sliding window of values from the recent past.