Abstract: A method and apparatus for configurable application feature experiments is described. The method may include receiving a set of metrics to be collected after a feature treatment is deployed to configurable applications executed by a plurality of end user systems, a significance threshold for detection of feature treatment impact on one or more metrics within the set of metrics, and a request to perform multiple comparison correction when detecting the feature treatment impact. The method may also include receiving, from the configurable applications, event messages that include metric values associated with the set of metrics. Further, the method may also include performing a statistical analysis of the metric values to determine whether the feature treatment caused a statistically significant change in values associated with one or more of the metrics, where the statistical analysis adjusts one or more parameters used to perform the statistical analysis based on a multiple comparison correction analysis.

Abstract: A method and apparatus for configurable application feature experiments is described. The method may include receiving a set of metrics to be collected after a feature treatment is deployed to configurable applications executed by a plurality of end user systems, a significance threshold for detection of feature treatment impact on one or more metrics within the set of metrics, and a request to perform multiple comparison correction when detecting the feature treatment impact. The method may also include receiving, from the configurable applications, event messages that include metric values associated with the set of metrics. Further, the method may also include performing a statistical analysis of the metric values to determine whether the feature treatment caused a statistically significant change in values associated with one or more of the metrics, where the statistical analysis adjusts one or more parameters used to perform the statistical analysis based on a multiple comparison correction analysis.

Abstract: A method and apparatus for application anomaly detection and remediation is described. The method may include receiving a plurality of event tracking messages generated by configurable applications after a feature treatment is deployed to configurable applications running on a first set of end user systems, and associating the feature treatment with values of a metric from the event messages that are attributable to the feature treatment being executed by the one or more configurable applications. The method may also include determining an impact of the feature treatment on the degradation of the metric when compared to a control value of the metric determined from a second plurality of end user systems that are not exposed to the feature treatment. Then method may further include, in response to detecting the statistically significant degradation of the metric, performing one or more actions to remediate the undesired impact of the feature treatment on the execution of the application.