Abstract: A method and apparatus for configurable application feature experiments is described. The method may include receiving a set of metrics to be collected after a feature treatment is deployed to configurable applications executed by a plurality of end user systems, a significance threshold for detection of feature treatment impact on one or more metrics within the set of metrics, and a request to perform multiple comparison correction when detecting the feature treatment impact. The method may also include receiving, from the configurable applications, event messages that include metric values associated with the set of metrics. Further, the method may also include performing a statistical analysis of the metric values to determine whether the feature treatment caused a statistically significant change in values associated with one or more of the metrics, where the statistical analysis adjusts one or more parameters used to perform the statistical analysis based on a multiple comparison correction analysis.

Abstract: A method and apparatus for event attribution during software experimentation is described. The method may include receiving, by a server computer system, a plurality of event tracking messages associated with an end user system, each event tracking message including at least a customer identifier, an end user identifier, and a timestamp. The method may also include storing each event tracking messages with the received customer identifier, end user identifier, and timestamp in a customer data store. Furthermore, the method may include applying, by the server computer system, a feature treatment to a configurable application executing on the end user system, the feature treatment specified by a customer system associated with the customer identifier, and the feature treatment configures one or more features of the configurable application associated with the end user identifier.

Abstract: A method and apparatus for configurable application feature experiments is described. The method may include receiving a set of metrics to be collected after a feature treatment is deployed to configurable applications executed by a plurality of end user systems, a significance threshold for detection of feature treatment impact on one or more metrics within the set of metrics, and a request to perform multiple comparison correction when detecting the feature treatment impact. The method may also include receiving, from the configurable applications, event messages that include metric values associated with the set of metrics. Further, the method may also include performing a statistical analysis of the metric values to determine whether the feature treatment caused a statistically significant change in values associated with one or more of the metrics, where the statistical analysis adjusts one or more parameters used to perform the statistical analysis based on a multiple comparison correction analysis.

Abstract: A method and apparatus for event attribution during software experimentation is described. The method may include receiving, by a server computer system, a plurality of event tracking messages associated with an end user system, each event tracking message including at least a customer identifier, an end user identifier, and a timestamp. The method may also include storing each event tracking messages with the received customer identifier, end user identifier, and timestamp in a customer data store. Furthermore, the method may include applying, by the server computer system, a feature treatment to a configurable application executing on the end user system, the feature treatment specified by a customer system associated with the customer identifier, and the feature treatment configures one or more features of the configurable application associated with the end user identifier.

Abstract: A method and apparatus for event attribution during software experimentation is described. The method may include receiving, by a server computer system, a plurality of event tracking messages associated with an end user system, each event tracking message including at least a customer identifier, an end user identifier, and a timestamp. The method may also include storing each event tracking messages with the received customer identifier, end user identifier, and timestamp in a customer data store. Furthermore, the method may include applying, by the server computer system, a feature treatment to a configurable application executing on the end user system, the feature treatment specified by a customer system associated with the customer identifier, and the feature treatment configures one or more features of the configurable application associated with the end user identifier.

Abstract: A method and apparatus for application anomaly detection and remediation is described. The method may include receiving a plurality of event tracking messages generated by configurable applications after a feature treatment is deployed to configurable applications running on a first set of end user systems, and associating the feature treatment with values of a metric from the event messages that are attributable to the feature treatment being executed by the one or more configurable applications. The method may also include determining an impact of the feature treatment on the degradation of the metric when compared to a control value of the metric determined from a second plurality of end user systems that are not exposed to the feature treatment. Then method may further include, in response to detecting the statistically significant degradation of the metric, performing one or more actions to remediate the undesired impact of the feature treatment on the execution of the application.

Abstract: A method and apparatus for event attribution during software experimentation is described. The method may include receiving, by a server computer system, a plurality of event tracking messages associated with an end user system, each event tracking message including at least a customer identifier, an end user identifier, and a timestamp. The method may also include storing each event tracking messages with the received customer identifier, end user identifier, and timestamp in a customer data store. Furthermore, the method may include applying, by the server computer system, a feature treatment to a configurable application executing on the end user system, the feature treatment specified by a customer system associated with the customer identifier, and the feature treatment configures one or more features of the configurable application associated with the end user identifier.