Abstract: A machine may be configured to perform A/B testing on customized experiment units. For example, the machine receives a specification of an experiment unit that identifies a type of subject of an experiment for execution on a social networking service (SNS), and a value of the experiment unit. The machine generates, for the value of the experiment unit, tracking data that tracks user interactions by one or more users of the SNS, via one or more browsers, with content provided during an execution of the experiment. The machine generates, for the value of the experiment unit, metric data that measures an attribute associated with the experimental unit. The machine generates an experiment report based on the tracking data and the metric data. The machine causes a presentation of the experiment report in a user interface of a client device.

Abstract: The disclosed embodiments provide a system for evaluating a performance of a mobile application. During operation, the system obtains a first set of data associated with adopters of a new version of a mobile application in a partial rollout of the new version and a second set of data associated with non-adopters of the new version in the partial rollout. Next, the system applies a statistical model to the first and second sets of data to select a subset of the non-adopters as potential adopters of the new version. The system then reduces a bias in a quasi-experimental design associated with the mobile application by using the first set of data and a third set of data associated with the potential adopters to estimate an average treatment effect (ATE) between the new version and an older version of the mobile application.

Abstract: The disclosed embodiments provide a system for evaluating a performance of a mobile application. During operation, the system obtains, for a statistical model used in a quasi-experimental design, a first predicted outcome produced from a first set of data that is collected from two substantially identical versions of a mobile application. Next, the system uses the first predicted outcome to assess a bias of the statistical model. The system then improves an accuracy of the statistical model by using the assessed bias to normalize a second predicted outcome of the statistical model.

Abstract: The disclosed embodiments provide a system for managing an A/B test. During operation, the system calculates a first risk associated with ramping up exposure to a first A/B test by a first ramp amount. Next, the system uses a first sequential hypothesis test to compare the first risk with a first risk tolerance for the first A/B test. When the first sequential hypothesis test indicates that the first risk is within the first risk tolerance, the system automatically triggers a ramp-up of exposure to the first A/B test by the first ramp amount.

Abstract: A machine may be configured to facilitate following metrics generated by an A/B testing system. For example, the machine determines, based on a metric identifier of a metric that measures an aspect of a service provided on a Social Networking Service, one or more experiment identifiers of experiments that impact the metric. The machine accesses, based on the metric identifier, a subscription record in a database. The subscription record identifies one or more users who requested to follow the particular metric. The machine identifies, based on the subscription record, a user identifier of a user who requested to follow the metric. The machine generates a digital content item that references the metric and the experiments that impact the metric. The machine causes a presentation of the digital content item in a user interface of a client device associated with the user.

Abstract: The disclosed embodiments provide a system for evaluating a performance of a mobile application. During operation, the system obtains, for a statistical model used in a quasi-experimental design, a first predicted outcome produced from a first set of data that is collected from two substantially identical versions of a mobile application. Next, the system uses the first predicted outcome to assess a bias of the statistical model. The system then improves an accuracy of the statistical model by using the assessed bias to normalize a second predicted outcome of the statistical model.

Abstract: The disclosed embodiments provide a system for evaluating a performance of a mobile application. During operation, the system obtains a first set of data associated with adopters of a new version of a mobile application in a partial rollout of the new version and a second set of data associated with non-adopters of the new version in the partial rollout. Next, the system applies a statistical model to the first and second sets of data to select a subset of the non-adopters as potential adopters of the new version. The system then reduces a bias in a quasi-experimental design associated with the mobile application by using the first set of data and a third set of data associated with the potential adopters to estimate an average treatment effect (ATE) between the new version and an older version of the mobile application.

Abstract: A machine may be configured to manage alerts related to ramping A/B experiments. For example, the machine identifies an A/B experiment that targets users of a social networking service (SNS). The machine accesses a first value of a metric associated with operation of the SNS. The first value of the metric is generated as a result of a previous execution of the A/B experiment targeting a first segment of users. The machine generates a predicted second value of the metric based on executing a prediction model associated with the A/B experiment. The executing of the prediction model targets a second segment of users that is greater than the first segment. The machine determines that the predicted second value of the metric indicates an inferred negative impact of the A/B experiment on the metric. The machine causes a display of an alert in a user interface displayed on a client device.

Abstract: Techniques for conducting A/B experimentation of online content are described. According to various embodiments, a user specification of a metric being recorded as a result of an online A/B experiment of online content is received, the online A/B experiment being targeted at a segment of members of an online social networking service. Thereafter, a power value for the A/B experiment that is associated with the metric is calculated, the power value indicating an inferred ability to detect changes in a value of the metric during performance of the A/B experiment. The power value for the A/B experiment is then displayed via a user interface displayed on a client device.