Abstract: The disclosed embodiments provide a system for evaluating a performance of a mobile application. During operation, the system obtains, for a statistical model used in a quasi-experimental design, a first predicted outcome produced from a first set of data that is collected from two substantially identical versions of a mobile application. Next, the system uses the first predicted outcome to assess a bias of the statistical model. The system then improves an accuracy of the statistical model by using the assessed bias to normalize a second predicted outcome of the statistical model.

Abstract: The disclosed embodiments provide a system for managing an A/B test. During operation, the system calculates a first risk associated with ramping up exposure to a first A/B test by a first ramp amount. Next, the system uses a first sequential hypothesis test to compare the first risk with a first risk tolerance for the first A/B test. When the first sequential hypothesis test indicates that the first risk is within the first risk tolerance, the system automatically triggers a ramp-up of exposure to the first A/B test by the first ramp amount.

Abstract: In one embodiment, a method for heuristic event aggregation and correlation includes grouping events from a same device into event groups. Neighboring events in an event group have a timestamp gap of less than a defined timestamp gap threshold. Correlation of events is performed within event groups between devices. One or more alerts are generated based on the correlation. The method can systematically reduce millions of events into alerts which can be understood by human administrators, who can triage and troubleshoot for problems in ways which would not have been possible without the method.

Abstract: A machine may be configured to facilitate following A/B tests generated by an A/B testing system. For example, the machine accesses a subscription record associated with an experiment identifier of an experiment in a database. The experiment may pertain to a service provided on a Social Networking Service (SNS). The subscription record identifies one or more users who requested to follow the experiment. The accessing is based on the experiment identifier. The machine identifies, based on the subscription record, a user identifier of a user of the one or more users who requested to follow the experiment. The machine generates a digital content item that references the experiment followed by the user based on a presentation template and data pertaining to the experiment followed by the user. The machine causes a presentation of the digital content item in a user interface of a client device associated with the user.

Abstract: A machine may be configured to facilitate following metrics generated by an A/B testing system. For example, the machine determines, based on a metric identifier of a metric that measures an aspect of a service provided on a Social Networking Service, one or more experiment identifiers of experiments that impact the metric. The machine accesses, based on the metric identifier, a subscription record in a database. The subscription record identifies one or more users who requested to follow the particular metric. The machine identifies, based on the subscription record, a user identifier of a user who requested to follow the metric. The machine generates a digital content item that references the metric and the experiments that impact the metric. The machine causes a presentation of the digital content item in a user interface of a client device associated with the user.

Abstract: A machine may be configured to generate A/B test reports on demand. For example, the machine causes a display of a user interface for receiving a request of a customized report of result data of an A/B test. The machine receives, via the user interface, an identifier of the A/B test, a specification of a metric associated with the result data, a specification of a dimension of the metric, a specification of a location of the result data, and a request to generate the customized report. The machine generates the metric based on the identifier of the A/B test, the specification of the metric, and the result data. The machine generates the customized report pertaining to the dimension of the metric based on the generated metric and the specification of the dimension of the metric. The machine causes a display of the customized report in the user interface.

Abstract: The disclosed embodiments provide a system for evaluating a performance of a mobile application. During operation, the system obtains a first set of data associated with adopters of a new version of a mobile application in a partial rollout of the new version and a second set of data associated with non-adopters of the new version in the partial rollout. Next, the system applies a statistical model to the first and second sets of data to select a subset of the non-adopters as potential adopters of the new version. The system then reduces a bias in a quasi-experimental design associated with the mobile application by using the first set of data and a third set of data associated with the potential adopters to estimate an average treatment effect (ATE) between the new version and an older version of the mobile application.

Abstract: The disclosed embodiments provide a system for evaluating a performance of a mobile application. During operation, the system obtains, for a statistical model used in a quasi-experimental design, a first predicted outcome produced from a first set of data that is collected from two substantially identical versions of a mobile application. Next, the system uses the first predicted outcome to assess a bias of the statistical model. The system then improves an accuracy of the statistical model by using the assessed bias to normalize a second predicted outcome of the statistical model.

Abstract: A machine may be configured to manage alerts related to ramping A/B experiments. For example, the machine identifies an A/B experiment that targets users of a social networking service (SNS). The machine accesses a first value of a metric associated with operation of the SNS. The first value of the metric is generated as a result of a previous execution of the A/B experiment targeting a first segment of users. The machine generates a predicted second value of the metric based on executing a prediction model associated with the A/B experiment. The executing of the prediction model targets a second segment of users that is greater than the first segment. The machine determines that the predicted second value of the metric indicates an inferred negative impact of the A/B experiment on the metric. The machine causes a display of an alert in a user interface displayed on a client device.

Abstract: The disclosed embodiments provide a system for performing network A/B testing. During operation, the system obtains a graph of a social network and calculates a set of equally sized clusters of users in the social network by iteratively switching memberships of the nodes among the equally sized clusters to increase a number of edges in each of the equally sized clusters. Next, the system randomly selects a subset of the equally sized clusters for exposure to a treatment version of a message. The system then performs an A/B test by presenting the treatment version to the selected clusters and tracking a response of the selected clusters to the treatment version.

Abstract: Techniques for conducting A/B experimentation of online content are described. According to various embodiments, a user specification of a metric associated with operation of an online social networking service is received. A set of one or more A/B experiments of online content is then identified, each A/B experiment being targeted at a segment of members of the online social networking service. Thereafter, each of the A/B experiments is ranked, based on an inferred impact on the value of the metric in response to application of a treatment variant of each A/B experiment to the online social networking service. A list of one or more of the ranked A/B experiments is then displayed, via a user interface displayed on a client device.

Abstract: Techniques for conducting A/B experimentation of online content are described. According to various embodiments, it is determined that a member of an online social networking service is accessing electronic content that is associated with an A/B experiment, the member being associated with a given member ID. The given member ID and a request for a correct variant of the A/B experiment for display for the given member ID is transmitted, from a service associated with the online content, to an A/B experiment server. The correct variant of the A/B experiment for display for the given member ID is accessed from a database, and information regarding the correct variant of the A/B experiment is transmitted to the service associated with the online content. Thereafter, the correct variant of the A/B experiment in conjunction with the online content is displayed, to a client device associated with the member.

Abstract: Techniques for conducting A/B experimentation of online content are described. According to various embodiments, a user specification of a metric being recorded as a result of an online A/B experiment of online content is received, the online A/B experiment being targeted at a segment of members of an online social networking service. Thereafter, a power value for the A/B experiment that is associated with the metric is calculated, the power value indicating an inferred ability to detect changes in a value of the metric during performance of the A/B experiment. The power value for the A/B experiment is then displayed via a user interface displayed on a client device.

Abstract: Techniques for conducting A/B experimentation of online content are described. According to various embodiments, a user specification of targeting criteria defining a targeted segment of members of an online social networking service for an A/B experiment is received. A set of members of the online social networking service satisfying the user-specified targeting criteria is then identified. Thereafter, a user specification of allocation criteria defining one or more variants of an A/B experiment and one or more corresponding allocation percentages is received. A subset of the set of members (corresponding to the user-specific allocation percentage associated with the respective user-specified variant) is then assigned to each of the user-specified variants. Further, at least one data record associated with each of the variants is recorded in a database, each data record indicating the subset of the set of members assigned to the respective variant.

Abstract: The disclosed embodiments provide a method and system for performing network A/B testing. During operation, the system obtains, for a set of users in a social network, a set of treatment assignments of the users in an A/B test, wherein the treatment assignments indicate exposure of the users to a control version or a treatment version of a message. Next, the system obtains, for each of the users, a fraction of neighbors exposed to the treatment version in the A/B test. The system then applies a statistical model to the treatment assignments and the fraction of neighbors exposed to the treatment version to estimate an average treatment effect (ATE) for the set of users. Finally, the system selects, based on the ATE, a fraction of additional users in the social network for subsequent exposure to the treatment version and presents the treatment version to the fraction of additional users.

Abstract: Techniques for conducting A/B experimentation of online content are described. According to various embodiments, a site-wide impact value for an A/B experiment that is associated with a metric is calculated, the site-wide impact value indicating a predicted percentage change in the value of a metric responsive to application of a treatment variant to an entire portion of a targeted segment of members, in comparison to application of a control variant to an entire portion of the targeted segment of members.

Abstract: In one embodiment, the integrity of forwarding paths within a packet switching device is investigated. A packet switching device creates a probe packet. The packet switching device then communicates the probe packet within the packet switching device in a normal forwarding manner, while monitoring at multiple positions along forwarding paths through the packet switching device for the appearance of the probe packet. The traveling within the packet switching device of the probe packet, including as identified by the monitored positions, is analyzed to identify whether or not the probe packet was correctly forwarded at one or more locations within the packet switching device.

Abstract: Techniques for conducting A/B testing are described. According to various embodiments, a user request to configure an A/B test is received, and a targeting user interface is displayed. The targeting user interface includes a group of one or more windows, each of the windows being configured to display one or more user-specified targeting criteria statements that identify members of an online social networking service based on member attributes of the members. Thereafter, a user specification of one or more targeting criteria statements is received in conjunction with one or more of the windows. Targeting criteria information defining a set of members of the online social networking service to be targeted for the A/B test is then generated, the targeting criteria information indicating that all members satisfying all the targeting criteria statements for any window in the group of windows are to be targeted for the A/B test.

Abstract: Enhancements of the user experience are provided when a user returns to a previously viewed page, such as a previously viewed page of search results. When a user returns to a previously viewed page, additional context information from a user's actions since the initial view of a page can be used to modify the previously viewed page and/or obtain a new version of the previously viewed page. In situations where the previously viewed page corresponds to a page of responsive results from a search engine, the modified and/or new version of the search engine results page can include an expanded or reduced group of results, different types of results, different rankings for existing results, or a combination thereof.

Abstract: In an OTN (Optical Transport Network), an OTN interface of a network element may receive a BDI (Backwards Defect Indicator) signal from a far end network element to which OTN frames are being transmitted. A BDI signal indicates the occurrence of a unidirectional failure in the transmission of OTN frames to which the far end network element responds by switching its routing to the network element. The OTN interface generates an AIS (Alarm Indication Signal) for its network element so that the network element switches network communication channels to the far end network element to ensure bidirectional switching upon a unidirectional failure.