Abstract: An online system updates the labels on negative examples to account for the possibility that the example is a false negative. The system generates a set of initial training examples that each include a query input by the user and item data for an item presented as a result to the user's query. Each training example also includes an initial label, which represents whether the user interacted with the item presented as a search result. The online system updates the initial label for a negative training example by identifying a set of bridge queries and computing a similarity score between the query for the training example and the bridge queries. The online system computes an updated label for the negative example based on the similarity scores and updates the training example with the updated label.

Abstract: An online concierge system may conduct experiments in presentation of prioritized items for content campaigns with offline simulations. The offline simulation may use a joint budget for the content campaign used by several experimental variations that affect prioritized content presentation. To correct for distortions that may occur from differing rates of budget use in the variations when the budget is reached before a total period for the experiment, the budget use of each variation is compared to a “fair value” to determine an adjustment to the metrics determined in the experiment. Variants that exceed the fair value may have their metrics capping to the portion allocable to a budget use that does not exceed the fair value, while variants that use less than the fair value may have the metrics extrapolated to account for the additional budget that would be available with a fair value budget.

Abstract: A sample ratio mismatch (SRM) analyzer receives data from an online controlled experiment (OCE) and provides information to help determine a root cause of an SRM. The SRM analyzer may identify one or more segments in the data that include an SRM and may determine whether a triggered scorecard of the OCE includes an SRM. The data may include one or more scorecards. The SRM analyzer may determine whether each scorecard has an SRM. The SRM analyzer may test a difference in proportion of users assigned to treatment between a last scorecard without an SRM and a first scorecard with an SRM. If the difference in proportion is statistically meaningful, the SRM analyzer may determine that the SRM arose after the last scorecard. If the difference in proportions is not statistically meaningful, the SRM analyzer may determine that the SRM existed from a beginning of the OCE.

Abstract: A sample ratio mismatch (SRM) analyzer receives data from an online controlled experiment (OCE) and provides information to help determine a root cause of an SRM. The SRM analyzer may identify one or more segments in the data that include an SRM and may determine whether a triggered scorecard of the OCE includes an SRM. The data may include one or more scorecards. The SRM analyzer may determine whether each scorecard has an SRM. The SRM analyzer may test a difference in proportion of users assigned to treatment between a last scorecard without an SRM and a first scorecard with an SRM. If the difference in proportion is statistically meaningful, the SRM analyzer may determine that the SRM arose after the last scorecard. If the difference in proportions is not statistically meaningful, the SRM analyzer may determine that the SRM existed from a beginning of the OCE.