Abstract: Test periods for an A/B test to be run in one or more geographic regions are set. Each test period in each geographic region is assignable to a treatment or control group. For each of plural test periods other than a first test period and for each geographic region, a biased probability indicating a probability of the test period being assigned to the treatment group of the A/B test is set. The biased probability is set based on a log of previous assignments for the geographic region indicating respective assignments for each previous test period including the first test period. The test period of the geographic region is assigned to one of the treatment and control groups of the A/B test based on the set biased probability. The A/B test is run in the geographic region and during the test period based on the assignment.

Abstract: Methods, systems, and apparatus, for refining log file join data. In one aspect, a method includes receiving first join data defining first joins of records in a first log file to records in a second log file. Each first join of a record in the first log file to a record in the second log file is based on an association of the first identifier of the record in the first log file to the second identifier of the record in the second log file. Associations of the first identifiers to the second identifiers in the first join data that meet a confidence threshold are stored in a mapping of first identifiers to second identifiers as a mapped association. For each mapped association, records that include the first identifier from the first log file are associated with records that include the second identifier from the second log file.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for evaluating statistical significance using placebo actions. One method includes receiving exposure data and control data describing exposed and control subjects and determining a test statistic for each of a plurality of subject actions including a desired action and a plurality of placebo actions. The test statistic for an action corresponds to an estimated effect that exposure to the treatment has on a likelihood that a subject will take the action. The method further includes comparing the test statistic for the desired action to the test statistics for the plurality of placebo actions, and determining whether the test statistic for the desired action is statistically significant according to the comparison. Another method includes automatically identifying placebo actions using a taxonomy that classifies subject actions or treatments, and determining test statistics for the placebo actions.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for evaluating statistical significance using placebo actions. One method includes receiving exposure data and control data describing exposed and control subjects and determining a test statistic for each of a plurality of subject actions including a desired action and a plurality of placebo actions. The test statistic for an action corresponds to an estimated effect that exposure to the treatment has on a likelihood that a subject will take the action. The method further includes comparing the test statistic for the desired action to the test statistics for the plurality of placebo actions, and determining whether the test statistic for the desired action is statistically significant according to the comparison. Another method includes automatically identifying placebo actions using a taxonomy that classifies subject actions or treatments, and determining test statistics for the placebo actions.