Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for evaluating statistical significance using placebo actions. One method includes receiving exposure data and control data describing exposed and control subjects and determining a test statistic for each of a plurality of subject actions including a desired action and a plurality of placebo actions. The test statistic for an action corresponds to an estimated effect that exposure to the treatment has on a likelihood that a subject will take the action. The method further includes comparing the test statistic for the desired action to the test statistics for the plurality of placebo actions, and determining whether the test statistic for the desired action is statistically significant according to the comparison. Another method includes automatically identifying placebo actions using a taxonomy that classifies subject actions or treatments, and determining test statistics for the placebo actions.

Abstract: Systems, methods and computer program products for determining lost opportunities resulting from changes to advertising spending are described. To assist advertisers in evaluating and allocating a proper budget to advertising, an analyzer can be used to develop an analytical model that gathers data pertaining to the incremental value of search advertising. (e.g., the true cost of the additional click lost or gained), which can be presented to the advertisers when changes have been made/proposed to the advertiser's advertising spending. The analyzer can detect large changes in advertising spending, and indicate (e.g., by prediction) how many total clicks were lost or gained as a result of the change in advertising spending to allow the advertisers to visualize the impact to changes in advertising spending, and determine when to decrease advertising budget on ads that yield low return-on-investment or to increase advertising budget to maximize the effectiveness of an active ad campaign.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for evaluating statistical significance using placebo actions. One method includes receiving exposure data and control data describing exposed and control subjects and determining a test statistic for each of a plurality of subject actions including a desired action and a plurality of placebo actions. The test statistic for an action corresponds to an estimated effect that exposure to the treatment has on a likelihood that a subject will take the action. The method further includes comparing the test statistic for the desired action to the test statistics for the plurality of placebo actions, and determining whether the test statistic for the desired action is statistically significant according to the comparison. Another method includes automatically identifying placebo actions using a taxonomy that classifies subject actions or treatments, and determining test statistics for the placebo actions.