Abstract: Systems and methods for model validation includes generating a first and a second time series of segmentation states for a data set representative of a simulated population, e.g., a collection of membership counts corresponding to respective segments of the simulated population. The first and second time series of segmentation states are generated by respectively processing the data set through a first and a second simulation each comprising iterative application of a plurality of event functions. The first and the second simulation differ in at least one capacity, e.g., one including a first event function configured with a first parameter, and the second not. Analysis of differences between the first and second time series may be compared to analysis of one of the time series using a subject model. The comparison is then used to validate the model or demonstrate accuracies, inaccuracies, and/or model bias with respect to a performance metric.

Abstract: Systems, methods, and computer-readable storage media for attribution modeling using withheld or near impression data are provided. One method involves determining, for a first content item impression, withheld or near impressions for a competing content item within a content auction. The method further involves identifying a first set of paths including a sequence of events that includes an interaction with the first content item impression. The method further involves identifying a second set of paths, each including the sequence of events with the competing content item impression replacing the first content item impression. The method compares conversion metrics for the first and second paths to determine attribution credit for the first content item impression.

Abstract: Systems and methods for model validation includes generating a first and a second time series of segmentation states for a data set representative of a simulated population, e.g., a collection of membership counts corresponding to respective segments of the simulated population. The first and second time series of segmentation states are generated by respectively processing the data set through a first and a second simulation each comprising iterative application of a plurality of event functions. The first and the second simulation differ in at least one capacity, e.g., one including a first event function configured with a first parameter, and the second not. Analysis of differences between the first and second time series may be compared to analysis of one of the time series using a subject model. The comparison is then used to validate the model or demonstrate accuracies, inaccuracies, and/or model bias with respect to a performance metric.

Abstract: Methods, systems, and apparatus including computer programs encoded on computer-readable storage media are provided for evaluating an attribution model, based on simulated activity streams. Parameters are specified that describe how users behave in the absence and presence of advertising, and parameters that regulate advertising in a simulation. A first set of simulated activity streams is generated with advertising turned on, observational metrics associated with the first set of streams are determined, and an attribution model is applied to determine a first fraction of incremental conversions associated with one or more advertising channels. Further sets of simulated activity streams are generated, each with a single advertising channel turned off, observational metrics associated with the further sets of streams are determined, and a second fraction of incremental conversions is determined for each advertising channel.

Abstract: Methods, systems, and apparatus including computer programs encoded on computer-readable storage media are provided for evaluating attribution models and comparing estimates produced by the attribution models with causal measurements from controlled experiments. An attribution model is identified for use in determining an estimate of an effectiveness of a campaign. An experiment is identified including experiment data that reflects implementation of a change in an experiment environment including identifying a measure of effectiveness of the change within a predetermined confidence level. The estimate is evaluated as compared to the identified measure of effectiveness. A determination is made that the attribution model is an effective measure of the change for the campaign, based at least in part on the evaluating.

Abstract: The present disclosure includes systems and techniques relating to identifying value marketing activities. In some implementations, an apparatus, systems, or methods can include receiving conversion path information including data relating to user interactions with a content item associated with a marketing activity, determining a first attribution credit by applying a first attribution model to the received information, and a second attribution credit by applying a second attribution model to the received information, determining an attribution contrast ratio based on the first and second attribution credit, identifying an opportunity based on the determined attribution contrast ratio, and presenting a recommendation for the marketing activity based on the identified opportunity.

Abstract: The present disclosure includes systems and techniques relating to identifying value marketing activities. In some implementations, an apparatus, systems, or methods can include receiving conversion path information including data relating to user interactions with a content item associated with a marketing activity, determining a first attribution credit by applying a first attribution model to the received information, and a second attribution credit by applying a second attribution model to the received information, determining an attribution contrast ratio based on the first and second attribution credit, identifying an opportunity based on the determined attribution contrast ratio, and presenting a recommendation for the marketing activity based on the identified opportunity.

Abstract: The present disclosure includes systems and techniques relating to identifying value marketing activities. In some implementations, an apparatus, systems, or methods can include receiving conversion path information including data relating to user interactions with a content item associated with a marketing activity, determining a first attribution credit by applying a first attribution model to the received information, and a second attribution credit by applying a second attribution model to the received information, determining an attribution contrast ratio based on the first and second attribution credit, identifying an opportunity based on the determined attribution contrast ratio, and presenting a recommendation for the marketing activity based on the identified opportunity.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for evaluating clusters of geographic entities, for example, to be used in a randomized geographic experiment. One method includes using a clustering algorithm to cluster geographic entities into a set of clusters, and identifying whether each geographic entity is an ambiguously classified entity or a definitively classified entity. The method further includes determining a measurement for the set of clusters according to a quantification of an attribute of the definitively classified entities and the ambiguously classified entities. Similar measurements can be calculated for other sets of clusters, and the clusters can be compared according to their measurements.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for evaluating clusters of geographic entities, for example, to be used in a randomized geographic experiment. One method includes using a clustering algorithm to cluster geographic entities into a set of clusters, and identifying whether each geographic entity is an ambiguously classified entity or a definitively classified entity. The method further includes determining a measurement for the set of clusters according to a quantification of an attribute of the definitively classified entities and the ambiguously classified entities. Similar measurements can be calculated for other sets of clusters, and the clusters can be compared according to their measurements.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing geography-based advertising experiments. One method includes receiving pre-spend data for geographic regions, identifying the geographic regions as control or treatment regions, obtaining change in ad spend data for each region, estimating a variance in a return on ad spend according to the pre-spend and the change in ad spend data. The method further includes determining whether the variance satisfies an acceptance criterion, and either allocating the change in ad spend data for use in an advertising experiment or selecting different change in ad spend data. Another method includes receiving pre-spend data for geographic regions, determining a change in ad spend for each geographic region, fitting a model to the pre-spend data, the change in ad spend, and test data, and determining a return on ad spend from the fitted model.