Abstract: Sequential hypothesis testing in a digital medium environment is described using continuous data. To begin, a model is received that defines at least one data distribution. Testing data is also received that describes an effect of user interactions with the plurality of options of digital content on achieving an action using continuous non-binary data. Values of parameters of the model are then estimated for each option of the plurality of options based on the testing data. In one example. A variance estimate is then generated based on the estimated values of the parameters of the model for each option of the plurality of options. From this, a determination is made as to a decision boundary based on the variance estimate and an estimate for a mean value of each option of the plurality of options based on the testing data.

Abstract: Metric forecasting techniques and systems in a digital medium environment are described that leverage similarity of elements, one to another, in order to generate a forecast value for a metric for a particular element. In one example, training data is received that describes a time series of values of the metric for a plurality of elements. The model is trained to generate the forecast value of the metric, the training using machine learning of a neural network based on the training data. The training includes generating dimensional-transformation data configured to transform the training data into a simplified representation to determine similarity of the plurality of elements, one to another, with respect to the metric over the time series. The training also includes generating model parameters of the neural network based on the simplified representation to generate the forecast value of the metric.

Abstract: Methods, systems, and non-transitory computer readable storage media are disclosed for determining parameters for digital campaign content in connection with executing digital campaigns using a rank-one assumption and a best-arm identification algorithm. For example, the disclosed system alternately explores response data in the first dimension and response data in the second dimension using the rank-one assumption and the best-arm identification algorithm to estimate highest sampling values from each dimension. In one or more embodiments, the disclosed system uses the estimated highest sampling values from the first and second dimension to determine a combination with a highest sampling value in a parameter matrix constructed based on the first dimension and the second dimension, and then executes the digital campaign using the determined combination.

Abstract: Methods, systems, and non-transitory computer readable storage media are disclosed for determining parameters for digital campaign content in connection with executing digital campaigns using a rank-one assumption and a best-arm identification algorithm. For example, the disclosed system alternately explores response data in the first dimension and response data in the second dimension using the rank-one assumption and the best-arm identification algorithm to estimate highest sampling values from each dimension. In one or more embodiments, the disclosed system uses the estimated highest sampling values from the first and second dimension to determine a combination with a highest sampling value in a parameter matrix constructed based on the first dimension and the second dimension, and then executes the digital campaign using the determined combination.

Abstract: Techniques are provided for k-best option identification of options subject to a supplied tolerance. One technique includes: sampling the options for a first period on a plurality of computers; computing an average and a sample count for each option based on the sampling; splitting the options into a highest group and a lowest group based on the computed averages; selecting a weakest one of the highest group (option A) and a strongest one of the lowest group (option B); and deciding whether or not to terminate based on the supplied tolerance and the selecting of options A and B. In some cases, the technique further includes outputting the highest group and terminating in response to a termination decision; otherwise continue with sampling options A and B for a next period; and updating the computed average and the sample count for options A and B based on corresponding next period sampling.

Abstract: Methods, systems, and non-transitory computer readable storage media are disclosed for determining parameters for digital campaign content in connection with executing digital campaigns using a rank-one assumption and a best-arm identification algorithm. For example, the disclosed system alternately explores response data in the first dimension and response data in the second dimension using the rank-one assumption and the best-arm identification algorithm to estimate highest sampling values from each dimension. In one or more embodiments, the disclosed system uses the estimated highest sampling values from the first and second dimension to determine a combination with a highest sampling value in a parameter matrix constructed based on the first dimension and the second dimension, and then executes the digital campaign using the determined combination.

Abstract: Sequential hypothesis testing in a digital medium environment is described using continuous data. To begin, a model is received that defines at least one data distribution. Testing data is also received that describes an effect of user interactions with the plurality of options of digital content on achieving an action using continuous non-binary data. Values of parameters of the model are then estimated for each option of the plurality of options based on the testing data. In one example. A variance estimate is then generated based on the estimated values of the parameters of the model for each option of the plurality of options. From this, a determination is made as to a decision boundary based on the variance estimate and an estimate for a mean value of each option of the plurality of options based on the testing data.

Abstract: Certain embodiments involve determining and outputting correlations between metrics in large-scale web analytics datasets. For example, a processor identifies pairs of data metrics in a web analytics data set and determines a Maximal Information Coefficient (MIC) score for each pair of data metrics that indicates a strength of a correlation between the pair of data metrics. The processor generates an interactive user interface that graphically displays each pair of correlated data metrics having an MIC score above a threshold and the interactive user interface indicates the strength of the correlation between each displayed pair of correlated data metrics. The processor receives user input indicating an adjustment to the threshold and modifies the interactive user interface in response to receiving the user input by adding pairs of correlated data metrics to, or removing pairs of correlated metrics from, the interactive user interface based on the adjustment to the threshold.

Abstract: Sequential hypothesis testing in a digital medium environment is described using continuous data. To begin, a model is received that defines at least one data distribution. Testing data is also received that describes an effect of user interactions with the plurality of options of digital content on achieving an action using continuous non-binary data. Values of parameters of the model are then estimated for each option of the plurality of options based on the testing data. In one example. A variance estimate is then generated based on the estimated values of the parameters of the model for each option of the plurality of options. From this, a determination is made as to a decision boundary based on the variance estimate and an estimate for a mean value of each option of the plurality of options based on the testing data.

Abstract: A digital medium environment includes an action processing application that performs actions including personalized recommendation. A learning algorithm operates on a sample-by-sample basis (e.g., each instance a user visits a web page) and recommends an optimistic action, such as an action found by maximizing an expected reward, or a base action, such as an action from a baseline policy with known expected reward, subject to a safety constraint. The safety constraint requires that the expected performance of playing optimistic actions is at least as good as a predetermined percentage of the known performance of playing base actions. Thus, the learning algorithm is conservative during exploratory early stages of learning, and does not play unsafe actions. Furthermore, since the learning algorithm is online and can learn with each sample, it converges quickly and is able to track time varying parameters better than learning algorithms that learn on a block basis.

Abstract: Sample size determination techniques in sequential hypothesis testing in a digital medium environment are described. The sample size may be determined before a test to define a number of samples (e.g., user interactions with digital marketing content) that are likely to be tested as part of the sequential hypothesis testing in order to achieve a result. The sample size may also be determined in real time to define a number of samples that likely remain for testing in order to achieve a result. The sample size may be determined in a variety of ways, such as through simulation, based on a gap between conversion rates for different options being tested, and so on.

Abstract: Sequential hypothesis testing techniques are described, which involve testing sequences of increasingly larger number of samples until a winner is determined. In particular, sequential hypothesis testing techniques is based on whether a result of a statistic has reached statistical significance that defines a confidence level in the accuracy of the results. Sequential hypothesis testing also permits the user to “peek” into the test through use of a user interface (e.g., dashboard) to monitor the test in real time as it is being run. Real time output of this information in a user interface as a part of sequential hypothesis testing may be leveraged in a variety of ways. In a first example, a user may make changes as the test is run. In another example, flexible execution is also made possible in that the test may continue to run even if initial accuracy guarantees have been met.

Abstract: Embodiments of the present invention are directed at providing a sequential multiple hypothesis testing system. In one embodiment, feedback is collected for hypothesis tests of a multiple hypothesis tests. Based on the collected feedback, a sequential p-value is calculated for each of the hypothesis tests utilizing a sequential statistic procedure that is designed to compare an alternative case with a base case for a respective hypothesis test. A sequential rejection procedure can then be applied to determine whether any of the hypothesis tests have concluded based on the respective p-value. A result of the determination can then be output to apprise a user of a state of the multiple hypothesis test. This process can then be repeated until a maximum sample size is reached, termination criterion is met, or all tests are concluded. Other embodiments may be described and/or claimed.

Abstract: Methods, systems, and non-transitory computer readable storage media are disclosed for determining parameters for digital campaign content in connection with executing digital campaigns using a rank-one assumption and a best-arm identification algorithm. For example, the disclosed system alternately explores response data in the first dimension and response data in the second dimension using the rank-one assumption and the best-arm identification algorithm to estimate highest sampling values from each dimension. In one or more embodiments, the disclosed system uses the estimated highest sampling values from the first and second dimension to determine a combination with a highest sampling value in a parameter matrix constructed based on the first dimension and the second dimension, and then executes the digital campaign using the determined combination.

Abstract: A digital medium environment is described to recommend advertisements using ranking functions. A ranking function is configured to compute a score by applying a user context vector associated with a user to individual ranking weight vectors associated with advertisements, and provide the advertisement with the highest score to the user. In order to learn the ranking weight vectors for the ranking function, training data is obtained that includes user interactions with advertisements during previous sessions as well as user context vectors. The ranking weight vectors for the ranking function associated with each advertisement can then be learned by controlling the score generated by the ranking function to be higher for positive interactions than the negative interactions. To do so, the ranking weight vectors may be learned by optimizing an area under the curve ranking loss (AUCL) for the ranking function.

Abstract: Systems and methods for selecting optimal policies that maximize expected return subject to given risk tolerance and confidence levels. In particular, methods and systems for selecting an optimal ad recommendation policy—based on user data, a set of ad recommendation policies, and risk thresholds—by sampling the user data and estimating gradients. The system or methods utilize the estimated gradients to select a good ad recommendation policy (an ad recommendation policy with high expected return) subject to the risk tolerance and confidence levels. To assist in selecting a risk-sensitive ad recommendation policy, a gradient-based algorithm is disclosed to find a near-optimal policy for conditional-value-at-risk (CVaR) risk-sensitive optimization.

Abstract: Certain embodiments involve determining and outputting correlations between metrics in large-scale web analytics datasets. For example, a processor identifies pairs of data metrics in a web analytics data set and determines a Maximal Information Coefficient (MIC) score for each pair of data metrics that indicates a strength of a correlation between the pair of data metrics. The processor generates an interactive user interface that graphically displays each pair of correlated data metrics having an MIC score above a threshold and the interactive user interface indicates the strength of the correlation between each displayed pair of correlated data metrics. The processor receives user input indicating an adjustment to the threshold and modifies the interactive user interface in response to receiving the user input by adding pairs of correlated data metrics to, or removing pairs of correlated metrics from, the interactive user interface based on the adjustment to the threshold.

Abstract: Techniques are provided for k-best option identification of options subject to a supplied tolerance. One technique includes: sampling the options for a first period on a plurality of computers; computing an average and a sample count for each option based on the sampling; splitting the options into a highest group and a lowest group based on the computed averages; selecting a weakest one of the highest group (option A) and a strongest one of the lowest group (option B); and deciding whether or not to terminate based on the supplied tolerance and the selecting of options A and B. In some cases, the technique further includes outputting the highest group and terminating in response to a termination decision; otherwise continue with sampling options A and B for a next period; and updating the computed average and the sample count for options A and B based on corresponding next period sampling.

Abstract: Influence maximization determination within a social network system is described. In one example, a subset is selected from a plurality of user accounts of a social network system. Exposure of digital marketing content is then caused to the subset of user accounts. A determination is made as to a probability of each user account of the plurality of user accounts as being influenced by the exposure of the digital marketing content to the subset of user accounts. The determined probability is then output, such as to control output of digital marketing content.

Abstract: Metric forecasting techniques and systems in a digital medium environment are described that leverage similarity of elements, one to another, in order to generate a forecast value for a metric for a particular element. In one example, training data is received that describes a time series of values of the metric for a plurality of elements. The model is trained to generate the forecast value of the metric, the training using machine learning of a neural network based on the training data. The training includes generating dimensional-transformation data configured to transform the training data into a simplified representation to determine similarity of the plurality of elements, one to another, with respect to the metric over the time series. The training also includes generating model parameters of the neural network based on the simplified representation to generate the forecast value of the metric.