Abstract: Certain embodiments involve extracting seasonal, level, and spike components from a time series of metrics data, which describe interactions with an online service over a time period. For example, an analytical system decomposes the time series into latent components that include a seasonal component series, a level component series, a spike component series, and an error component series. The decomposition involves configuring an optimization algorithm with a constraint indicating that the time series is a sum of these latent components. The decomposition also involves executing the optimization algorithm to minimize an objective function subject to the constraint and identifying, from the executed optimization algorithm, the seasonal component series, the level component series, the spike component series, and the error component series that minimize the objective function. The analytical system outputs at least some latent components for anomaly-detection or data-forecasting.

Abstract: Certain embodiments involve extracting seasonal, level, and spike components from a time series of metrics data, which describe interactions with an online service over a time period. For example, an analytical system decomposes the time series into latent components that include a seasonal component series, a level component series, a spike component series, and an error component series. The decomposition involves configuring an optimization algorithm with a constraint indicating that the time series is a sum of these latent components. The decomposition also involves executing the optimization algorithm to minimize an objective function subject to the constraint and identifying, from the executed optimization algorithm, the seasonal component series, the level component series, the spike component series, and the error component series that minimize the objective function. The analytical system outputs at least some latent components for anomaly-detection or data-forecasting.

Abstract: Techniques are disclosed for identifying anomalies in small data sets, by identifying anomalies using a Generalized Extreme Student Deviate test (GESD test). In an embodiment, a data set, such as business data or a website metric, is checked for skewness and, if found to be skewed, is transformed to a normal distribution (e.g., by applying a Box-Cox transformation). The data set is checked for presence of trends and, if a trend is found, has the trend removed (e.g., by running a linear regression). In one embodiment, a maximum number of anomalies is estimated for the data set, by applying an adjusted box plot to the data set. The data set and the estimated number of anomalies is run through a GESD test, and the test identifies anomalous data points in the data set, based on the provided estimated number of anomalies. In an embodiment, a confidence interval is generated for the identified anomalies.

Abstract: This disclosure involves allocating content-delivery resources to electronic content-delivery channels based on attribution models accuracy. For instance, a simulation is executed that involves simulating user exposures, times between user exposures, and user responses. The simulation is performed based on parameters associated with simulating user exposures to electronic content-delivery channels and user responses to the user exposures. An accuracy of a channel attribution model when estimating an attribution of an electronic content-delivery channel to a user response is evaluated based on the simulation. A channel attribution model is selected based on the evaluation. An attribution of the electronic content-delivery channel is determined by applying the selected channel attribution model to actual user exposures and actual user responses.

Abstract: Techniques are described for jointly modeling knowledge tracing and hint-taking propensity. During a read phase, a co-learning model accepts as inputs an identification of a question and the current knowledge state for a learner, and the model predicts probabilities that the learner will answer the question correctly and that the learner will use a learning aid (e.g., accept a hint). The predictions are used to personalize an e-learning plan, for example, to provide a personalized assessment. By using these predictions to personalize a learner's experience, for example, by offering hints at optimal times, the co-learning system increases efficiencies in learning and improves learning outcomes. Once a learner has interacted with a question, the interaction is encoded and provided to the co-learning model to update the learner's knowledge state during an update phase.

Abstract: This disclosure involves predictive modeling with entity representations computed from neural network models simultaneously trained on multiple tasks. For example, a method includes a processing device performing operations including accessing input data for an entity and transforming the input data into a dense vector entity representation representing the entity. Transforming the input data includes applying, to the input data, a neural network including simultaneously trained propensity models. Each propensity model predicts a different task based on the input data. Transforming the input data also includes extracting the dense vector entity representation from a common layer of the neural network to which the propensity models are connected.

Abstract: The present disclosure is directed toward systems and methods for identifying contributing factors associated with a multi-variable metric anomaly. One or more embodiments described herein identify one or more contributing factors that led to an anomaly in a multi-variable metric by calculating linearizing weights such that the total deviation in the multi-variable metric can be written as a weighted sum of deviations for dimension elements associated with the multi-variable metric.

Abstract: Techniques for analyzing marketing channels are described. Users are exposed to the marketing channels. User responses (e.g., purchases and no-purchases) to the exposures are tracked. Upon a request from a marketer to analyze an attribution of a marketing channel, the user responses are analyzed. The attribution represents the credit that the marketing channel should get for influencing the users exposed thereto into exhibiting a particular user response (e.g., a purchase). The analysis involves multiple steps. In a first step, a non-parametric estimation is used to generate a value function at a user-level. In a second step, a coalitional game approach is used to estimate the attribution based on the value function. A response is provided to the marketer with data about the attribution.

Abstract: Techniques for managing a marketing campaign of a marketer are described. In an example, the marketing campaign uses multiple marketing channels. Attribution of each marketing channel to a user conversion is estimated. Usage of a marketing channel within the marketing campaign is set according to the respective attribution. A marketing channel attribution model is selected from candidate marketing channel attribution models and is applied to estimate the attributions. The selection is based on the accuracy of each of the models associated with estimating the attributions given a set of parameters. To evaluate the accuracy, user journeys are simulated given the set of parameters. True attributions of each marketing channel are determined from the simulation. Each of the marketing channel attribution models is also applied to the simulation to generate estimated attributions. The true and estimated attributions are compared to derive the accuracies.

Abstract: Embodiments of the present invention relate to providing business customers with predictive capabilities, such as identifying valuable customers or estimating the likelihood that a product will be purchased. An adaptive sampling scheme is utilized, which helps generate sample data points from large scale data that is imbalanced (for example, digital website traffic with hundreds of millions of visitors but only a small portion of them are of interest). In embodiments, a stream of sample data points is received. Positive samples are added to a positive list until the desired number of positives is reached and negative samples are added to a negative list until the desired number of negative samples is reached. The positive list and the negative list can then be combined, shuffled, and fed into a prediction model.

Abstract: Certain embodiments involve extracting seasonal, level, and spike components from a time series of metrics data, which describe interactions with an online service over a time period. For example, an analytical system decomposes the time series into latent components that include a seasonal component series, a level component series, a spike component series, and an error component series. The decomposition involves configuring an optimization algorithm with a constraint indicating that the time series is a sum of these latent components. The decomposition also involves executing the optimization algorithm to minimize an objective function subject to the constraint and identifying, from the executed optimization algorithm, the seasonal component series, the level component series, the spike component series, and the error component series that minimize the objective function. The analytical system outputs at least some latent components for anomaly-detection or data-forecasting.

Abstract: Techniques for automatic identification of sources of web metric changes are described. In one or more implementations, changes in a web metric that indicate a measurable attribute associated with a website are determined, and the web metric is analyzed to identify sources that contributed to the changes in the web metric. In implementations, data is queried to obtain actual values for dimension elements along one or more dimensions of the web metric. In addition, expected values for the dimension elements are estimated along the dimensions of the web metric based on historical data. Then, deviations between the actual values and the expected values are calculated by using comparable statistics. Subsequently, the comparable statistics can be analyzed to identify corresponding dimension elements as the sources that contributed to the changes in the web metric.

Abstract: A computer based decision simulation tool system that includes data storage containing sales data for a plurality of products in a product line of a single brand. The sales data is organized to include quantity sold, selling price and sale date of a product over a period of time, at a predetermined level of temporal granularity. A processor is operatively coupled to the storage, and the processor is configured to execute instructions that when executed cause the processor to retrieve selected portions of the sales data. The processor operates to identify dependencies among products within the product line to generate a cross-product price elasticity that is indicative of percentage change in quantity sold of a focal product with respect to one percentage change in price of a different product in the product line. The process further operates to respond to user inputs to provide visual indications of the cross-product price elasticity.

Abstract: Techniques of forecasting web metrics involve generating, prior to the end of a period of time, a probability of a metric taking on an anomalous value, e.g., a value indicative of an anomaly with respect to web traffic, at the end of the period based on previous values of the metric. Such a probability is based on a distribution of predicted values of the metric at some previous period of time. For example, a web server may use actual values of the number of bounces collected at hourly intervals in the middle of a day to predict a number of bounces at the end of the current day. Further, the web server may also compute a confidence interval to determine whether a predicted end-of-day number of bounces may be considered anomalous. The width of the confidence interval indicates the probability that a predicted end-of-day number of bounces has an anomalous value.

Abstract: Embodiments are disclosed for bundling and arranging online content fragments for presentation based on content-specific metrics and inter-content constraints. For example, a content management application accesses candidate content fragments, a content-specific metric, and an inter-content constraint. The content management application computes minimum and maximum contribution values for the candidate content fragments. The content management application selects, based on the computed minimum and maximum contribution values, a subset of the candidate content fragments. The content management application applies, subject to the inter-content constraint, a bundle-selection function to the selected candidate content fragments and thereby identifies a bundle of online content fragments. The content management application outputs the identified bundle of online content fragments for presentation via an online service.

Abstract: Techniques are disclosed for identifying anomalies in small data sets, by identifying anomalies using a Generalized Extreme Student Deviate test (GESD test). In an embodiment, a data set, such as business data or a website metric, is checked for skewness and, if found to be skewed, is transformed to a normal distribution (e.g., by applying a Box-Cox transformation). The data set is checked for presence of trends and, if a trend is found, has the trend removed (e.g., by running a linear regression). In one embodiment, a maximum number of anomalies is estimated for the data set, by applying an adjusted box plot to the data set. The data set and the estimated number of anomalies is run through a GESD test, and the test identifies anomalous data points in the data set, based on the provided estimated number of anomalies. In an embodiment, a confidence interval is generated for the identified anomalies.

Abstract: Techniques are disclosed for identifying, assessing, and presenting differences between segments of customers. The techniques identify differences in categorical features of the customers in two segments. The techniques use observed data to identify differences in a categorical feature. The techniques then assess whether the observed difference is a real difference applicable to the entire customer population or the result of random chance. The categorical features with the most significant differences (i.e., unlikely due to random chance) are presented, for example, to allow a marketer to easily appreciate the most significant segment differences. Certain techniques account for segment overlap (i.e., customers being in both segments) in assessing whether differences are due to random chance. Certain techniques limit the presented categorical features to account for common knowledge and/or false testing issues.

Abstract: Metric anomaly detection techniques in a digital medium environment are described. An input time interval is identified by an analytics system for the presence of an anomaly of a metric. Inclusion of a scheduled event in the input time interval is identified, and a historical time interval is determined that also includes the scheduled event. Usage data describing values of the metric is then obtained for both the input time interval and the historical time interval. The usage data corresponding to the input time interval is then compared with the usage data corresponding to the historical time interval to detect effects of the scheduled event and whether the input time interval includes an anomaly in the metric.

Abstract: Methods and systems are provided herein for summarizing a set of anomalies corresponding to a group of metrics of interest to a monitoring system user. Initially, a set of anomalies corresponding to a group of metrics is identified as having values that are outside of a predetermined range. A correlation value is determined for at least a portion of pairs of anomalies in the set of anomalies. For each anomaly in the set of anomalies, an informativeness value is computed that indicates how informative each anomaly in the set of anomalies is to the monitoring system user. The correlation values and the informativeness values are then used to identify at least one key anomaly and a plurality of non-key anomalies from the set of anomalies. A summary is generated of the identified at least one key anomaly to provide information to the monitoring system user about the set of anomalies for a particular time period.

Abstract: In various implementations, a method includes receiving a set of time series data that corresponds to a metric. A seasonal pattern is extracted from the set of time series data and the extracted seasonal pattern is filtered from the set of time series data. A predictive model is generated from the filtered set of data. The extracted seasonal pattern is filtered from another set of time series data where the second set of time series data corresponds to the metric. The filtered second set of time series data is compared to the predictive model. An alert is generated to a user for a value within the filtered second set of time series data which falls outside of the predictive model.