Abstract: In implementations of systems for generating recommendations, a computing device implements a recommendation system to receive prior interaction data describing prior interactions of entities with items. The recommendation system processes the prior interaction data and segments the entities into a first set and a second set. The entities included in the first set have greater numbers of prior interactions with the items than the entities included in the second set. The recommendation system then generates subset data describing a subset of the entities in the first set. This subset excludes entities having numbers of the prior interactions with the items below a threshold. The recommendation system forms a recommendation model based on the subset data and the system uses the recommendation model to generate a recommendation for display in a user interface.

Abstract: Certain embodiments involve generating personalized recommendations for users by inferring a propensity of each individual user to accept a recommendation. For example, a system generates a personalized user model based on a historical transition matrix that provides state transition probabilities from a general population of users. The probabilities are adjusted based on the propensity for a user to accept a recommendation. The system determines a recommended action for the user to transition between predefined states based on the user model. Once the user has performed an activity that transitions from a current state, the system adjusts a probability distribution for an estimate of the propensity based on whether the activity is the recommended action.

Abstract: Methods, systems, and non-transitory computer readable storage media are disclosed for utilizing offline models to warm start online bandit learner models. For example, the disclosed system can determine relevant offline models for an environment based on reward estimate differences between the offline models and the online model. The disclosed system can then utilize the relevant offline models (if any) to select an arm for the environment. The disclosed system can update the online model based on observed rewards for the selected arm. Additionally, the disclosed system can also use entropy reduction of arms to determine the utility of the arms in differentiating relevant and irrelevant offline models. For example, the disclosed system can select an arm based on a combination of the entropy reduction of the arm and the reward estimate for the arm and use the observed reward to update an observation history.

Abstract: Systems and methods are described for a decision-making process that includes an increasing set of actions, compute a policy function for a Markov decision process (MDP) for the decision-making process, wherein the policy function is computed based on a state conditional function mapping states into an embedding space, an inverse dynamics function mapping state transitions into the embedding space, and an action selection function mapping the elements of the embedding space to actions, identify an additional set of actions in the increasing set of actions, update the inverse dynamics function based at least in part on the additional set of actions, update the policy function based on the updated inverse dynamics function and parameters learned during the computing the policy function, and select an action based on the updated policy function.

Abstract: Systems and methods are described for a decision-making process including actions characterized by stochastic availability, provide an Markov decision process (MDP) model that includes a stochastic action set based on the decision-making process, compute a policy function for the MDP model using a policy gradient based at least in part on a function representing the stochasticity of the stochastic action set, identify a probability distribution for one or more actions available at a time period using the policy function, and select an action for the time period based on the probability distribution.

Abstract: Certain embodiments involve learning user preferences and predicting user behavior based on sequential user behavior data. For example, a system obtains data about a sequence of prior actions taken by multiple users. The system determines a similarity between a prior action taken by the various users and groups the various users into groups or clusters based at least in part on the similarity. The system trains a machine-learning algorithm such that the machine-learning algorithm can be used to predict a subsequent action of a user among the various users based on the various clusters. The system further obtains data about a current action of a new user and determines which of the clusters to associate with the new user based on the new user's current action. The system determines an action to be recommended to the new user based on the cluster associated with the new user. The action can include a series or sequence of actions to be taken by the new user.

Abstract: The present disclosure relates to generating proposed digital actions in high-dimensional action spaces for client devices utilizing reinforcement learning models. For example, the disclosed systems can utilize a supervised machine learning model to train a latent representation decoder to determine proposed digital actions based on latent representations. Additionally, the disclosed systems can utilize a latent representation policy gradient model to train a state-based latent representation generation policy to generate latent representations based on the current state of client devices. Subsequently, the disclosed systems can identify the current state of a client device and a plurality of available actions, utilize the state-based latent representation generation policy to generate a latent representation based on the current state, and utilize the latent representation decoder to determine a proposed digital action from the plurality of available actions by analyzing the latent representation.

Abstract: Optimizing customer lifetime value (LTV) techniques are described. In one or more implementations, a simulator is configured to derive a prediction model based on data indicative of user interaction online with marketing offers. The prediction model may be produced by automatically classifying variables according to feature types and matching each feature type to a response function that defines how the variable responds to input actions. The classification of variables and/or corresponding response functions per the prediction model may consider dependencies between variables and dependencies between successive states. An evaluator may then be invoked to apply the prediction model to test a proposed marketing strategy offline. Application of the prediction model is designed to predict user response to simulated offers/actions and enable evaluation of marketing strategies with respect to one or more long-term objectives.

Abstract: The present disclosure relates to generating and modifying recommended event sequences utilizing a dynamic user preference interface. For example, in one or more embodiments, the system generates a recommended event sequence using a recommendation model trained based on a plurality of historical event sequences. The system then provides, for display via a client device, the recommendation, a plurality of interactive elements for entry of user preferences, and a visual representation of historical event sequences. Upon detecting input of user preferences, the system can modify a reward function of the recommendation model and provide a modified recommended event sequence together with the plurality of interactive elements. In one or more embodiments, as a user enters user preferences, the system additionally modifies the visual representation to display subsets of the plurality of historical event sequences corresponding to the preferences.

Abstract: A digital medium environment is described to recommend advertisements using ranking functions. A ranking function is configured to compute a score by applying a user context vector associated with a user to individual ranking weight vectors associated with advertisements, and provide the advertisement with the highest score to the user. In order to learn the ranking weight vectors for the ranking function, training data is obtained that includes user interactions with advertisements during previous sessions as well as user context vectors. The ranking weight vectors for the ranking function associated with each advertisement can then be learned by controlling the score generated by the ranking function to be higher for positive interactions than the negative interactions. To do so, the ranking weight vectors may be learned by optimizing an area under the curve ranking loss (AUCL) for the ranking function.

Abstract: Systems and methods provide a recommendation system for recommending sequential content. The training of a reinforcement learning (RL) agent is bootstrapped from passive data. The RL agent of the sequential recommendations system is trained using the passive data over a number of epochs involving interactions between the sequential recommendation system and user devices. At each epoch, available active data from previous epochs is obtained, and transition probabilities are generated from the passive data and at least one parameter derived from the currently available active data. Recommended content is selected based on a current state and the generated transition probabilities, and the active data is updated from the current epoch based on the recommended content and a resulting new state. A clustering approach can also be employed when deriving parameters from active data to balance model expressiveness and data sparsity.

Abstract: The present disclosure relates to recommending points of interest to a plurality of users based on a type of each user as well as constraints associated with the points of interest. For example, one or more embodiments determine a user type for each user and determine user preferences based on the user type. Additionally, the system can determine resource constraints associated with each point of interest, indicating limitations on the capacity of each associated resource. The system can then provide recommendations to the plurality of users based on the user types and the resource constraints. In particular, the system can recommend points of interest that satisfy the preferences corresponding to each user type subject to the resource constraints of each point of interest. For example, one or more embodiments involve solving a linear program that takes into account user types to obtain recommendation policies subject to the resource constraints.

Abstract: Systems and methods provide for bootstrapping a sequential recommendation system from passive data. A learning agent of the sequential recommendations system is trained using the passive data over a number of epochs involving interactions between the sequential recommendation system and user devices. At each epoch, available active data from previous epochs is obtained, and transition probabilities are generated from the passive data and at least one parameter derived from the currently available active data. A recommended action is selected given a current state and the generated transition probabilities, and the active data is updated from the current epoch based on the recommended action and a resulting new state. A clustering approach can also be employed when deriving parameters from active data to balance model expressiveness and data sparsity.

Abstract: Metric forecasting techniques and systems in a digital medium environment are described that leverage similarity of elements, one to another, in order to generate a forecast value for a metric for a particular element. In one example, training data is received that describes a time series of values of the metric for a plurality of elements. The model is trained to generate the forecast value of the metric, the training using machine learning of a neural network based on the training data. The training includes generating dimensional-transformation data configured to transform the training data into a simplified representation to determine similarity of the plurality of elements, one to another, with respect to the metric over the time series. The training also includes generating model parameters of the neural network based on the simplified representation to generate the forecast value of the metric.

Abstract: Metric forecasting techniques in a digital medium environment are described. A time series interval is identified by an analytics system that is exhibited by input usage data. The input usage data describes values of a metric involved in the provision of the digital content by a service provider system. A determination is then made by the analytics system as to whether historical usage data includes the identified time series interval. A forecast model is then selected by the analytics system from a plurality of forecast models based on a result of the determination and the identified time series interval. Forecast data is then generated by a forecast module of the analytics system. The forecast data is configured to predict at least one value of the metric based on the selected forecast model, a result of the determination, and the input usage data.

Abstract: Certain embodiments involve generating personalized recommendations for users by inferring a propensity of each individual user to accept a recommendation. For example, a system generates a personalized user model based on a historical transition matrix that provides state transition probabilities from a general population of users. The probabilities are adjusted based on the propensity for a user to accept a recommendation. The system determines a recommended action for the user to transition between predefined states based on the user model. Once the user has performed an activity that transitions from a current state, the system adjusts a probability distribution for an estimate of the propensity based on whether the activity is the recommended action.

Abstract: Certain embodiments involve learning user preferences and predicting user behavior based on sequential user behavior data. For example, a system obtains data about a sequence of prior actions taken by multiple users. The system determines a similarity between a prior action taken by the various users and groups the various users into groups or clusters based at least in part on the similarity. The system trains a machine-learning algorithm such that the machine-learning algorithm can be used to predict a subsequent action of a user among the various users based on the various clusters. The system further obtains data about a current action of a new user and determines which of the clusters to associate with the new user based on the new user's current action. The system determines an action to be recommended to the new user based on the cluster associated with the new user. The action can include a series or sequence of actions to be taken by the new user.

Abstract: A digital medium environment is described to recommend advertisements using ranking functions. A ranking function is configured to compute a score by applying a user context vector associated with a user to individual ranking weight vectors associated with advertisements, and provide the advertisement with the highest score to the user. In order to learn the ranking weight vectors for the ranking function, training data is obtained that includes user interactions with advertisements during previous sessions as well as user context vectors. The ranking weight vectors for the ranking function associated with each advertisement can then be learned by controlling the score generated by the ranking function to be higher for positive interactions than the negative interactions. To do so, the ranking weight vectors may be learned by optimizing an area under the curve ranking loss (AUCL) for the ranking function.

Abstract: Risk quantification, policy search, and automated safe policy deployment techniques are described. In one or more implementations, techniques are utilized to determine safety of a policy, such as to express a level of confidence that a new policy will exhibit an increased measure of performance (e.g., interactions or conversions) over a currently deployed policy. In order to make this determination, reinforcement learning and concentration inequalities are utilized, which generate and bound confidence values regarding the measurement of performance of the policy and thus provide a statistical guarantee of this performance. These techniques are usable to quantify risk in deployment of a policy, select a policy for deployment based on estimated performance and a confidence level in this estimate (e.g.

Abstract: Risk quantification, policy search, and automated safe policy deployment techniques are described. In one or more implementations, techniques are utilized to determine safety of a policy, such as to express a level of confidence that a new policy will exhibit an increased measure of performance (e.g., interactions or conversions) over a currently deployed policy. In order to make this determination, reinforcement learning and concentration inequalities are utilized, which generate and bound confidence values regarding the measurement of performance of the policy and thus provide a statistical guarantee of this performance. These techniques are usable to quantify risk in deployment of a policy, select a policy for deployment based on estimated performance and a confidence level in this estimate (e.g.