Abstract: The present disclosure relates to systems, methods, and non-transitory computer readable media that provide a trajectory-based explainability framework for reinforcement learning models. For example, the disclosed systems generate trajectory clusters from trajectories utilized to train a reinforcement learning agent. In some embodiments, the disclosed system generates a complementary target data set by removing a target trajectory cluster from the trajectory clusters. In some cases, the disclosed system trains a test reinforcement learning agent utilizing the complementary target data set and generates a cluster attribution by comparing the result of the test reinforcement learning agent with the result of the reinforcement learning agent.

Abstract: A single unified machine learning model (e.g., a neural network) is trained to perform both supervised event predictions and unsupervised time-varying clustering for a sequence of events (e.g., a sequence representing a user behavior) using sequences of events for multiple users using a combined loss function. The unified model can then be used for, given a sequence of events as input, predict a next event to occur after the last event in the sequence and generate a clustering result by performing a clustering operation on the sequence of events. As part of predicting the next event, the unified model is trained to predict an event type for the next event and a time of occurrence for the next event. In certain embodiments, the unified model is a neural network comprising a recurrent neural network (RNN) such as an Long Short Term Memory (LSTM) network.

Abstract: Systems and methods for machine learning are described. Embodiments of the present disclosure receive state information that describes a state of a decision making agent in an environment; compute an action vector from an action embedding space based on the state information using a policy neural network of the decision making agent, wherein the policy neural network is trained using reinforcement learning based on a topology loss that constrains changes in a mapping between an action set and the action embedding space; and perform an action that modifies the state of the decision making agent in the environment based on the action vector, wherein the action is selected based on the mapping.

Abstract: A control system facilitates active management of a streaming data system. Given historical data traffic for each data stream processed by a streaming data system, the control system uses a machine learning model to predict future data traffic for each data stream. The control system selects a matching between data streams and servers for a future time that minimizes a cost comprising a switching cost and a server imbalance cost based on the predicted data traffic for the future time. In some configurations, the matching is selected using a planning window comprising a number of future time steps dynamically selected based on uncertainty associated with the predicted data traffic. Given the selected matching, the control system may manage the streaming data system by causing data streams to be moved between servers based on the matching.

Abstract: Techniques are provided for analyzing user actions that have occurred over a time period. The user actions can be, for example, with respect to the user's navigation of content or interaction with an application. Such user data is provided in an action string, which is converted into a highly searchable format. As such, the presence and frequency of particular user actions and patterns of user actions within an action string of a particular user, as well as among multiple action strings of multiple users, are determinable. Subsequences of one or more action strings are identified and both the number of action strings that include a particular subsequence and the frequency that a particular subsequence is present in a given action string are determinable. The conversion involves breaking that string into a sorted list of locations for the actions within that string. Queries can be readily applied against the sorted list.

Abstract: The present disclosure relates to generating and modifying recommended event sequences utilizing a dynamic user preference interface. For example, in one or more embodiments, the system generates a recommended event sequence using a recommendation model trained based on a plurality of historical event sequences. The system then provides, for display via a client device, the recommendation, a plurality of interactive elements for entry of user preferences, and a visual representation of historical event sequences. Upon detecting input of user preferences, the system can modify a reward function of the recommendation model and provide a modified recommended event sequence together with the plurality of interactive elements. In one or more embodiments, as a user enters user preferences, the system additionally modifies the visual representation to display subsets of the plurality of historical event sequences corresponding to the preferences.

Abstract: The present disclosure describes methods, systems, and non-transitory computer-readable media for generating a projected value metric that projects a performance of a target policy within a digital action space. For instance, in one or more embodiments, the disclosed systems identify a target policy for performing digital actions represented within a digital action space. The disclosed systems further determine a set of sampled digital actions performed according to a logging policy and represented within the digital action space. Utilizing an embedding model, the disclosed systems generate a set of action embedding vectors representing the set of sampled digital actions within an embedding space. Further, utilizing the set of action embedding vectors, the disclosed systems generate a projected value metric indicating a projected performance of the target policy.

Abstract: In implementations of item transfer control systems, a computing device implements a transfer system to receive input data describing types of requested items and corresponding quantities of the types of requested items to receive at each of a plurality of destination sites and types of available items and corresponding quantities of the types of available items that are available at each of a plurality of source sites. The transfer system constructs a flow network having a source node for each of the plurality of the source sites and a destination node for each of the plurality of the destination sites. An integral approximate solution is generated that transfers the corresponding quantities of the types of requested items to each of the plurality of the destination sites using a maximum flow solver and the flow network.

Abstract: Systems and methods for machine learning are described. Embodiments of the present disclosure receive state information that describes a state of a decision making agent in an environment; compute an action vector from an action embedding space based on the state information using a policy neural network of the decision making agent, wherein the policy neural network is trained using reinforcement learning based on a topology loss that constrains changes in a mapping between an action set and the action embedding space; and perform an action that modifies the state of the decision making agent in the environment based on the action vector, wherein the action is selected based on the mapping.

Abstract: A control system facilitates active management of a streaming data system. Given historical data traffic for each data stream processed by a streaming data system, the control system uses a machine learning model to predict future data traffic for each data stream. The control system selects a matching between data streams and servers for a future time that minimizes a cost comprising a switching cost and a server imbalance cost based on the predicted data traffic for the future time. In some configurations, the matching is selected using a planning window comprising a number of future time steps dynamically selected based on uncertainty associated with the predicted data traffic. Given the selected matching, the control system may manage the streaming data system by causing data streams to be moved between servers based on the matching.

Abstract: Methods, systems, and non-transitory computer readable storage media are disclosed for utilizing offline models to warm start online bandit learner models. For example, the disclosed system can determine relevant offline models for an environment based on reward estimate differences between the offline models and the online model. The disclosed system can then utilize the relevant offline models (if any) to select an arm for the environment. The disclosed system can update the online model based on observed rewards for the selected arm. Additionally, the disclosed system can also use entropy reduction of arms to determine the utility of the arms in differentiating relevant and irrelevant offline models. For example, the disclosed system can select an arm based on a combination of the entropy reduction of the arm and the reward estimate for the arm and use the observed reward to update an observation history.

Abstract: In implementations of item transfer control systems, a computing device implements a transfer system to receive input data describing types of requested items and corresponding quantities of the types of requested items to receive at each of a plurality of destination sites and types of available items and corresponding quantities of the types of available items that are available at each of a plurality of source sites. The transfer system constructs a flow network having a source node for each of the plurality of the source sites and a destination node for each of the plurality of the destination sites. An integral approximate solution is generated that transfers the corresponding quantities of the types of requested items to each of the plurality of the destination sites using a maximum flow solver and the flow network.

Abstract: Methods, systems, and non-transitory computer readable storage media are disclosed for utilizing offline models to warm start online bandit learner models. For example, the disclosed system can determine relevant offline models for an environment based on reward estimate differences between the offline models and the online model. The disclosed system can then utilize the relevant offline models (if any) to select an arm for the environment. The disclosed system can update the online model based on observed rewards for the selected arm. Additionally, the disclosed system can also use entropy reduction of arms to determine the utility of the arms in differentiating relevant and irrelevant offline models. For example, the disclosed system can select an arm based on a combination of the entropy reduction of the arm and the reward estimate for the arm and use the observed reward to update an observation history.

Abstract: The present disclosure relates to methods, systems, and non-transitory computer-readable media for determining a cognitive, action-selection bias of a user that influences how the user will select a sequence of digital actions for execution of a task. For example, the disclosed systems can identify, from a digital behavior log of a user, a set of digital action sequences that correspond to a set of sessions for a task previously executed by the user. The disclosed systems can utilize a machine learning model to analyze the set of sessions to generate session weights. The session weights can correspond to an action-selection bias that indicates an extent to which a future session for the task executed by the user is predicted to be influenced by the set of sessions. The disclosed systems can provide a visual indication of the action-selection bias of the user for display on a graphical user interface.

Abstract: An item recommendation system receives a set of recommendable items and a request to select, from the set of recommendable items, a contrast group. The item recommendation system selects a contrast group from the set of recommendable items by applying a recommendation model to the set of recommendable items. The recommendation model includes an item selection model configured to determine an unbiased conversion rate for each item of the set of recommendable items and select a recommended item from the set of recommendable items having a greatest unbiased conversion rate. The recommendation model includes a contrast group selection model configured to select, for the recommended item, a contrast group comprising the recommended item and one or more contrast items. The item recommendation system transmits the contrast group responsive to the request.

Abstract: The present disclosure relates to systems, methods, and non-transitory computer-readable media that generate digital item recommendations for client devices utilizing coagent recommendation models of a distributed asynchronous coagent network. Indeed, in one or more embodiments, the disclosed systems operate on an edge computing device of a distributed asynchronous coagent network. In some cases, the disclosed systems utilize recommendation scores generated at the edge computing device via local coagents and additional recommendation scores received from other coagents of other edge computing devices to generate a digital item recommendation. In some cases, the disclosed systems progressively refines the recommendation as delayed scores from the other coagents are received. Further, in some embodiments, the disclosed systems update parameters of the local coagents using local policy gradients determined from responses to the generated recommendations.

Abstract: Metric forecasting techniques and systems in a digital medium environment are described that leverage similarity of elements, one to another, in order to generate a forecast value for a metric for a particular element. In one example, training data is received that describes a time series of values of the metric for a plurality of elements. The model is trained to generate the forecast value of the metric, the training using machine learning of a neural network based on the training data. The training includes generating dimensional-transformation data configured to transform the training data into a simplified representation to determine similarity of the plurality of elements, one to another, with respect to the metric over the time series. The training also includes generating model parameters of the neural network based on the simplified representation to generate the forecast value of the metric.

Abstract: In implementations of item transfer control systems, a computing device implements a transfer system to receive input data describing types of requested items and corresponding quantities of the types of requested items to receive at each of a plurality of destination sites and types of available items and corresponding quantities of the types of available items that are available at each of a plurality of source sites. The transfer system constructs a flow network having a source node for each of the plurality of the source sites and a destination node for each of the plurality of the destination sites. An integral approximate solution is generated that transfers the corresponding quantities of the types of requested items to each of the plurality of the destination sites using a maximum flow solver and the flow network.

Abstract: Systems and methods are described for a decision-making process including actions characterized by stochastic availability, provide an Markov decision process (MDP) model that includes a stochastic action set based on the decision-making process, compute a policy function for the MDP model using a policy gradient based at least in part on a function representing the stochasticity of the stochastic action set, identify a probability distribution for one or more actions available at a time period using the policy function, and select an action for the time period based on the probability distribution.

Abstract: In implementations of item transfer control systems, a computing device implements a transfer system to receive input data describing types of requested items and corresponding quantities of the types of requested items to receive at each of a plurality of destination sites and types of available items and corresponding quantities of the types of available items that are available at each of a plurality of source sites. The transfer system constructs a flow network having a source node for each of the plurality of the source sites and a destination node for each of the plurality of the destination sites. An integral approximate solution is generated that transfers the corresponding quantities of the types of requested items to each of the plurality of the destination sites using a maximum flow solver and the flow network.