Abstract: A user interface (UI), for analyzing model training runs, tracking and visualizing various aspects of machine learning experiments, can be used when training an artificial intelligent agent in, for example, a racing game environment. The UI can be web-based and can allow researchers to easily see the status of their experiments. The UI can include an experiment synchronized event viewer that can synchronizes visualizations, videos, and timeline/metrics graphs in the experiment. This viewer allows researchers to see how experiments unfold in great detail. The UI can further include experiment event annotations that can generate event annotations. These annotations can be displayed via the synchronized event viewer. The UI can be used to consider consolidated results across experiments and can further consider videos. For example, the UI can provide a reusable dashboard that can capture and compare metrics across multiple experiments.

Abstract: Experience replay (ER) is an important component of many deep reinforcement learning (RL) systems. However, uniform sampling from an ER buffer can lead to slow convergence and unstable asymptotic behaviors. Stratified Sampling from Event Tables (SSET), which partitions an ER buffer into Event Tables, each capturing important subsequences of optimal behavior. A theoretical advantage is proven over the traditional monolithic buffer approach and the combination of SSET with an existing prioritized sampling strategy can further improve learning speed and stability. Empirical results in challenging MiniGrid domains, benchmark RL environments, and a high-fidelity car racing simulator demonstrate the advantages and versatility of SSET over existing ER buffer sampling approaches.

Abstract: Systems and methods use deep, convolutional neural networks over exponentially long history windows to learn alphabets for context tree weighting (CTW) for prediction. Known issues of depth and breadth in conventional context tree weighting predictions are addressed by the systems and methods. To deal with depth, the history can be broken into time windows, permitting the ability to look exponentially far back while having less information the further one looks back. To deal with breadth, a deep neural network classifier can be used to learn to map arbitrary length histories to a small output symbol alphabet. The sequence of symbols produced by such a classifier over the history windows would then become the input sequence to CTW.

Abstract: An artificial intelligent agent can act as a player in a video game, such as a racing video game. The game can be completely external to the agent and can run in real time. In this way, the training system is much more like a real world system. The consoles on which the game runs for training the agent are provided in a cloud computing environment. The agents and the trainers can run on other computing devices in the cloud, where the system can choose the trainers and agent compute based on proximity to console, for example. Users can choose the game they want to run and submit code which can be built and deployed to the cloud system. A resource management service can monitor game console resources between human users and research usage and identify experiments for suspension to ensure enough game consoles for human users.

Abstract: An artificial intelligent agent can act as a player in a video game, such as a racing video game. The game can be completely external to the agent and can run in real time. In this way, the training system is much more like a real world system. The consoles on which the game runs for training the agent are provided in a cloud computing environment. The agents and the trainers can run on other computing devices in the cloud, where the system can choose the trainers and agent compute based on proximity to console, for example. Users can choose the game they want to run and submit code which can be built and deployed to the cloud system. A resource management service can monitor game console resources between human users and research usage and identify experiments for suspension to ensure enough game consoles for human users.

Abstract: An artificial intelligent agent can act as a player in a video game, such as a racing video game. The agent can race against, and often beat, the best players in the world. The game can be completely external to the agent and can run in real time. In this way, the training system is much more like a real world system. The consoles on which the game runs for training the agent are provided in a cloud computing environment. The agents and the trainers can run on other computing devices in the cloud, where the system can choose the trainers and agent compute based on proximity to console, for example. Users can choose the game they want to run and submit code which can be built and deployed to the cloud system. Metrics and logs and artifacts from the game can be sent to cloud storage.

Abstract: Dynamic driving aides, such as driving lines, turn indicators, braking indicators and acceleration indicators, for example, can be provided for players participating in a racing game. Typically, driving lines are provided for each class of cars. However, even within a class of cars, each car differs enough that the ideal driving lines and breaking points can vary. Therefore, with an agent trained via reinforcement learning, an ideal lines and other driving aides can be established for every individual car. These guides can even be varied to account for variations in the weather or other track conditions.

Abstract: A method for training an agent uses a mixture of scenarios designed to teach specific skills helpful in a larger domain, such as mixing general racing and very specific tactical racing scenarios. Aspects of the methods can include one or more of the following: (1) training the agent to be very good at time trials by having one or more cars spread out on the track; (2) running the agent in various racing scenarios with a variable number of opponents starting in different configurations around the track; (3) varying the opponents by using game-provided agents, agents trained according to aspects of the present invention, or agents controlled to follow specific driving lines; (4) setting up specific short scenarios with opponents in various racing situations with specific success criteria; and (5) having a dynamic curriculum based on how the agent performs on a variety of evaluation scenarios.

Abstract: Systems and methods are used to adapt the coefficients of a proportional-integral-derivative (PID) controller through reinforcement learning. The approach for adapting PID coefficients can include an outer loop of reinforcement learning where the PID coefficients are tuned to changes in the environment and an inner loop of PID control for quickly reacting to changing errors. The outer loop can learn and adapt as the environment changes and be configured to only run at a predetermined frequency, after a given number of steps. The outer loop can use summary statistics about the error terms and any other information sensed about the environment to calculate an observation. This observation can be used to evaluate the next action, for example, by feeding it into a neural network representing the policy. The resulting action is the coefficients of the PID controller and the tunable parameters of things such as the filters.

Abstract: A task prioritized experience replay (TaPER) algorithm enables simultaneous learning of multiple RL tasks off policy. The algorithm can prioritize samples that were part of fixed length episodes that led to the achievement of tasks. This enables the agent to quickly learn task policies by bootstrapping over its early successes. Finally, TaPER can improve performance on all tasks simultaneously, which is a desirable characteristic for multi-task RL. Unlike conventional ER algorithms that are applied to single RL task learning settings or that require rewards to be binary or abundant, or are provided as a parameterized specification of goals, TaPER poses no such restrictions and supports arbitrary reward and task specifications.

Abstract: Reinforcement learning methods can use actor-critic networks where (1) additional laboratory-only state information is used to train a policy that much act without this additional laboratory-only information in a production setting; and (2) complex resource-demanding policies are distilled into a less-demanding policy that can be more easily run at production with limited computational resources. The production actor network can be optimized using a frozen version of a large critic network, previously trained with a large actor network. Aspects of these methods can leverage actor-critic methods in which the critic network models the action value function, as opposed to the state value function.

Abstract: A real-time temporal convolution network (RT-TCN) algorithm reuses the output of prior convolution operations in all layers of the network to minimize the computational requirements and memory footprint of a TCN during real-time evaluation. Further, a TCN trained via the fixed-window view, where the TCN is trained using fixed time splices of the input time series, can be executed in real-time continually using RT-TCN.

Abstract: A method and system for training and/or operating an artificial intelligent agent can use multi-input and/or multi-forecast networks. Multi-forecasts are computational constructs, typically, but not necessarily, neural networks, whose shared network weights can be used to compute multiple related forecasts. This allows for more efficient training, in terms of the amount of data and/or experience needed, and in some instances, for more efficient computation of those forecasts. There are several related and sometimes composable approaches to multi-forecast networks.

Abstract: A method and system for teaching an artificial intelligent agent where the agent can be placed in a state that it would like it to learn how to achieve. By giving the agent several examples, it can learn to identify what is important about these example states. Once the agent has the ability to recognize a goal configuration, it can use that information to then learn how to achieve the goal states on its own. An agent may be provided with positive and negative examples to demonstrate a goal configuration. Once the agent has learned certain goal configurations, the agent can learn policies and skills that achieve the learned goal configuration. The agent may create a collection of these policies and skills from which to select based on a particular command or state.

Abstract: Systems and methods use deep, convolutional neural networks over exponentially long history windows to learn alphabets for context tree weighting (CTW) for prediction. Known issues of depth and breadth in conventional context tree weighting predictions are addressed by the systems and methods. To deal with depth, the history can be broken into time windows, permitting the ability to look exponentially far back while having less information the further one looks back. To deal with breadth, a deep neural network classifier can be used to learn to map arbitrary length histories to a small output symbol alphabet. The sequence of symbols produced by such a classifier over the history windows would then become the input sequence to CTW.

Abstract: A method for retrieving inventory items within an inventory system includes receiving a retrieval request that identifies an inventory item and selecting, from a plurality of inventory stations, an inventory station to fulfill an order associated with the retrieval request. The method also includes selecting an inventory holder from a plurality of inventory holders that store the inventory item and selecting, from a plurality of mobile drive units, a mobile drive unit to move the selected inventory holder to the selected inventory station.

Abstract: A method for storing inventory items in an inventory system includes receiving a replenish request identifying an inventory item to be stored in an inventory system and determining a classification of the inventory item to be stored. The method also includes selecting an inventory holder from a plurality of inventory holders based, at least in part, on the classification of the inventory item to be stored and a classification of one or more other inventory items currently stored in the selected inventory holder. Additionally, the method includes transmitting information identifying the selected inventory holder to a mobile drive unit.

Abstract: A method for storing inventory items in an inventory system includes detecting that an inventory holder is available for storage and determining a classification of the inventory holder. The method also includes determining a classification of each of a plurality of storage locations and selecting a storage location for the inventory holder based, at least in part, on the classification of the inventory holder and the classification of the selected storage location. The method additionally includes transmitting information identifying the storage location to a mobile drive unit associated with the inventory holder.