Abstract: There are provided systems, methods, and apparatus, for optimizing a policy controller to control a robotic agent that interacts with an environment to perform a robotic task. One of the methods includes optimizing the policy controller using a neural network that generates numeric embeddings of images of the environment and a demonstration sequence of demonstration images of another agent performing a version of the robotic task.

Abstract: There are provided systems, methods, and apparatus, for optimizing a policy controller to control a robotic agent that interacts with an environment to perform a robotic task. One of the methods includes optimizing the policy controller using a neural network that generates numeric embeddings of images of the environment and a demonstration sequence of demonstration images of another agent performing a version of the robotic task.

Abstract: Utilizing an initial set of offline positive-only robotic demonstration data for pre-training an actor network and a critic network for robotic control, followed by further training of the networks based on online robotic episodes that utilize the network(s). Implementations enable the actor network to be effectively pre-trained, while mitigating occurrences of and/or the extent of forgetting when further trained based on episode data. Implementations additionally or alternatively enable the actor network to be trained to a given degree of effectiveness in fewer training steps. In various implementations, one or more adaptation techniques are utilized in performing the robotic episodes and/or in performing the robotic training. The adaptation techniques can each, individually, result in one or more corresponding advantages and, when used in any combination, the corresponding advantages can accumulate.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, of training a global policy neural network. One of the methods includes initializing an instance of the robotic task for multiple local workers, generating a trajectory of state-action pairs by selecting actions to be performed by the robotic agent while performing the instance of the robotic task, optimizing a local policy controller on the trajectory, generating an optimized trajectory using the optimized local controller, and storing the optimized trajectory in a replay memory associated with the local worker. The method includes sampling, for multiple global workers, an optimized trajectory from one of one or more replay memories associated with the global worker, and training the replica of the global policy neural network maintained by the global worker on the sampled optimized trajectory to determine delta values for the parameters of the global policy neural network.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, of training a global policy neural network. One of the methods includes initializing a plurality of instances of the robotic task. For each instance of the robotic task, the method includes generating a trajectory of state-action pairs by selecting actions to be performed by the robotic agent while performing the instance of the robotic task in accordance with current values of the parameters of the global policy neural network, and optimizing a local policy controller that is specific to the instance on the trajectory of state-action pairs for the instance. The method further includes generating training data for the global policy neural network using the local policy controllers, and training the global policy neural network on the training data to adjust the current values of the parameters of the global policy neural network.

Abstract: A machine-learning control system is trained to perform a task using a simulation. The simulation is governed by parameters that, in various embodiments, are not precisely known. In an embodiment, the parameters are specified with an initial value and expected range. After training on the simulation, the machine-learning control system attempts to perform the task in the real world. In an embodiment, the results of the attempt are compared to the expected results of the simulation, and the parameters that govern the simulation are adjusted so that the simulated result matches the real-world attempt. In an embodiment, the machine-learning control system is retrained on the updated simulation. In an embodiment, as additional real-world attempts are made, the simulation parameters are refined and the control system is retrained until the simulation is accurate and the control system is able to successfully perform the task in the real world.

Abstract: There are provided systems, methods, and apparatus, for optimizing a policy controller to control a robotic agent that interacts with an environment to perform a robotic task. One of the methods includes optimizing the policy controller using a neural network that generates numeric embeddings of images of the environment and a demonstration sequence of demonstration images of another agent performing a version of the robotic task.