Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing reinforcement learning with centralized inference and training One of the methods includes receiving, at a current time-step in a plurality of time-steps, a respective observation by an actor for each environment of a plurality of environments; obtaining, for each environment, a respective reward for the actor as a result of the actor performing a respective action at a previous time-step preceding the current time-step; processing, for each environment, the respective observation and respective reward through a policy model; providing, to the actor, the respective policy outputs for each of the plurality of environments; maintaining at a repository and for each environment, a respective sequence of tuples corresponding to the actor; determining that a maintained sequence meets a threshold condition; and in response, training the policy model on the maintained sequence.