Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training an actor neural network used to select actions to be performed by an agent interacting with an environment. One of the methods includes obtaining a minibatch of experience tuples; and updating current values of the parameters of the actor neural network, comprising: for each experience tuple in the minibatch: processing the training observation and the training action in the experience tuple using a critic neural network to determine a neural network output for the experience tuple, and determining a target neural network output for the experience tuple; updating current values of the parameters of the critic neural network using errors between the target neural network outputs and the neural network outputs; and updating the current values of the parameters of the actor neural network using the critic neural network.

Abstract: A recurrent unit is proposed which, at each of a series of time steps receives a corresponding input vector and generates an output at the time step having at least one component for each of a two-dimensional array of pixels. The recurrent unit is configured, at each of the series of time steps except the first, to receive the output of the recurrent unit at the preceding time step, and to apply to the output of the recurrent unit at the preceding time step at least one convolution which depends on the input vector at the time step. The convolution further depends upon the output of the recurrent unit at the preceding time step. This convolution generates a warped dataset which has at least one component for each pixel of the array. The output of the recurrent unit at each time step is based on the warped dataset and the input vector.

Abstract: Systems and methods for image search result filtering can include obtaining a search query, determining a plurality of candidate image search results, processing the search query with a generative model to determine a plurality of search result criteria, and refining the plurality of candidate image search results based on determining whether the candidate results satisfy the plurality of search results criteria. The systems and methods can perform a plurality of determinations based on the output of the generative model.

Abstract: Methods, systems, and apparatus for providing a sequence of actions to perform a task. In one aspect, a method comprises: using a policy neural network to, at each of a sequence of time steps, select one or more actions to be performed according to an action selection policy learned by the policy neural network; providing the selected one or more actions to a simulator; implementing the selected one or more actions for the time steps using the simulator to generate a simulator output; discriminating between the simulator output and training data using a discriminator neural network to produce a discriminator output; and updating parameters of the policy recurrent neural network using a reinforcement learning procedure according to a reward signal determined from the discriminator output; and updating parameters of the discriminator neural network according to a difference between the simulator output and the training data.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for aligning entities across time. One of the methods includes obtaining respective current feature representations for each of a set of current entities that have been detected in an environment at a current time point; obtaining respective historical feature representations for each of a set of historical entities that have been detected in the environment at one or more earlier time points preceding the current time point; and processing an alignment input comprising (i) the respective historical feature representations for the set of historical entities and (ii) the current feature representations for the set of current entities using an alignment neural network to generate an alignment output that defines, for each of one or more of the current entities, a corresponding historical entity that is the same as the current entity.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for controlling agents. In particular, an interactive agent can be controlled based on multi-modal inputs that include both an observation image and a natural language text sequence.

Abstract: Methods and systems for performing a sequence of machine learning tasks. One system includes a sequence of deep neural networks (DNNs), including: a first DNN corresponding to a first machine learning task, wherein the first DNN comprises a first plurality of indexed layers, and each layer in the first plurality of indexed layers is configured to receive a respective layer input and process the layer input to generate a respective layer output; and one or more subsequent DNNs corresponding to one or more respective machine learning tasks, wherein each subsequent DNN comprises a respective plurality of indexed layers, and each layer in a respective plurality of indexed layers with index greater than one receives input from a preceding layer of the respective subsequent DNN, and one or more preceding layers of respective preceding DNNs, wherein a preceding layer is a layer whose index is one less than the current index.

Abstract: A computer-implemented method for controlling a particular computer to execute a task is described. The method includes receiving a control input comprising a visual input, the visual input including one or more screen frames of a computer display that represent at least a current state of the particular computer, processing the control input using a neural network to generate one or more control outputs that are used to control the particular computer to execute the task, in which the one or more control outputs include an action type output that specifies at least one of a pointing device action or a keyboard action to be performed to control the particular computer; determining one or more actions from the one or more control outputs; and executing the one or more actions to control the particular computer.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a speaker neural network using one or more listener neural networks.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing persistent message passing using graph neural networks.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium for encoding input data comprising input data values corresponding to respective input data grid points of an input data grid, such as image, video or audio data.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing protein structure prediction. In one aspect, a method comprises generating a distance map for a given protein, wherein the given protein is defined by a sequence of amino acid residues arranged in a structure, wherein the distance map characterizes estimated distances between the amino acid residues in the structure, comprising: generating a plurality of distance map crops, wherein each distance map crop characterizes estimated distances between (i) amino acid residues in each of one or more respective first positions in the sequence and (ii) amino acid residues in each of one or more respective second positions in the sequence in the structure of the protein, wherein the first positions are a proper subset of the sequence; and generating the distance map for the given protein using the plurality of distance map crops.

Abstract: A system comprising a causal convolutional neural network to autoregressively generate a succession of values of a data item conditioned upon previously generated values of the data item. The system includes support memory for a set of support data patches each of which comprises an encoding of an example data item. A soft attention mechanism attends to one or more patches when generating the current item value. The soft attention mechanism determines a set of scores for the support data patches, for example in the form of a soft attention query vector dependent upon the previously generated values of the data item. The soft attention query vector is used to query the memory. When generating the value of the data item at a current iteration layers of the causal convolutional neural network are conditioned upon the support data patches weighted by the scores.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a neural network by estimating the objective function curvature based on current and previous gradients. In one aspect, a method comprises: sampling a batch of training data; and for each neural network parameter: determining, based on the current batch of training data, a respective current gradient of the objective function at the current iteration with respect to the current neural network parameter; estimating an objective function curvature with respect to the current neural network parameter based on (i) the current gradient of the objective function at the current iteration, and (ii) a respective previous gradient of the objective function at each of a plurality of previous iterations; and updating a current value of the neural network parameter based on the estimate of the curvature of the objective function.

Abstract: Methods, systems, and apparatus for selecting actions to be performed by an agent interacting with an environment. One system includes a high-level controller neural network, low-level controller network, and subsystem. The high-level controller neural network receives an input observation and processes the input observation to generate a high-level output defining a control signal for the low-level controller. The low-level controller neural network receives a designated component of an input observation and processes the designated component and an input control signal to generate a low-level output that defines an action to be performed by the agent in response to the input observation.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining a predicted structure of a protein that is specified by an amino acid sequence. In one aspect, a method comprises: obtaining an initial embedding and initial values of structure parameters for each amino acid in the amino acid sequence, wherein the structure parameters for each amino acid comprise location parameters that specify a predicted three-dimensional spatial location of the amino acid in the structure of the protein; and processing a network input comprising the initial embedding and the initial values of the structure parameters for each amino acid in the amino acid sequence using a folding neural network to generate a network output comprising final values of the structure parameters for each amino acid in the amino acid sequence.

Abstract: A system including an attention neural network that is configured to receive an input sequence and to process the input sequence to generate an output is described. The attention neural network includes: an attention block configured to receive a query input, a key input, and a value input that are derived from an attention block input. The attention block includes an attention neural network layer configured to: receive an attention layer input derived from the query input, the key input, and the value input, and apply an attention mechanism to the query input, the key input, and the value input to generate an attention layer output for the attention neural network layer; and a gating neural network layer configured to apply a gating mechanism to the attention block input and the attention layer output of the attention neural network layer to generate a gated attention output.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a policy neural network for use in controlling a real-world agent in a real-world environment. One of the methods includes training the policy neural network by optimizing a first task-specific objective that measures a performance of the policy neural network in controlling a simulated version of the real-world agent; and then training the policy neural network by jointly optimizing (i) a self-supervised objective that measures at least a performance of internal representations generated by the policy neural network on a self-supervised task performed on real-world data and (ii) a second task-specific objective that measures the performance of the policy neural network in controlling the simulated version of the real-world agent.

Abstract: A neural network control system for controlling an agent to perform a task in a real-world environment, operates based on both image data and proprioceptive data describing the configuration of the agent. The training of the control system includes both imitation learning, using datasets generated from previous performances of the task, and reinforcement learning, based on rewards calculated from control data output by the control system.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for data-efficient reinforcement learning. One of the systems is a system for training an actor neural network used to select actions to be performed by an agent that interacts with an environment by receiving observations characterizing states of the environment and, in response to each observation, performing an action selected from a continuous space of possible actions, wherein the actor neural network maps observations to next actions in accordance with values of parameters of the actor neural network, and wherein the system comprises: a plurality of workers, wherein each worker is configured to operate independently of each other worker, wherein each worker is associated with a respective agent replica that interacts with a respective replica of the environment during the training of the actor neural network.