Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for rendering a new image that depicts a scene from a perspective of a camera at a new camera location. In one aspect, a method comprises: receiving a plurality of observations characterizing the scene; generating a latent variable representing the scene from the plurality of observations characterizing the scene; conditioning a scene representation neural network on the latent variable representing the scene, wherein the scene representation neural network conditioned on the latent variable representing the scene defines a geometric model of the scene as a three-dimensional (3D) radiance field; and rendering the new image that depicts the scene from the perspective of the camera at the new camera location using the scene representation neural network conditioned on the latent variable representing the scene.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for image rendering. In one aspect, a method comprises receiving a plurality of observations characterizing a particular scene, each observation comprising an image of the particular scene and data identifying a location of a camera that captured the image. In another aspect, the method comprises receiving a plurality of observations characterizing a particular video, each observation comprising a video frame from t31he particular video and data identifying a time stamp of the video frame in the particular video. In yet another aspect, the method comprises receiving a plurality of observations characterizing a particular image, each observation comprising a crop of the particular image and data characterizing the crop of the particular image. The method processes each of the plurality of observations using an observation neural network to determine a numeric representation as output.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for image rendering. In one aspect, a method comprises receiving a plurality of observations characterizing a particular scene, each observation comprising an image of the particular scene and data identifying a location of a camera that captured the image. In another aspect, the method comprises receiving a plurality of observations characterizing a particular video, each observation comprising a video frame from the particular video and data identifying a time stamp of the video frame in the particular video. In yet another aspect, the method comprises receiving a plurality of observations characterizing a particular image, each observation comprising a crop of the particular image and data characterizing the crop of the particular image. The method processes each of the plurality of observations using an observation neural network to determine a numeric representation as output.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using an environment model to simulate state transitions of an environment being interacted with by an agent that is controlled using a policy neural network. One of the methods includes initializing an internal representation of a state of the environment at a current time point; repeatedly performing the following operations: receiving an action to be performed by the agent; generating, based on the internal representation, a predicted latent representation that is a prediction of a latent representation that would have been generated by the policy neural network by processing an observation characterizing the state of the environment corresponding to the internal representation; and updating the internal representation to simulate a state transition caused by the agent performing the received action by processing the predicted latent representation and the received action using the environment model.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a plurality of possible segmentations of an image. In one aspect, a method comprises: receiving a request to generate a plurality of possible segmentations of an image; sampling a plurality of latent variables from a latent space, wherein each latent variable is sampled from the latent space in accordance with a respective probability distribution over the latent space that is determined based on the image; generating a plurality of possible segmentations of the image, comprising, for each latent variable, processing the image and the latent variable using a segmentation neural network having a plurality of segmentation neural network parameters to generate the possible segmentation of the image; and providing the plurality of possible segmentations of the image in response to the request.

Abstract: A neural network system is proposed. The neural network can be trained by model-based reinforcement learning to select actions to be performed by an agent interacting with an environment, to perform a task in an attempt to achieve a specified result. The system may comprise at least one imagination core which receives a current observation characterizing a current state of the environment, and optionally historical observations, and which includes a model of the environment. The imagination core may be configured to output trajectory data in response to the current observation, and/or historical observations. The trajectory data comprising a sequence of future features of the environment imagined by the imagination core. The system may also include a rollout encoder to encode the features, and an output stage to receive data derived from the rollout embedding and to output action policy data for identifying an action based on the current observation.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a machine learning model. One of the methods includes, for each training observation: determining a plurality of latent variable value configurations, each latent variable value configuration being a combination of latent variable values that includes a respective value for each of the latent variables; determining, for each of the plurality of latent variable value configurations, a respective local learning signal that is minimally dependent on each of the other latent variable value configurations in the plurality of latent variable value configurations; determining an unbiased estimate of a gradient of the objective function using the local learning signals; and updating current values of the parameters of the machine learning model using the unbiased estimate of the gradient.

Abstract: According to a first aspect a network system to generate output data values from input data values according to one or In more learned data distributions comprises an input to receive a set of observations, each comprising a respective first data value for a first variable and a respective second data value for a second variable dependent upon the first variable. The system may comprise an encoder neural network system configured to encode each observation of the set of observations to provide an encoded output for each observation. The system may further comprise an aggregator configured to aggregate the encoded outputs for the set of observations and provide an aggregated output. The system may further comprise a decoder neural network system configured to receive a combination of the aggregated output and a target input value and to provide a decoder output. The target input value may comprise a value for the first variable and the decoder output may predict a corresponding value for the second variable.

Abstract: A neural network system is proposed. The neural network can be trained by model-based reinforcement learning to select actions to be performed by an agent interacting with an environment, to perform a task in an attempt to achieve a specified result. The system may comprise at least one imagination core which receives a current observation characterizing a current state of the environment, and optionally historical observations, and which includes a model of the environment. The imagination core may be configured to output trajectory data in response to the current observation, and/or historical observations. The trajectory data comprising a sequence of future features of the environment imagined by the imagination core. The system may also include a rollout encoder to encode the features, and an output stage to receive data derived from the rollout embedding and to output action policy data for identifying an action based on the current observation.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating data items. One of the systems is a neural network system comprising a memory storing a plurality of template data items; one or more processors configured to select a memory address based upon a received input data item, and retrieve a template data item from the memory based upon the selected memory address; an encoder neural network configured to process the received input data item and the retrieved template data item to generate a latent variable representation; and a decoder neural network configured to process the retrieved template data item and the latent variable representation to generate an output data item.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a plurality of possible segmentations of an image. In one aspect, a method comprises: receiving a request to generate a plurality of possible segmentations of an image; sampling a plurality of latent variables from a latent space, wherein each latent variable is sampled from the latent space in accordance with a respective probability distribution over the latent space that is determined based on the image; generating a plurality of possible segmentations of the image, comprising, for each latent variable, processing the image and the latent variable using a segmentation neural network having a plurality of segmentation neural network parameters to generate the possible segmentation of the image; and providing the plurality of possible segmentations of the image in response to the request.

Abstract: A neural network system is proposed. The neural network can be trained by model-based reinforcement learning to select actions to be performed by an agent interacting with an environment, to perform a task in an attempt to achieve a specified result. The system may comprise at least one imagination core which receives a current observation characterizing a current state of the environment, and optionally historical observations, and which includes a model of the environment. The imagination core may be configured to output trajectory data in response to the current observation, and/or historical observations. The trajectory data comprising a sequence of future features of the environment imagined by the imagination core. The system may also include a rollout encoder to encode the features, and an output stage to receive data derived from the rollout embedding and to output action policy data for identifying an action based on the current observation.

Abstract: A system comprising a causal convolutional neural network to autoregressively generate a succession of values of a data item conditioned upon previously generated values of the data item. The system includes support memory for a set of support data patches each of which comprises an encoding of an example data item. A soft attention mechanism attends to one or more patches when generating the current item value. The soft attention mechanism determines a set of scores for the support data patches, for example in the form of a soft attention query vector dependent upon the previously generated values of the data item. The soft attention query vector is used to query the memory. When generating the value of the data item at a current iteration layers of the causal convolutional neural network are conditioned upon the support data patches weighted by the scores.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for a neural network system. In one aspect, a neural network system includes a recurrent neural network that is configured to, for each time step of a predetermined number of time steps, receive a set of latent variables for the time step and process the latent variables to update a hidden state of the recurrent neural network; and a generative subsystem that is configured to, for each time step, generate the set of latent variables for the time step and provide the set of latent variables as input to the recurrent neural network; update a hidden canvas using the updated hidden state of the recurrent neural network; and, for a last time step, generate an output image using the updated hidden canvas for the last time step.

Abstract: A neural network system is proposed. The neural network can be trained by model-based reinforcement learning to select actions to be performed by an agent interacting with an environment, to perform a task in an attempt to achieve a specified result. The system may comprise at least one imagination core which receives a current observation characterizing a current state of the environment, and optionally historical observations, and which includes a model of the environment. The imagination core may be configured to output trajectory data in response to the current observation, and/or historical observations. The trajectory data comprising a sequence of future features of the environment imagined by the imagination core. The system may also include a rollout encoder to encode the features, and an output stage to receive data derived from the rollout embedding and to output action policy data for identifying an action based on the current observation.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating data items. One of the systems is a neural network system comprising a memory storing a plurality of template data items; one or more processors configured to select a memory address based upon a received input data item, and retrieve a template data item from the memory based upon the selected memory address; an encoder neural network configured to process the received input data item and the retrieved template data item to generate a latent variable representation; and a decoder neural network configured to process the retrieved template data item and the latent variable representation to generate an output data item.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for image rendering. In one aspect, a method comprises receiving a plurality of observations characterizing a particular scene, each observation comprising an image of the particular scene and data identifying a location of a camera that captured the image. In another aspect, the method comprises receiving a plurality of observations characterizing a particular video, each observation comprising a video frame from the particular video and data identifying a time stamp of the video frame in the particular video. In yet another aspect, the method comprises receiving a plurality of observations characterizing a particular image, each observation comprising a crop of the particular image and data characterizing the crop of the particular image. The method processes each of the plurality of observations using an observation neural network to determine a numeric representation as output.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for a neural network system. In one aspect, a neural network system includes a recurrent neural network that is configured to, for each time step of a predetermined number of time steps, receive a set of latent variables for the time step and process the latent variables to update a hidden state of the recurrent neural network; and a generative subsystem that is configured to, for each time step, generate the set of latent variables for the time step and provide the set of latent variables as input to the recurrent neural network; update a hidden canvas using the updated hidden state of the recurrent neural network; and, for a last time step, generate an output image using the updated hidden canvas for the last time step.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for a neural network system. In one aspect, a neural network system includes a recurrent neural network that is configured to, for each time step of a predetermined number of time steps, receive a set of latent variables for the time step and process the latent variables to update a hidden state of the recurrent neural network; and a generative subsystem that is configured to, for each time step, generate the set of latent variables for the time step and provide the set of latent variables as input to the recurrent neural network; update a hidden canvas using the updated hidden state of the recurrent neural network; and, for a last time step, generate an output image using the updated hidden canvas for the last time step.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for a neural network system. In one aspect, a neural network system includes a recurrent neural network that is configured to, for each time step of a predetermined number of time steps, receive a set of latent variables for the time step and process the latent variables to update a hidden state of the recurrent neural network; and a generative subsystem that is configured to, for each time step, generate the set of latent variables for the time step and provide the set of latent variables as input to the recurrent neural network; update a hidden canvas using the updated hidden state of the recurrent neural network; and, for a last time step, generate an output image using the updated hidden canvas for the last time step.