Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a neural network having one or more conditional computation layers, where each conditional computation layer includes a gating sub-layer having multiple gating parameters and an expert sub-layer having multiple expert neural networks.

Abstract: A recurrent unit is proposed which, at each of a series of time steps receives a corresponding input vector and generates an output at the time step having at least one component for each of a two-dimensional array of pixels. The recurrent unit is configured, at each of the series of time steps except the first, to receive the output of the recurrent unit at the preceding time step, and to apply to the output of the recurrent unit at the preceding time step at least one convolution which depends on the input vector at the time step. The convolution further depends upon the output of the recurrent unit at the preceding time step. This convolution generates a warped dataset which has at least one component for each pixel of the array. The output of the recurrent unit at each time step is based on the warped dataset and the input vector.

Abstract: The present disclosure proposes the use of a dual discriminator network that comprises a temporal discriminator network for discriminating based on temporal features of a series of images and a spatial discriminator network for discriminating based on spatial features of individual images. The training methods described herein provide improvements in computational efficiency. This is achieved by applying the spatial discriminator network to a set of one or more images that have reduced temporal resolution and applying the temporal discriminator network to a set of images that have reduced spatial resolution. This allows each of the discriminator networks to be applied more efficiently in order to produce a discriminator score for use in training the generator, whilst maintaining accuracy of the discriminator network. In addition, this allows a generator network to be trained to more accurately generate sequences of images, through the use of the improved discriminator.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating output audio examples using a generative neural network. One of the methods includes obtaining a training conditioning text input; processing a training generative input comprising the training conditioning text input using a feedforward generative neural network to generate a training audio output; processing the training audio output using each of a plurality of discriminators, wherein the plurality of discriminators comprises one or more conditional discriminators and one or more unconditional discriminators; determining a first combined prediction by combining the respective predictions of the plurality of discriminators; and determining an update to current values of a plurality of generative parameters of the feedforward generative neural network to increase a first error in the first combined prediction.