Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for executing a signal generation neural network on parallel processing hardware. One of the methods includes receiving weight matrices of a layer of a signal generation neural network. Rows of a first matrix for the layer are interleaved by assigning groups of rows of the first matrix to respective thread blocks of a plurality of thread blocks. A first subset of rows of the one or more other weight matrices are assigned to a first subset of the plurality of thread blocks and a second subset of rows of the one or more other weight matrices are assigned to a second subset of the plurality of thread blocks. The first matrix operation is performed substantially in parallel by the plurality of thread blocks. The other matrix operations are performed substantially in parallel by the plurality of thread blocks.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating output examples using neural networks. One of the methods includes receiving a request to generate an output example of a particular type, accessing dependency data, and generating the output example by, at each of a plurality of generation time steps: identifying one or more current blocks for the generation time step, wherein each current block is a block for which the values of the bits in all of the other blocks identified in the dependency for the block have already been generated; and generating the values of the bits in the current blocks for the generation time step conditioned on, for each current block, the already generated values of the bits in the other blocks identified in the dependency for the current block.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for interleaving matrix operations of a gated activation unit. One of the methods includes receiving a plurality of weight matrices of a gated activation unit of the neural network, the gated activation unit having two or more layers, each layer defining operations comprising: (i) a matrix operation between a weight matrix for the layer and concatenated input vectors and (ii) a nonlinear activation operation using a result of the matrix operation. Rows of the plurality of weight matrices are interleaved by assigning groups of corresponding rows to respective thread blocks, each thread block being a computation unit for execution by an independent processing unit of a plurality of independent processing units of a parallel processing device.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using neural networks having contiguous sparsity patterns. One of the methods includes storing a first parameter matrix of a neural network having a contiguous sparsity pattern in storage associated with a computing device. The computing device performs an inference pass of the neural network to generate an output vector, including reading, from the storage associated with the computing device, one or more activation values from the input vector, reading, from the storage associated with the computing device, a block of non-zero parameter values, and multiplying each of the one or more activation values by one or more of the block of non-zero parameter values.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network. The neural network has a plurality of differentiable weights and a plurality of non-differentiable weights. One of the methods includes determining trained values of the plurality of differentiable weights and the non-differentiable weights by repeatedly performing operations that include determining an update to the current values of the plurality of differentiable weights using a machine learning gradient-based training technique and determining, using an evolution strategies (ES) technique, an update to the current values of a plurality of distribution parameters.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for distributing a top k computation across multiple computing units of an integrated circuit One of the methods includes computing, by each of the plurality of computing units and for each candidate vector in a respective subset of the candidate vectors assigned to the computing unit, a respective distance between the query vector and the candidate vector; initializing, by the integrated circuit, a cut-off distance value; determining, by the integrated circuit, a final cut-off distance value; and providing, by the integrated circuit and as an output of a top k computation for the query vector and the set of candidate vectors, the candidate vectors that have respective distances that satisfy the final cut-off distance value.

Abstract: A computer-implemented method of training a neural network. The method comprises repeatedly determining a forward-pass set of network parameters by selecting a first sub-set of parameters of the neural network and setting all other parameters of the forward-pass set of network parameters to zero. The method then processes a training data item using the neural network in accordance with the forward-pass set of network parameters to generate a neural network output, determines a value of an objective function from the neural network output and the training data item, selects a second sub-set of network parameters, determines a backward-pass set of network parameters comprising the first and second sub-sets of parameters, and updates parameters corresponding to the backward-pass set of network parameters using a gradient estimate determined from the value of the objective function.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating output examples using neural networks. One of the methods includes receiving a request to generate an output example of a particular type, accessing dependency data, and generating the output example by, at each of a plurality of generation time steps: identifying one or more current blocks for the generation time step, wherein each current block is a block for which the values of the bits in all of the other blocks identified in the dependency for the block have already been generated; and generating the values of the bits in the current blocks for the generation time step conditioned on, for each current block, the already generated values of the bits in the other blocks identified in the dependency for the current block.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for parallelizing matrix operations. One of the methods includes implementing a neural network on a parallel processing device, the neural network comprising at least one sparse neural network layer, the sparse neural network layer being configured to receive an input matrix and perform matrix multiplication between the input matrix and a sparse weight matrix to generate an output matrix, the method comprising: for each row of the M rows of the output matrix, determining a plurality of tiles that each include one or more elements from the row; assigning, for each tile of each row, the tile to a respective one of a plurality of thread blocks of the parallel processing device; and computing, for each tile, respective values for each element in the tile using the respective thread block to which the tile was assigned.

Abstract: A neural network system includes at least one layer which applies a 1×1 convolution to a dense activation matrix, using a kernel defined by a sparse weight matrix. The layer is implemented by a processor with access to a sparsity dataset which indicates where the null weights are located in the weight matrix. The processor selects the feature values corresponding to the other weights from a memory unit configured to store the activation matrix, and then uses these extracted feature values for calculating the convolved values.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating output examples using neural networks. One of the methods includes receiving a request to generate an output example of a particular type, accessing dependency data, and generating the output example by, at each of a plurality of generation time steps: identifying one or more current blocks for the generation time step, wherein each current block is a block for which the values of the bits in all of the other blocks identified in the dependency for the block have already been generated; and generating the values of the bits in the current blocks for the generation time step conditioned on, for each current block, the already generated values of the bits in the other blocks identified in the dependency for the current block.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using a generative neural network to convert conditioning text inputs to audio outputs. The generative neural network includes an alignment neural network that is configured to receive a generative input that includes the conditioning text input and to process the generative input to generate an aligned conditioning sequence that comprises a respective feature representation at each of a plurality of first time steps and that is temporally aligned with the audio output.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for computing a multivalue reduction using a parallel processing device. One of the methods includes performing a parallel M-value reduction by parallel processing units of a parallel processing device. A plurality of initial reductions are performed in serial, each initial reduction operating on data in a different respective register space of at least M register spaces. Data is moved from the M register spaces so that all results from the plurality of initial reductions are in a same first register space. One or more subsequent reductions are performed in parallel to compute M final values, each subsequent reduction operating only on data in the first register space.

Abstract: A computer-implemented method for training a recurrent neural network using forward propagation rather than back propagation through time. The method is particularly suited to training sparse recurrent neural networks, and may be implemented on specialized hardware.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for interleaving matrix operations of a gated activation unit. One of the methods includes receiving a plurality of weight matrices of a gated activation unit of the neural network, the gated activation unit having two or more layers, each layer defining operations comprising: (i) a matrix operation between a weight matrix for the layer and concatenated input vectors and (ii) a nonlinear activation operation using a result of the matrix operation. Rows of the plurality of weight matrices are interleaved by assigning groups of corresponding rows to respective thread blocks, each thread block being a computation unit for execution by an independent processing unit of a plurality of independent processing units of a parallel processing device.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating output audio examples using a generative neural network. One of the methods includes obtaining a training conditioning text input; processing a training generative input comprising the training conditioning text input using a feedforward generative neural network to generate a training audio output; processing the training audio output using each of a plurality of discriminators, wherein the plurality of discriminators comprises one or more conditional discriminators and one or more unconditional discriminators; determining a first combined prediction by combining the respective predictions of the plurality of discriminators; and determining an update to current values of a plurality of generative parameters of the feedforward generative neural network to increase a first error in the first combined prediction.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for executing a signal generation neural network on parallel processing hardware. One of the methods includes receiving weight matrices of a layer of a signal generation neural network. Rows of a first matrix for the layer are interleaved by assigning groups of rows of the first matrix to respective thread blocks of a plurality of thread blocks. A first subset of rows of the one or more other weight matrices are assigned to a first subset of the plurality of thread blocks and a second subset of rows of the one or more other weight matrices are assigned to a second subset of the plurality of thread blocks. The first matrix operation is performed substantially in parallel by the plurality of thread blocks. The other matrix operations are performed substantially in parallel by the plurality of thread blocks.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using neural networks having Receive an input vector contiguous sparsity patterns. One of the methods includes storing a first parameter matrix of a neural network having a contiguous sparsity pattern in storage associated with a computing device. The computing device performs an inference pass of the neural network to generate an output vector, including reading, from the storage associated with the computing device, one or more activation values from the input vector, reading, from the storage associated with the computing device, a block of non-zero parameter values, and multiplying each of the one or more activation values by one or more of the block of non-zero parameter values.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating output examples using neural networks. One of the methods includes, at each generation time step, processing a first recurrent input comprising an N-bit output value at the preceding generation time step in the sequence using a recurrent neural network and in accordance with a hidden state to generate a first score distribution; selecting, using the first score distribution, values for the first half of the N bits; processing a second recurrent input comprising (i) the N-bit output value at the preceding generation time step and (ii) the values for the first half of the N bits using the recurrent neural network and in accordance with the same hidden state to generate a second score distribution; and selecting, using the second score distribution, values for the second half of the N bits of the output value.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for executing a signal generation neural network on parallel processing hardware. One of the methods includes receiving weight matrices of a layer of a signal generation neural network. Rows of a first matrix for the layer are interleaved by assigning groups of rows of the first matrix to respective thread blocks of a plurality of thread blocks. A first subset of rows of the one or more other weight matrices are assigned to a first subset of the plurality of thread blocks and a second subset of rows of the one or more other weight matrices are assigned to a second subset of the plurality of thread blocks. The first matrix operation is performed substantially in parallel by the plurality of thread blocks. The other matrix operations are performed substantially in parallel by the plurality of thread blocks.