Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating speech from text. One of the systems includes one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to implement: a sequence-to-sequence recurrent neural network configured to: receive a sequence of characters in a particular natural language, and process the sequence of characters to generate a spectrogram of a verbal utterance of the sequence of characters in the particular natural language; and a subsystem configured to: receive the sequence of characters in the particular natural language, and provide the sequence of characters as input to the sequence-to-sequence recurrent neural network to obtain as output the spectrogram of the verbal utterance of the sequence of characters in the particular natural language.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network using a priority queue. One of the methods includes maintaining data identifying a set of K output sequences that were previously generated; selecting at least one of the output sequences from the set of output sequences; for each selected output sequence, determining a respective score; determining, for each selected sequence, a respective first update to the current values of the controller parameters; generating a batch of new output sequences using the controller neural network; obtaining a respective reward for each of the new output sequences; determining, from the new output sequences and the output sequences in the maintained data, the K output sequences that have the highest rewards; and modifying the maintained data.

Abstract: The present disclosure is directed to systems and methods for performing reading comprehension with machine learning. More specifically, the present disclosure is directed to a Neural Symbolic Reader (example implementations of which may be referred to as NeRd), which includes a reader to encode the passage and question, and a programmer to generate a program for multi-step reasoning. By using operators like span selection, the program can be executed over a natural language text passage to generate an answer to a natural language text question. NeRd is domain-agnostic such that the same neural architecture works for different domains. Further, NeRd is compositional such that complex programs can be generated by compositionally applying the symbolic operators.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing neural architecture search for machine learning models. In one aspect, a method comprises receiving training data for a machine learning, generating a plurality of candidate neural networks for performing the machine learning task, wherein each candidate neural network comprises a plurality of instances of a layer block composed of a plurality of layers, for each candidate neural network, selecting a respective type for each of the plurality of layers from a set of layer types that comprises, training the candidate neural network and evaluating performance scores for the trained candidate neural networks as applied to the machine learning task, and determining a final neural network for performing the machine learning task based at least on the performance scores for the candidate neural networks.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a machine learning model using teacher annealing.

Abstract: Systems and methods are provided that train a machine-learned language encoding model through the use of a contrastive learning task. In particular, the present disclosure describes a contrastive learning task where the encoder learns to distinguish input tokens from plausible alternatives. In some implementations, on each training example the proposed method masks out some subset (e.g., 15%) of the original input tokens, replaces the masked tokens with samples from a “generator” (e.g., which may be a small masked language model), and then trains the encoder to predict whether each token comes from the original data or is a replacement produced by the generator.

Abstract: A method for determining a placement for machine learning model operations across multiple hardware devices is described.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing inputs using recurrent neural networks. One of the systems includes a main recurrent neural network comprising one or more recurrent neural network layers and a respective hyper recurrent neural network corresponding to each of the one or more recurrent neural network layers, wherein each hyper recurrent neural network is configured to, at each of a plurality of time steps: process the layer input at the time step to the corresponding recurrent neural network layer, the current layer hidden state of the corresponding recurrent neural network layer, and a current hypernetwork hidden state of the hyper recurrent neural network to generate an updated hypernetwork hidden state.

Abstract: A method for determining a final architecture for a neural network to perform a particular machine learning task is described.

Abstract: Aspects of the disclosure are directed to jointly searching machine learning model architectures and hardware architectures in a combined space of models, hardware, and mapping strategies. A search strategy is utilized where all models, hardware, and mappings are evaluated together at once via weight sharing and a supernetwork. A multi-objective reward function is utilized with objectives for quality, performance, power, and area.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating author vectors. One of the methods includes obtaining a set of sequences of words, the set of sequences of words comprising a plurality of first sequences of words and, for each first sequence of words, a respective second sequence of words that follows the first sequence of words, wherein each first sequence of words and each second sequence of words has been classified as being authored by a first author; and training a neural network system on the first sequences and the second sequences to determine an author vector for the first author, wherein the author vector characterizes the first author.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a document classification neural network. One of the methods includes training an autoencoder neural network to autoencode input documents, wherein the autoencoder neural network comprises the one or more LSTM neural network layers and an autoencoder output layer, and wherein training the autoencoder neural network comprises determining pre-trained values of the parameters of the one or more LSTM neural network layers from initial values of the parameters of the one or more LSTM neural network layers; and training the document classification neural network on a plurality of training documents to determine trained values of the parameters of the one or more LSTM neural network layers from the pre-trained values of the parameters of the one or more LSTM neural network layers.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating speech from text. One of the systems includes one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to implement: a sequence-to-sequence recurrent neural network configured to: receive a sequence of characters in a particular natural language, and process the sequence of characters to generate a spectrogram of a verbal utterance of the sequence of characters in the particular natural language; and a subsystem configured to: receive the sequence of characters in the particular natural language, and provide the sequence of characters as input to the sequence-to-sequence recurrent neural network to obtain as output the spectrogram of the verbal utterance of the sequence of characters in the particular natural language.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating a computer chip placement. One of the methods includes obtaining netlist data for a computer chip; and generating a computer chip placement, comprising placing a respective macro node at each time step in a sequence comprising a plurality of time steps, the placing comprising, for each time step: generating an input representation for the time step; processing the input representation using a node placement neural network having a plurality of network parameters, wherein the node placement neural network is configured to process the input representation in accordance with current values of the network parameters to generate a score distribution over a plurality of positions on the surface of the computer chip; and assigning the macro node to be placed at the time step to a position from the plurality of positions using the score distribution.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating document vector representations. One of the methods includes obtaining a new document; and determining a vector representation for the new document using a trained neural network system, wherein the trained neural network system has been trained to receive an input document and a sequence of words from the input document and to generate a respective word score for each word in a set of words, wherein each of the respective word scores represents a predicted likelihood that the corresponding word follows a last word in the sequence in the input document, and wherein determining the vector representation for the new document using the trained neural network system comprises iteratively providing each of the plurality of sequences of words to the trained neural network system to determine the vector representation for the new document using gradient descent.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a machine learning model. One of the methods includes obtaining a training data set for training a machine learning model, the training data set comprising a plurality of training inputs; determining a plurality of data augmentation policies, wherein each data augmentation policy defines a procedure for processing a training input to generate a transformed training input; for each data augmentation policy, training the machine learning model using the data augmentation policy; determining, for each data augmentation policy, a quality measure of the machine learning model that has been trained using the data augmentation policy; and selecting a final data augmentation policy based using the quality measures of the machine learning models.

Abstract: Example embodiments of aspects of the present disclosure provide an example computer-implemented method for improved prompting of a machine-learned model. The example method can include obtaining an instructive sequence descriptive of an instructive query, an instructive response, and an instructive trace of intermediate states from the instructive query to the instructive response. The example method can include inputting, to a machine-learned model, the instructive sequence and an operative query, wherein the machine-learned model is configured to process the operative query with attention over the instructive sequence. The example method can include generating, using the machine-learned model and responsive to the operative query, an operative response.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining neural network architectures. One of the methods includes generating, using a controller neural network, a batch of output sequences, each output sequence in the batch defining a respective architecture of a child neural network that is configured to perform a particular neural network task; for each output sequence in the batch: training a respective instance of the child neural network having the architecture defined by the output sequence; evaluating a performance of the trained instance of the child neural network on the particular neural network task to determine a performance metric for the trained instance of the child neural network on the particular neural network task; and using the performance metrics for the trained instances of the child neural network to adjust the current values of the controller parameters of the controller neural network.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining neural network architectures. One of the methods includes generating, using a controller neural network, a batch of output sequences, each output sequence in the batch defining a respective architecture of a child neural network that is configured to perform a particular neural network task; for each output sequence in the batch: training a respective instance of the child neural network having the architecture defined by the output sequence; evaluating a performance of the trained instance of the child neural network on the particular neural network task to determine a performance metric for the trained instance of the child neural network on the particular neural network task; and using the performance metrics for the trained instances of the child neural network to adjust the current values of the controller parameters of the controller neural network.

Abstract: A computer-implemented method for performing computer vision with reduced computational cost and improved accuracy can include obtaining, by a computing system including one or more computing devices, input data comprising an input tensor having one or more dimensions, providing, by the computing system, the input data to a machine-learned convolutional attention network, the machine-learned convolutional attention network including two or more network stages, and, in response to providing the input data to the machine-learned convolutional attention network, receiving, by the computing system, a machine-learning prediction from the machine-learned convolutional attention network. The convolutional attention network can include at least one attention block, wherein the attention block includes a relative attention mechanism, the relative attention mechanism including the sum of a static convolution kernel with an adaptive attention matrix.