Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for machine translation using neural networks. In some implementations, a text in one language is translated into a second language using a neural network model. The model can include an encoder neural network comprising a plurality of bidirectional recurrent neural network layers. The encoding vectors are processed using a multi-headed attention module configured to generate multiple attention context vectors for each encoding vector. A decoder neural network generates a sequence of decoder output vectors using the attention context vectors. The decoder output vectors can represent distributions over various language elements of the second language, allowing a translation of the text into the second language to be determined based on the sequence of decoder output vectors.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for machine translation using neural networks. In some implementations, a text in one language is translated into a second language using a neural network model. The model can include an encoder neural network comprising a plurality of bidirectional recurrent neural network layers. The encoding vectors are processed using a multi-headed attention module configured to generate multiple attention context vectors for each encoding vector. A decoder neural network generates a sequence of decoder output vectors using the attention context vectors. The decoder output vectors can represent distributions over various language elements of the second language, allowing a translation of the text into the second language to be determined based on the sequence of decoder output vectors.

Abstract: The technology provides a model-based approach for multilingual text rewriting that is applicable across many languages and across different styles including formality levels or other textual attributes. The model is configured to manipulate both language and textual attributes jointly. This approach supports zero-shot formality-sensitive translation, with no labeled data in the target language. An encoder-decoder architectural approach with attribute extraction is used to train rewriter models that can thus be used in “universal” textual rewriting across many different languages. A cross-lingual learning signal can be incorporated into the training approach. Certain training processes do not employ any exemplars. This approach enables not just straight translation, but also the ability to create new sentences with different attributes.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing machine learning task on a network input to generate a network output. In one aspect, one of the systems includes an attention neural network configured to perform the machine learning task, the attention neural network including one or more attention layers, each attention layer comprising an attention sub-layer and a feed-forward sub-layer. Some or all of the attention layers have a feed-forward sub-layer that applies conditional computation to the inputs to the sub-layer.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating labeled training data using a pre-trained language model neural network. In particular, the language model neural network can generate the text input in a new labeled training example from an input sequence that includes (i) one or more context inputs and (ii) a text label that identifies the ground truth category for the new labeled training example.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, that translate text depicted in images from a source language into a target language. Methods can include obtaining a first image that depicts first text written in a source language. The first image is input into an image translation model, which includes a feature extractor and a decoder. The feature extractor accepts the first image as input and in response, generates a first set of image features that are a description of a portion of the first image in which the text is depicted is obtained. The first set of image features are input into a decoder. In response to the input first set of image features, the decoder outputs a second text that is a predicted translation of text in the source language that is represented by the first set of image features.

Abstract: Systems and methods for routing in mixture-of-expert models. In some aspects of the technology, a transformer may have at least one Mixture-of-Experts (“MoE”) layer in each of its encoder and decoder, with the at least one MoE layer of the encoder having a learned gating function configured to route each token of a task to two or more selected expert feed-forward networks, and the at least one MoE layer of the decoder having a learned gating function configured to route each task to two or more selected expert feed-forward networks.

Abstract: Training and/or using a multilingual classification neural network model to perform a natural language processing classification task, where the model reuses an encoder portion of a multilingual neural machine translation model. In a variety of implementations, a client device can generate a natural language data stream from a spoken input from a user. The natural language data stream can be applied as input to an encoder portion of the multilingual classification model. The output generated by the encoder portion can be applied as input to a classifier portion of the multilingual classification model. The classifier portion can generate a predicted classification label of the natural language data stream. In many implementations, an output can be generated based on the predicted classification label, and a client device can present the output.

Abstract: Adapters for neural machine translation systems. A method includes determining a set of similar n-grams that are similar to a source n-gram, and each similar n-gram and the source n-gram is in a first language; determining, for each n-gram in the set of similar n-grams, a target n-gram is a translation of the similar n-gram in the first language to the target n-gram in the second language; generating a source encoding of the source n-gram, and, for each target n-gram determined from the set of similar n-grams determined for the source n-gram, a target encoding of the target n-gram and a conditional source target memory that is an encoding of each of the target encodings; providing, as input to a first prediction model, the source encoding and the condition source target memory; and generating a predicted translation of the source n-gram from the first language to the second language.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for machine translation using neural networks. In some implementations, a text in one language is translated into a second language using a neural network model. The model can include an encoder neural network comprising a plurality of bidirectional recurrent neural network layers. The encoding vectors are processed using a multi-headed attention module configured to generate multiple attention context vectors for each encoding vector. A decoder neural network generates a sequence of decoder output vectors using the attention context vectors. The decoder output vectors can represent distributions over various language elements of the second language, allowing a translation of the text into the second language to be determined based on the sequence of decoder output vectors.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for machine translation using neural networks. In some implementations, a text in one language is translated into a second language using a neural network model. The model can include an encoder neural network comprising a plurality of bidirectional recurrent neural network layers. The encoding vectors are processed using a multi-headed attention module configured to generate multiple attention context vectors for each encoding vector. A decoder neural network generates a sequence of decoder output vectors using the attention context vectors. The decoder output vectors can represent distributions over various language elements of the second language, allowing a translation of the text into the second language to be determined based on the sequence of decoder output vectors.

Abstract: Adapters for neural machine translation systems. A method includes determining a set of similar n-grams that are similar to a source n-gram, and each similar n-gram and the source n-gram is in a first language; determining, for each n-gram in the set of similar n-grams, a target n-gram is a translation of the similar n-gram in the first language to the target n-gram in the second language; generating a source encoding of the source n-gram, and, for each target n-gram determined from the set of similar n-grams determined for the source n-gram, a target encoding of the target n-gram and a conditional source target memory that is an encoding of each of the target encodings; providing, as input to a first prediction model, the source encoding and the condition source target memory; and generating a predicted translation of the source n-gram from the first language to the second language.

Abstract: Training and/or using a multilingual classification neural network model to perform a natural language processing classification task, where the model reuses an encoder portion of a multilingual neural machine translation model. In a variety of implementations, a client device can generate a natural language data stream from a spoken input from a user. The natural language data stream can be applied as input to an encoder portion of the multilingual classification model. The output generated by the encoder portion can be applied as input to a classifier portion of the multilingual classification model. The classifier portion can generate a predicted classification label of the natural language data stream. In many implementations, an output can be generated based on the predicted classification label, and a client device can present the output.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for machine translation using neural networks. In some implementations, a text in one language is translated into a second language using a neural network model. The model can include an encoder neural network comprising a plurality of bidirectional recurrent neural network layers. The encoding vectors are processed using a multi-headed attention module configured to generate multiple attention context vectors for each encoding vector. A decoder neural network generates a sequence of decoder output vectors using the attention context vectors. The decoder output vectors can represent distributions over various language elements of the second language, allowing a translation of the text into the second language to be determined based on the sequence of decoder output vectors.