Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for translating terms using numeric representations. One of the methods includes obtaining data that associates each term in a vocabulary of terms in a first language with a respective high-dimensional representation of the term; obtaining data that associates each term in a vocabulary of terms in a second language with a respective high-dimensional representation of the term; receiving a first language term; and determining a translation into the second language of the first language term from the high-dimensional representation of the first language term and the high-dimensional representations of terms in the vocabulary of terms in the second language.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting likelihoods of conditions being satisfied using recurrent neural networks. One of the systems is configured to process a temporal sequence comprising a respective input at each of a plurality of time steps and comprises: one or more recurrent neural network layers; one or more logistic regression nodes, wherein each of the logistic regression nodes corresponds to a respective condition from a predetermined set of conditions, and wherein each of the logistic regression nodes is configured to, for each of the plurality of time steps: receive the network internal state for the time step; and process the network internal state for the time step in accordance with current values of a set of parameters of the logistic regression node to generate a future condition score for the corresponding condition for the time step.

Abstract: A system can be configured to perform tasks such as converting recorded speech to a sequence of phonemes that represent the speech, converting an input sequence of graphemes into a target sequence of phonemes, translating an input sequence of words in one language into a corresponding sequence of words in another language, or predicting a target sequence of words that follow an input sequence of words in a language (e.g., a language model). In a speech recognizer, the RNN system may be used to convert speech to a target sequence of phonemes in real-time so that a transcription of the speech can be generated and presented to a user, even before the user has completed uttering the entire speech input.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing operations using data from a data source. In one aspect, a method includes a neural network system including a controller neural network configured to: receive a controller input for a time step and process the controller input and a representation of a system input to generate: an operation score distribution that assigns a respective operation score to an operation and a data score distribution that assigns a respective data score in the data source. The neural network system can also include an operation subsystem configured to: perform operations to generate operation outputs, wherein at least one of the operations is performed on data in the data source, and combine the operation outputs in accordance with the operation score distribution and the data score distribution to generate a time step output for the time step.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for implementing a convolutional gated recurrent neural network (CGRN). In one of the systems, the CGRN is configured to maintain a state that is a tensor having dimensions x by y by m, wherein x, y, and m are each greater than one, and for each of a plurality of time steps, update a currently maintained state by processing the currently maintained state through a plurality of convolutional gates.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a system output from a system input. In one aspect, a neural network system includes a memory storing a set of register vectors and data defining modules, wherein each module is a respective function that takes as input one or more first vectors and outputs a second vector. The system also includes a controller neural network configured to receive a neural network input for each time step and process the neural network input to generate a neural network output. The system further includes a subsystem configured to determine inputs to each of the modules, process the input to the module to generate a respective module output, determine updated values for the register vectors, and generate a neural network input for the next time step from the updated values of the register vectors.

Abstract: A parallel convolutional neural network is provided. The CNN is implemented by a plurality of convolutional neural networks each on a respective processing node. Each CNN has a plurality of layers. A subset of the layers are interconnected between processing nodes such that activations are fed forward across nodes. The remaining subset is not so interconnected.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting likelihoods of conditions being satisfied using recurrent neural networks. One of the systems is configured to process a temporal sequence comprising a respective input at each of a plurality of time steps and comprises: one or more recurrent neural network layers; one or more logistic regression nodes, wherein each of the logistic regression nodes corresponds to a respective condition from a predetermined set of conditions, and wherein each of the logistic regression nodes is configured to, for each of the plurality of time steps: receive the network internal state for the time step; and process the network internal state for the time step in accordance with current values of a set of parameters of the logistic regression node to generate a future condition score for the corresponding condition for the time step.

Abstract: A parallel convolutional neural network is provided. The CNN is implemented by a plurality of convolutional neural networks each on a respective processing node. Each CNN has a plurality of layers. A subset of the layers are interconnected between processing nodes such that activations are fed forward across nodes. The remaining subset is not so interconnected.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting likelihoods of conditions being satisfied using recurrent neural networks. One of the systems is configured to process a temporal sequence comprising a respective input at each of a plurality of time steps and comprises: one or more recurrent neural network layers; one or more logistic regression nodes, wherein each of the logistic regression nodes corresponds to a respective condition from a predetermined set of conditions, and wherein each of the logistic regression nodes is configured to, for each of the plurality of time steps: receive the network internal state for the time step; and process the network internal state for the time step in accordance with current values of a set of parameters of the logistic regression node to generate a future condition score for the corresponding condition for the time step.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using recurrent neural networks to analyze health events. One of the methods includes obtaining a first temporal sequence of health events, wherein the first temporal sequence comprises respective health-related data associated with a particular patient at each of a plurality of time steps; processing the first temporal sequence of health events using a recurrent neural network to generate a neural network output for the first temporal sequence; and generating, from the neural network output for the first temporal sequence, health analysis data that characterizes future health events that may occur after a last time step in the temporal sequence.

Abstract: A system for training a neural network. A switch is linked to feature detectors in at least some of the layers of the neural network. For each training case, the switch randomly selectively disables each of the feature detectors in accordance with a preconfigured probability. The weights from each training case are then normalized for applying the neural network to test data.

Abstract: A system for training a neural network. A switch is linked to feature detectors in at least some of the layers of the neural network. For each training case, the switch randomly selectively disables each of the feature detectors in accordance with a preconfigured probability. The weights from each training case are then normalized for applying the neural network to test data.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting likelihoods of conditions being satisfied using recurrent neural networks. One of the systems is configured to process a temporal sequence comprising a respective input at each of a plurality of time steps and comprises: one or more recurrent neural network layers; one or more logistic regression nodes, wherein each of the logistic regression nodes corresponds to a respective condition from a predetermined set of conditions, and wherein each of the logistic regression nodes is configured to, for each of the plurality of time steps: receive the network internal state for the time step; and process the network internal state for the time step in accordance with current values of a set of parameters of the logistic regression node to generate a future condition score for the corresponding condition for the time step.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for neural translation systems with rare word processing. One of the methods is a method training a neural network translation system to track the source in source sentences of unknown words in target sentences, in a source language and a target language, respectively and includes deriving alignment data from a parallel corpus, the alignment data identifying, in each pair of source and target language sentences in the parallel corpus, aligned source and target words; annotating the sentences in the parallel corpus according to the alignment data and a rare word model to generate a training dataset of paired source and target language sentences; and training a neural network translation model on the training dataset.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network. One of the methods includes obtaining partitioned training data for the neural network, wherein the partitioned training data comprises a plurality of training items each of which is assigned to a respective one of a plurality of partitions, wherein each partition is associated with a respective difficulty level; and training the neural network on each of the partitions in a sequence from a partition associated with an easiest difficulty level to a partition associated with a hardest difficulty level, wherein, for each of the partitions, training the neural network comprises: training the neural network on a sequence of training items that includes training items selected from the training items in the partition interspersed with training items selected from the training items in all of the partitions.

Abstract: A system and method for generating training images. An existing training image is associated with a classification. The system includes an image processing module that performs color-space deformation on each pixel of the existing training image and then associates the classification to the color-space deformed training image. The technique may be applied to increase the size of a training set for training a neural network.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating representations of input sequences. One of the methods includes obtaining an input sequence, the input sequence comprising a plurality of inputs arranged according to an input order; processing the input sequence using a first long short term memory (LSTM) neural network to convert the input sequence into an alternative representation for the input sequence; and processing the alternative representation for the input sequence using a second LSTM neural network to generate a target sequence for the input sequence, the target sequence comprising a plurality of outputs arranged according to an output order.

Abstract: A parallel convolutional neural network is provided. The CNN is implemented by a plurality of convolutional neural networks each on a respective processing node. Each CNN has a plurality of layers. A subset of the layers are interconnected between processing nodes such that activations are fed forward across nodes. The remaining subset is not so interconnected.

Abstract: A system for training a neural network. A switch is linked to feature detectors in at least some of the layers of the neural network. For each training case, the switch randomly selectively disables each of the feature detectors in accordance with a preconfigured probability. The weights from each training case are then normalized for applying the neural network to test data.