Patents by Inventor Lukasz Mieczyslaw Kaiser
Lukasz Mieczyslaw Kaiser has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).
-
Publication number: 20240144006Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. In one aspect, one of the systems includes an encoder neural network configured to receive the input sequence and generate encoded representations of the network inputs, the encoder neural network comprising a sequence of one or more encoder subnetworks, each encoder subnetwork configured to receive a respective encoder subnetwork input for each of the input positions and to generate a respective subnetwork output for each of the input positions, and each encoder subnetwork comprising: an encoder self-attention sub-layer that is configured to receive the subnetwork input for each of the input positions and, for each particular input position in the input order: apply an attention mechanism over the encoder subnetwork inputs using one or more queries derived from the encoder subnetwork input at the particular input position.Type: ApplicationFiled: January 8, 2024Publication date: May 2, 2024Inventors: Noam M. Shazeer, Aidan Nicholas Gomez, Lukasz Mieczyslaw Kaiser, Jakob D. Uszkoreit, Llion Owen Jones, Niki J. Parmar, Illia Polosukhin, Ashish Teku Vaswani
-
Publication number: 20240143691Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for implementing a sequence-to-sequence model that is recurrent in depth while employing self-attention to combine information from different parts of sequences.Type: ApplicationFiled: December 18, 2023Publication date: May 2, 2024Inventors: Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob D. Uszkoreit, Lukasz Mieczyslaw Kaiser
-
Patent number: 11893483Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. In one aspect, one of the systems includes an encoder neural network configured to receive the input sequence and generate encoded representations of the network inputs, the encoder neural network comprising a sequence of one or more encoder subnetworks, each encoder subnetwork configured to receive a respective encoder subnetwork input for each of the input positions and to generate a respective subnetwork output for each of the input positions, and each encoder subnetwork comprising: an encoder self-attention sub-layer that is configured to receive the subnetwork input for each of the input positions and, for each particular input position in the input order: apply an attention mechanism over the encoder subnetwork inputs using one or more queries derived from the encoder subnetwork input at the particular input position.Type: GrantFiled: August 7, 2020Date of Patent: February 6, 2024Assignee: Google LLCInventors: Noam M. Shazeer, Aidan Nicholas Gomez, Lukasz Mieczyslaw Kaiser, Jakob D. Uszkoreit, Llion Owen Jones, Niki J. Parmar, Illia Polosukhin, Ashish Teku Vaswani
-
Patent number: 11886998Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. One of the methods includes, at each of a plurality of generation time steps: generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence.Type: GrantFiled: January 13, 2023Date of Patent: January 30, 2024Assignee: Google LLCInventors: Noam M. Shazeer, Lukasz Mieczyslaw Kaiser, Etienne Pot, Mohammad Saleh, Ben Goodrich, Peter J. Liu, Ryan Sepassi
-
Patent number: 11860969Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for implementing a sequence to sequence model that is recurrent in depth while employing self-attention to combine information from different parts of sequences.Type: GrantFiled: August 10, 2020Date of Patent: January 2, 2024Assignee: Google LLCInventors: Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob D. Uszkoreit, Lukasz Mieczyslaw Kaiser
-
Patent number: 11816884Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output image. In one aspect, one of the methods includes generating the output image intensity value by intensity value according to a generation order of pixel—color channel pairs from the output image, comprising, for each particular generation order position in the generation order: generating a current output image representation of a current output image, processing the current output image representation using a decoder neural network to generate a probability distribution over possible intensity values for the pixel—color channel pair at the particular generation order position, wherein the decoder neural network includes one or more local masked self-attention sub-layers; and selecting an intensity value for the pixel—color channel pair at the particular generation order position using the probability distribution.Type: GrantFiled: July 18, 2022Date of Patent: November 14, 2023Assignee: Google LLCInventors: Noam M. Shazeer, Lukasz Mieczyslaw Kaiser, Jakob D. Uszkoreit, Niki J. Parmar, Ashish Teku Vaswani
-
Patent number: 11803711Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media for performing machine translation tasks. One method includes receiving an input text segment in an input language; processing the input text segment using an encoder neural network to generate an encoder neural network output, the encoder neural network comprising multiple depth wise separable convolutional neural network layers; processing the encoder neural network output using an autoregressive decoder neural network to generate a decoder neural network output; and processing the decoder neural network output to generate a predicted output text segment in a target natural language.Type: GrantFiled: November 20, 2020Date of Patent: October 31, 2023Assignee: Google LLCInventors: Aidan Nicholas Gomez, Lukasz Mieczyslaw Kaiser, Francois Chollet
-
Publication number: 20230153613Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. One of the methods includes, at each of a plurality of generation time steps: generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence.Type: ApplicationFiled: January 13, 2023Publication date: May 18, 2023Inventors: Noam M. Shazeer, Lukasz Mieczyslaw Kaiser, Etienne Pot, Mohammad Saleh, Ben Goodrich, Peter J. Liu, Ryan Sepassi
-
Publication number: 20230076971Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output image. In one aspect, one of the methods includes generating the output image intensity value by intensity value according to a generation order of pixel—color channel pairs from the output image, comprising, for each particular generation order position in the generation order: generating a current output image representation of a current output image, processing the current output image representation using a decoder neural network to generate a probability distribution over possible intensity values for the pixel—color channel pair at the particular generation order position, wherein the decoder neural network includes one or more local masked self-attention sub-layers; and selecting an intensity value for the pixel—color channel pair at the particular generation order position using the probability distribution.Type: ApplicationFiled: July 18, 2022Publication date: March 9, 2023Inventors: Noam M. Shazeer, Lukasz Mieczyslaw Kaiser, Jakob D. Uszkoreit, Niki J. Parmar, Ashish Teku Vaswani
-
Patent number: 11556786Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. One of the methods includes, at each of a plurality of generation time steps: generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence.Type: GrantFiled: October 29, 2018Date of Patent: January 17, 2023Assignee: Google LLCInventors: Noam M. Shazeer, Lukasz Mieczyslaw Kaiser, Etienne Pot, Mohammad Saleh, Ben David Goodrich, Peter J. Liu, Ryan Sepassi
-
Patent number: 11494561Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media for training a machine learning model to perform multiple machine learning tasks from multiple machine learning domains. One system includes a machine learning model that includes multiple input modality neural networks corresponding to respective different modalities and being configured to map received data inputs of the corresponding modality to mapped data inputs from a unified representation space; an encoder neural network configured to process mapped data inputs from the unified representation space to generate respective encoder data outputs; a decoder neural network configured to process encoder data outputs to generate respective decoder data outputs from the unified representation space; and multiple output modality neural networks corresponding to respective different modalities and being configured to map decoder data outputs to data outputs of the corresponding modality.Type: GrantFiled: August 4, 2020Date of Patent: November 8, 2022Assignee: Google LLCInventors: Noam M. Shazeer, Aidan Nicholas Gomez, Lukasz Mieczyslaw Kaiser, Jakob D. Uszkoreit, Llion Owen Jones, Niki J. Parmar, Ashish Teku Vaswani
-
Publication number: 20220253672Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input to generate a network output. In one aspect, one of the systems includes a neural network configured to perform the machine learning task, the neural network including one or more sparse attention layers.Type: ApplicationFiled: February 7, 2022Publication date: August 11, 2022Inventors: Aakanksha Chowdhery, Afroz Mohiuddin, Henryk Michalewski, Jonni Miikka Kanerva, Lukasz Mieczyslaw Kaiser, Sebastian Dariusz Jaszczur, Wojciech Gajewski
-
Patent number: 11392790Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output image. In one aspect, one of the methods includes generating the output image intensity value by intensity value according to a generation order of pixel—color channel pairs from the output image, comprising, for each particular generation order position in the generation order: generating a current output image representation of a current output image, processing the current output image representation using a decoder neural network to generate a probability distribution over possible intensity values for the pixel—color channel pair at the particular generation order position, wherein the decoder neural network includes one or more local masked self-attention sub-layers; and selecting an intensity value for the pixel—color channel pair at the particular generation order position using the probability distribution.Type: GrantFiled: November 13, 2020Date of Patent: July 19, 2022Assignee: Google LLCInventors: Noam M. Shazeer, Lukasz Mieczyslaw Kaiser, Jakob D. Uszkoreit, Niki J. Parmar, Ashish Teku Vaswani
-
Publication number: 20220051099Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. In one aspect, one of the systems includes an encoder neural network configured to receive the input sequence and generate encoded representations of the network inputs, the encoder neural network comprising a sequence of one or more encoder subnetworks, each encoder subnetwork configured to receive a respective encoder subnetwork input for each of the input positions and to generate a respective subnetwork output for each of the input positions, and each encoder subnetwork comprising: an encoder self-attention sub-layer that is configured to receive the subnetwork input for each of the input positions and, for each particular input position in the input order: apply an attention mechanism over the encoder subnetwork inputs using one or more queries derived from the encoder subnetwork input at the particular input position.Type: ApplicationFiled: September 3, 2021Publication date: February 17, 2022Inventors: Noam M. Shazeer, Aidan Nicholas Gomez, Lukasz Mieczyslaw Kaiser, Jakob D. Uszkoreit, Llion Owen Jones, Niki J. Parmar, Illia Polosukhin, Ashish Teku Vaswani
-
Publication number: 20210350244Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input to generate a network output. In one aspect, one of the systems includes an attention neural network configured to perform the machine learning task, the attention neural network including one or more LSH attention layers, each LSH attention layer comprising one or more LSH attention sub-layers, each LSH sub-layer configured to: receive a sequence of queries derived from an input sequence to the LSH attention layer, the sequence of queries having a respective query at each of a plurality of input positions; determine one or more respective hash values for each of the respective queries at each of the plurality of input positions; generate a plurality of LSH groupings; and generate an attended input sequence.Type: ApplicationFiled: February 1, 2021Publication date: November 11, 2021Inventors: Nikita Kitaev, Lukasz Mieczyslaw Kaiser, Anselm Caelifer Levskaya
-
Patent number: 11113602Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. In one aspect, one of the systems includes an encoder neural network configured to receive the input sequence and generate encoded representations of the network inputs, the encoder neural network comprising a sequence of one or more encoder subnetworks, each encoder subnetwork configured to receive a respective encoder subnetwork input for each of the input positions and to generate a respective subnetwork output for each of the input positions, and each encoder subnetwork comprising: an encoder self-attention sub-layer that is configured to receive the subnetwork input for each of the input positions and, for each particular input position in the input order: apply an attention mechanism over the encoder subnetwork inputs using one or more queries derived from the encoder subnetwork input at the particular input position.Type: GrantFiled: July 17, 2020Date of Patent: September 7, 2021Assignee: Google LLCInventors: Noam M. Shazeer, Aidan Nicholas Gomez, Lukasz Mieczyslaw Kaiser, Jakob D. Uszkoreit, Llion Owen Jones, Niki J. Parmar, Illia Polosukhin, Ashish Teku Vaswani
-
Patent number: 11093813Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for identifying answers to questions using neural networks. One of the methods includes receiving an input text passage and an input question string; processing the input text passage using an encoder neural network to generate a respective encoded representation for each passage token in the input text passage; at each time step: processing a decoder input using a decoder neural network to update the internal state of the decoder neural network; and processing the respective encoded representations and a preceding output of the decoder neural network using a matching vector neural network to generate a matching vector for the time step; and generating an answer score that indicates how well the input text passage answers a question posed by the input question string.Type: GrantFiled: October 18, 2017Date of Patent: August 17, 2021Assignee: GOOGLE LLCInventors: Ni Lao, Lukasz Mieczyslaw Kaiser, Nitin Gupta, Afroz Mohiuddin, Preyas Popat
-
Patent number: 10977547Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for implementing a convolutional gated recurrent neural network (CGRN). In one of the systems, the CGRN is configured to maintain a state that is a tensor having dimensions x by y by m, wherein x, y, and m are each greater than one, and for each of a plurality of time steps, update a currently maintained state by processing the currently maintained state through a plurality of convolutional gates.Type: GrantFiled: November 11, 2016Date of Patent: April 13, 2021Assignee: Google LLCInventors: Lukasz Mieczyslaw Kaiser, Ilya Sutskever
-
Patent number: 10956819Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. In one aspect, one of the systems includes an encoder neural network configured to receive the input sequence and generate encoded representations of the network inputs, the encoder neural network comprising a sequence of one or more encoder subnetworks, each encoder subnetwork configured to receive a respective encoder subnetwork input for each of the input positions and to generate a respective subnetwork output for each of the input positions, and each encoder subnetwork comprising: an encoder self-attention sub-layer that is configured to receive the subnetwork input for each of the input positions and, for each particular input position in the input order: apply an attention mechanism over the encoder subnetwork inputs using one or more queries derived from the encoder subnetwork input at the particular input position.Type: GrantFiled: August 7, 2020Date of Patent: March 23, 2021Assignee: Google LLCInventors: Noam M. Shazeer, Aidan Nicholas Gomez, Lukasz Mieczyslaw Kaiser, Jakob D. Uszkoreit, Llion Owen Jones, Niki J. Parmar, Illia Polosukhin, Ashish Teku Vaswani
-
Publication number: 20210073481Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media for performing machine translation tasks. One method includes receiving an input text segment in an input language; processing the input text segment using an encoder neural network to generate an encoder neural network output, the encoder neural network comprising multiple depth wise separable convolutional neural network layers; processing the encoder neural network output using an autoregressive decoder neural network to generate a decoder neural network output; and processing the decoder neural network output to generate a predicted output text segment in a target natural language.Type: ApplicationFiled: November 20, 2020Publication date: March 11, 2021Inventors: Aidan Nicholas Gomez, Lukasz Mieczyslaw Kaiser, Francois Chollet