Patents by Inventor Mohammad Norouzi

Mohammad Norouzi has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Publication number: 20230153959
    Abstract: A method includes receiving, by a computing device, training data comprising a plurality of pairs of images, wherein each pair comprises an image and at least one corresponding target version of the image. The method also includes training a neural network based on the training data to predict an enhanced version of an input image, wherein the training of the neural network comprises applying a forward Gaussian diffusion process that adds Gaussian noise to the at least one corresponding target version of each of the plurality of pairs of images to enable iterative denoising of the input image, wherein the iterative denoising is based on a reverse Markov chain associated with the forward Gaussian diffusion process. The method additionally includes outputting the trained neural network.
    Type: Application
    Filed: January 17, 2023
    Publication date: May 18, 2023
    Inventors: Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, Mohammad Norouzi
  • Publication number: 20230103638
    Abstract: A method includes receiving training data comprising a plurality of pairs of images. Each pair comprises a noisy image and a denoised version of the noisy image. The method also includes training a multi-task diffusion model to perform a plurality of image-to-image translation tasks, wherein the training comprises iteratively generating a forward diffusion process by predicting, at each iteration in a sequence of iterations and based on a current noisy estimate of the denoised version of the noisy image, noise data for a next noisy estimate of the denoised version of the noisy image, updating, at each iteration, the current noisy estimate to the next noisy estimate by combining the current noisy estimate with the predicted noise data, and determining a reverse diffusion process by inverting the forward diffusion process to predict the denoised version of the noisy image. The method additionally includes providing the trained diffusion model.
    Type: Application
    Filed: October 5, 2022
    Publication date: April 6, 2023
    Inventors: Chitwan Saharia, Mohammad Norouzi, William Chan, Huiwen Chang, David James Fleet, Christopher Albert Lee, Jonathan Ho, Tim Salimans
  • Publication number: 20230075716
    Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for sequence modeling. One of the methods includes receiving an input sequence having a plurality of input positions; determining a plurality of blocks of consecutive input positions; processing the input sequence using a neural network to generate a latent alignment, comprising, at each of a plurality of input time steps: receiving a partial latent alignment from a previous input time step; selecting an input position in each block, wherein the token at the selected input position of the partial latent alignment in each block is a mask token; and processing the partial latent alignment and the input sequence using the neural network to generate a new latent alignment, wherein the new latent alignment comprises, at the selected input position in each block, an output token or a blank token; and generating, using the latent alignment, an output sequence.
    Type: Application
    Filed: February 8, 2021
    Publication date: March 9, 2023
    Inventors: William Chan, Chitwan Saharia, Geoffrey E. Hinton, Mohammad Norouzi, Navdeep Jaitly
  • Publication number: 20230067841
    Abstract: A method includes receiving, by a computing device, training data comprising a plurality of pairs of images, wherein each pair comprises an image and at least one corresponding target version of the image. The method also includes training a neural network based on the training data to predict an enhanced version of an input image, wherein the training of the neural network comprises applying a forward Gaussian diffusion process that adds Gaussian noise to the at least one corresponding target version of each of the plurality of pairs of images to enable iterative denoising of the input image, wherein the iterative denoising is based on a reverse Markov chain associated with the forward Gaussian diffusion process. The method additionally includes outputting the trained neural network.
    Type: Application
    Filed: August 2, 2021
    Publication date: March 2, 2023
    Inventors: Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, Mohammad Norouzi
  • Publication number: 20220374658
    Abstract: Systems, methods, and computer program products for performing semi-supervised contrastive learning of visual representations are provided. For example, the present disclosure provides systems and methods that leverage particular data augmentation schemes and a learnable nonlinear transformation between the representation and the contrastive loss to provide improved visual representations. Further, the present disclosure also provides improvements for semi-supervised contrastive learning.
    Type: Application
    Filed: July 12, 2022
    Publication date: November 24, 2022
    Inventors: Ting Chen, Geoffrey Everest Hinton, Simon Kornblith, Mohammad Norouzi
  • Patent number: 11429844
    Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a policy neural network used to select actions to be performed by a reinforcement learning agent interacting with an environment. In one aspect, a method includes obtaining path data defining a path through the environment traversed by the agent. A consistency error is determined for the path from a combined reward, first and last soft-max state values, and a path likelihood. A value update for the current values of the policy neural network parameters is determined from at least the consistency error. The value update is used to adjust the current values of the policy neural network parameters.
    Type: Grant
    Filed: June 18, 2020
    Date of Patent: August 30, 2022
    Assignee: Google LLC
    Inventors: Ofir Nachum, Mohammad Norouzi, Dale Eric Schuurmans, Kelvin Xu
  • Patent number: 11386302
    Abstract: Systems, methods, and computer program products for performing semi-supervised contrastive learning of visual representations are provided. For example, the present disclosure provides systems and methods that leverage particular data augmentation schemes and a learnable nonlinear transformation between the representation and the contrastive loss to provide improved visual representations. Further, the present disclosure also provides improvements for semi-supervised contrastive learning.
    Type: Grant
    Filed: September 11, 2020
    Date of Patent: July 12, 2022
    Assignee: GOOGLE LLC
    Inventors: Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Everest Hinton, Kevin Jordan Swersky
  • Patent number: 11354778
    Abstract: Provided are systems and methods for contrastive learning of visual representations. In particular, the present disclosure provides systems and methods that leverage particular data augmentation schemes and a learnable nonlinear transformation between the representation and the contrastive loss to provide improved visual representations. In contrast to certain existing techniques, the contrastive self-supervised learning algorithms described herein do not require specialized architectures or a memory bank. Some example implementations of the proposed approaches can be referred to as a simple framework for contrastive learning of representations or “SimCLR.” Further example aspects are described below and provide the following benefits and insights.
    Type: Grant
    Filed: April 13, 2020
    Date of Patent: June 7, 2022
    Assignee: GOOGLE LLC
    Inventors: Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Everest Hinton
  • Publication number: 20210390271
    Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for neural machine translation. The method comprises obtaining a first sequence of words in a source language, generating a modified sequence of words in the source language by inserting a word boundary symbol only at the beginning of each word in the first sequence of words and not at the end of each word, dividing the modified sequence of words into wordpieces using a wordpiece model, generating, from the wordpieces, an input sequence of input tokens for a neural machine translation system; and generating an output sequence of words using the neural machine translation system based on the input sequence of input tokens.
    Type: Application
    Filed: August 27, 2021
    Publication date: December 16, 2021
    Inventors: Mohammad Norouzi, Zhifeng Chen, Yonghui Wu, Michael Schuster, Quoc V. Le
  • Publication number: 20210327029
    Abstract: Provided are systems and methods for contrastive learning of visual representations. In particular, the present disclosure provides systems and methods that leverage particular data augmentation schemes and a learnable nonlinear transformation between the representation and the contrastive loss to provide improved visual representations. In contrast to certain existing techniques, the contrastive self-supervised learning algorithms described herein do not require specialized architectures or a memory bank. Some example implementations of the proposed approaches can be referred to as a simple framework for contrastive learning of representations or “SimCLR.” Further example aspects are described below and provide the following benefits and insights.
    Type: Application
    Filed: April 13, 2020
    Publication date: October 21, 2021
    Inventors: Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Everest Hinton
  • Publication number: 20210319266
    Abstract: Systems, methods, and computer program products for performing semi-supervised contrastive learning of visual representations are provided. For example, the present disclosure provides systems and methods that leverage particular data augmentation schemes and a learnable nonlinear transformation between the representation and the contrastive loss to provide improved visual representations. Further, the present disclosure also provides improvements for semi-supervised contrastive learning.
    Type: Application
    Filed: September 11, 2020
    Publication date: October 14, 2021
    Inventors: Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Everest Hinton
  • Patent number: 11113480
    Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for neural machine translation. One of the systems includes an encoder neural network comprising: an input forward long short-term memory (LSTM) layer configured to process each input token in the input sequence in a forward order to generate a respective forward representation of each input token, an input backward LSTM layer configured to process each input token in a backward order to generate a respective backward representation of each input token and a plurality of hidden LSTM layers configured to process a respective combined representation of each of the input tokens in the forward order to generate a respective encoded representation of each of the input tokens; and a decoder subsystem configured to receive the respective encoded representations and to process the encoded representations to generate an output sequence.
    Type: Grant
    Filed: September 25, 2017
    Date of Patent: September 7, 2021
    Assignee: Google LLC
    Inventors: Mohammad Norouzi, Zhifeng Chen, Yonghui Wu, Michael Schuster, Quoc V. Le
  • Publication number: 20210256313
    Abstract: Methods and systems for learning policies using sparse and underspecified rewards. One of the methods includes training the policy jointly with an auxiliary reward function having a plurality of auxiliary reward parameters, the auxiliary reward function being configured to map, in accordance with the auxiliary reward parameters, trajectory features of at least a trajectory to an auxiliary reward value that indicates how well the trajectory performed a task in response to a context input.
    Type: Application
    Filed: February 19, 2021
    Publication date: August 19, 2021
    Inventors: Rishabh Agarwal, Chen Liang, Dale Eric Schuurmans, Mohammad Norouzi
  • Patent number: 11087504
    Abstract: Systems and methods for transforming grayscale images into color images using deep neural networks are described. One of the systems include one or more computers and one or more storage devices storing instructions that, when executed by one or more computers, cause the one or more computers to implement a coloring neural network, a refinement neural network, and a subsystem. The coloring neural network is configured to receive a first grayscale image having a first resolution and to process the first grayscale image to generate a first color image having a second resolution lower than the first resolution. The subsystem processes the first color image to generate a set of intermediate image outputs. The refinement neural network is configured to receive the set intermediate image outputs, and to process the set of intermediate image outputs to generate a second color image having a third resolution higher than the second resolution.
    Type: Grant
    Filed: May 21, 2018
    Date of Patent: August 10, 2021
    Assignee: Google LLC
    Inventors: Sergio Guadarrama Cotado, Jonathon Shlens, David Bieber, Mohammad Norouzi, Kevin Patrick Murphy, Ryan Lienhart Dahl
  • Publication number: 20210158162
    Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an action selection policy neural network used to select an action to be performed by an agent interacting with an environment. In one aspect, a method includes: receiving a latent representation characterizing a current state of the environment; generating a trajectory of latent representations that starts with the received latent representation; for each latent representation in the trajectory: determining a predicted reward; and processing the state latent representation using a value neural network to generate a predicted state value; determining a corresponding target state value for each latent representation in the trajectory; determining, based on the target state values, an update to the current values of the policy neural network parameters; and determining an update to the current values of the value neural network parameters.
    Type: Application
    Filed: November 24, 2020
    Publication date: May 27, 2021
    Inventors: Danijar Hafner, Mohammad Norouzi, Timothy Paul Lillicrap
  • Publication number: 20200380023
    Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for classifying data objects. One of the methods includes obtaining data that associates each term in a vocabulary of terms with a respective high-dimensional representation of the term; obtaining classification data for a data object, wherein the classification data includes a respective score for each of a plurality of categories, and wherein each of the categories is associated with a respective category label; computing an aggregate high-dimensional representation for the data object from high-dimensional representations for the category labels associated with the categories and the respective scores; identifying a first term in the vocabulary of terms having a high-dimensional representation that is closest to the aggregate high-dimensional representation; and selecting the first term as a category label for the data object.
    Type: Application
    Filed: August 20, 2020
    Publication date: December 3, 2020
    Inventors: Gregory Sean Corrado, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea L. Frome, Jeffrey Adgate Dean, Mohammad Norouzi
  • Publication number: 20200320372
    Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a policy neural network used to select actions to be performed by a reinforcement learning agent interacting with an environment. In one aspect, a method includes obtaining path data defining a path through the environment traversed by the agent. A consistency error is determined for the path from a combined reward, first and last soft-max state values, and a path likelihood. A value update for the current values of the policy neural network parameters is determined from at least the consistency error. The value update is used to adjust the current values of the policy neural network parameters.
    Type: Application
    Filed: June 18, 2020
    Publication date: October 8, 2020
    Inventors: Ofir Nachum, Mohammad Norouzi, Dale Eric Schuurmans, Kelvin Xu
  • Patent number: 10769191
    Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for classifying data objects. One of the methods includes obtaining data that associates each term in a vocabulary of terms with a respective high-dimensional representation of the term; obtaining classification data for a data object, wherein the classification data includes a respective score for each of a plurality of categories, and wherein each of the categories is associated with a respective category label; computing an aggregate high-dimensional representation for the data object from high-dimensional representations for the category labels associated with the categories and the respective scores; identifying a first term in the vocabulary of terms having a high-dimensional representation that is closest to the aggregate high-dimensional representation; and selecting the first term as a category label for the data object.
    Type: Grant
    Filed: December 19, 2014
    Date of Patent: September 8, 2020
    Assignee: Google LLC
    Inventors: Gregory Sean Corrado, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea L. Frome, Jeffrey Adgate Dean, Mohammad Norouzi
  • Publication number: 20200279163
    Abstract: A method for determining a placement for machine learning model operations across multiple hardware devices is described.
    Type: Application
    Filed: May 20, 2020
    Publication date: September 3, 2020
    Inventors: Samuel Bengio, Mohammad Norouzi, Benoit Steiner, Jeffrey Adgate Dean, Hieu Hy Pham, Azalia Mirhoseini, Quoc V. Le, Naveen Kumar, Yuefeng Zhou, Rasmus Munk Larsen
  • Patent number: 10733502
    Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a policy neural network used to select actions to be performed by a reinforcement learning agent interacting with an environment. In one aspect, a method includes obtaining path data defining a path through the environment traversed by the agent. A consistency error is determined for the path from a combined reward, first and last soft-max state values, and a path likelihood. A value update for the current values of the policy neural network parameters is determined from at least the consistency error. The value update is used to adjust the current values of the policy neural network parameters.
    Type: Grant
    Filed: July 8, 2019
    Date of Patent: August 4, 2020
    Assignee: Google LLC
    Inventors: Ofir Nachum, Mohammad Norouzi, Dale Eric Schuurmans, Kelvin Xu