Patents by Inventor Mohammad Norouzi
Mohammad Norouzi has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).
-
Publication number: 20230153959Abstract: A method includes receiving, by a computing device, training data comprising a plurality of pairs of images, wherein each pair comprises an image and at least one corresponding target version of the image. The method also includes training a neural network based on the training data to predict an enhanced version of an input image, wherein the training of the neural network comprises applying a forward Gaussian diffusion process that adds Gaussian noise to the at least one corresponding target version of each of the plurality of pairs of images to enable iterative denoising of the input image, wherein the iterative denoising is based on a reverse Markov chain associated with the forward Gaussian diffusion process. The method additionally includes outputting the trained neural network.Type: ApplicationFiled: January 17, 2023Publication date: May 18, 2023Inventors: Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, Mohammad Norouzi
-
Publication number: 20230103638Abstract: A method includes receiving training data comprising a plurality of pairs of images. Each pair comprises a noisy image and a denoised version of the noisy image. The method also includes training a multi-task diffusion model to perform a plurality of image-to-image translation tasks, wherein the training comprises iteratively generating a forward diffusion process by predicting, at each iteration in a sequence of iterations and based on a current noisy estimate of the denoised version of the noisy image, noise data for a next noisy estimate of the denoised version of the noisy image, updating, at each iteration, the current noisy estimate to the next noisy estimate by combining the current noisy estimate with the predicted noise data, and determining a reverse diffusion process by inverting the forward diffusion process to predict the denoised version of the noisy image. The method additionally includes providing the trained diffusion model.Type: ApplicationFiled: October 5, 2022Publication date: April 6, 2023Inventors: Chitwan Saharia, Mohammad Norouzi, William Chan, Huiwen Chang, David James Fleet, Christopher Albert Lee, Jonathan Ho, Tim Salimans
-
Publication number: 20230075716Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for sequence modeling. One of the methods includes receiving an input sequence having a plurality of input positions; determining a plurality of blocks of consecutive input positions; processing the input sequence using a neural network to generate a latent alignment, comprising, at each of a plurality of input time steps: receiving a partial latent alignment from a previous input time step; selecting an input position in each block, wherein the token at the selected input position of the partial latent alignment in each block is a mask token; and processing the partial latent alignment and the input sequence using the neural network to generate a new latent alignment, wherein the new latent alignment comprises, at the selected input position in each block, an output token or a blank token; and generating, using the latent alignment, an output sequence.Type: ApplicationFiled: February 8, 2021Publication date: March 9, 2023Inventors: William Chan, Chitwan Saharia, Geoffrey E. Hinton, Mohammad Norouzi, Navdeep Jaitly
-
Publication number: 20230067841Abstract: A method includes receiving, by a computing device, training data comprising a plurality of pairs of images, wherein each pair comprises an image and at least one corresponding target version of the image. The method also includes training a neural network based on the training data to predict an enhanced version of an input image, wherein the training of the neural network comprises applying a forward Gaussian diffusion process that adds Gaussian noise to the at least one corresponding target version of each of the plurality of pairs of images to enable iterative denoising of the input image, wherein the iterative denoising is based on a reverse Markov chain associated with the forward Gaussian diffusion process. The method additionally includes outputting the trained neural network.Type: ApplicationFiled: August 2, 2021Publication date: March 2, 2023Inventors: Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, Mohammad Norouzi
-
Publication number: 20220374658Abstract: Systems, methods, and computer program products for performing semi-supervised contrastive learning of visual representations are provided. For example, the present disclosure provides systems and methods that leverage particular data augmentation schemes and a learnable nonlinear transformation between the representation and the contrastive loss to provide improved visual representations. Further, the present disclosure also provides improvements for semi-supervised contrastive learning.Type: ApplicationFiled: July 12, 2022Publication date: November 24, 2022Inventors: Ting Chen, Geoffrey Everest Hinton, Simon Kornblith, Mohammad Norouzi
-
Patent number: 11429844Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a policy neural network used to select actions to be performed by a reinforcement learning agent interacting with an environment. In one aspect, a method includes obtaining path data defining a path through the environment traversed by the agent. A consistency error is determined for the path from a combined reward, first and last soft-max state values, and a path likelihood. A value update for the current values of the policy neural network parameters is determined from at least the consistency error. The value update is used to adjust the current values of the policy neural network parameters.Type: GrantFiled: June 18, 2020Date of Patent: August 30, 2022Assignee: Google LLCInventors: Ofir Nachum, Mohammad Norouzi, Dale Eric Schuurmans, Kelvin Xu
-
Patent number: 11386302Abstract: Systems, methods, and computer program products for performing semi-supervised contrastive learning of visual representations are provided. For example, the present disclosure provides systems and methods that leverage particular data augmentation schemes and a learnable nonlinear transformation between the representation and the contrastive loss to provide improved visual representations. Further, the present disclosure also provides improvements for semi-supervised contrastive learning.Type: GrantFiled: September 11, 2020Date of Patent: July 12, 2022Assignee: GOOGLE LLCInventors: Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Everest Hinton, Kevin Jordan Swersky
-
Patent number: 11354778Abstract: Provided are systems and methods for contrastive learning of visual representations. In particular, the present disclosure provides systems and methods that leverage particular data augmentation schemes and a learnable nonlinear transformation between the representation and the contrastive loss to provide improved visual representations. In contrast to certain existing techniques, the contrastive self-supervised learning algorithms described herein do not require specialized architectures or a memory bank. Some example implementations of the proposed approaches can be referred to as a simple framework for contrastive learning of representations or “SimCLR.” Further example aspects are described below and provide the following benefits and insights.Type: GrantFiled: April 13, 2020Date of Patent: June 7, 2022Assignee: GOOGLE LLCInventors: Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Everest Hinton
-
Publication number: 20210390271Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for neural machine translation. The method comprises obtaining a first sequence of words in a source language, generating a modified sequence of words in the source language by inserting a word boundary symbol only at the beginning of each word in the first sequence of words and not at the end of each word, dividing the modified sequence of words into wordpieces using a wordpiece model, generating, from the wordpieces, an input sequence of input tokens for a neural machine translation system; and generating an output sequence of words using the neural machine translation system based on the input sequence of input tokens.Type: ApplicationFiled: August 27, 2021Publication date: December 16, 2021Inventors: Mohammad Norouzi, Zhifeng Chen, Yonghui Wu, Michael Schuster, Quoc V. Le
-
Publication number: 20210327029Abstract: Provided are systems and methods for contrastive learning of visual representations. In particular, the present disclosure provides systems and methods that leverage particular data augmentation schemes and a learnable nonlinear transformation between the representation and the contrastive loss to provide improved visual representations. In contrast to certain existing techniques, the contrastive self-supervised learning algorithms described herein do not require specialized architectures or a memory bank. Some example implementations of the proposed approaches can be referred to as a simple framework for contrastive learning of representations or “SimCLR.” Further example aspects are described below and provide the following benefits and insights.Type: ApplicationFiled: April 13, 2020Publication date: October 21, 2021Inventors: Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Everest Hinton
-
Publication number: 20210319266Abstract: Systems, methods, and computer program products for performing semi-supervised contrastive learning of visual representations are provided. For example, the present disclosure provides systems and methods that leverage particular data augmentation schemes and a learnable nonlinear transformation between the representation and the contrastive loss to provide improved visual representations. Further, the present disclosure also provides improvements for semi-supervised contrastive learning.Type: ApplicationFiled: September 11, 2020Publication date: October 14, 2021Inventors: Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Everest Hinton
-
Patent number: 11113480Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for neural machine translation. One of the systems includes an encoder neural network comprising: an input forward long short-term memory (LSTM) layer configured to process each input token in the input sequence in a forward order to generate a respective forward representation of each input token, an input backward LSTM layer configured to process each input token in a backward order to generate a respective backward representation of each input token and a plurality of hidden LSTM layers configured to process a respective combined representation of each of the input tokens in the forward order to generate a respective encoded representation of each of the input tokens; and a decoder subsystem configured to receive the respective encoded representations and to process the encoded representations to generate an output sequence.Type: GrantFiled: September 25, 2017Date of Patent: September 7, 2021Assignee: Google LLCInventors: Mohammad Norouzi, Zhifeng Chen, Yonghui Wu, Michael Schuster, Quoc V. Le
-
Publication number: 20210256313Abstract: Methods and systems for learning policies using sparse and underspecified rewards. One of the methods includes training the policy jointly with an auxiliary reward function having a plurality of auxiliary reward parameters, the auxiliary reward function being configured to map, in accordance with the auxiliary reward parameters, trajectory features of at least a trajectory to an auxiliary reward value that indicates how well the trajectory performed a task in response to a context input.Type: ApplicationFiled: February 19, 2021Publication date: August 19, 2021Inventors: Rishabh Agarwal, Chen Liang, Dale Eric Schuurmans, Mohammad Norouzi
-
Patent number: 11087504Abstract: Systems and methods for transforming grayscale images into color images using deep neural networks are described. One of the systems include one or more computers and one or more storage devices storing instructions that, when executed by one or more computers, cause the one or more computers to implement a coloring neural network, a refinement neural network, and a subsystem. The coloring neural network is configured to receive a first grayscale image having a first resolution and to process the first grayscale image to generate a first color image having a second resolution lower than the first resolution. The subsystem processes the first color image to generate a set of intermediate image outputs. The refinement neural network is configured to receive the set intermediate image outputs, and to process the set of intermediate image outputs to generate a second color image having a third resolution higher than the second resolution.Type: GrantFiled: May 21, 2018Date of Patent: August 10, 2021Assignee: Google LLCInventors: Sergio Guadarrama Cotado, Jonathon Shlens, David Bieber, Mohammad Norouzi, Kevin Patrick Murphy, Ryan Lienhart Dahl
-
Publication number: 20210158162Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an action selection policy neural network used to select an action to be performed by an agent interacting with an environment. In one aspect, a method includes: receiving a latent representation characterizing a current state of the environment; generating a trajectory of latent representations that starts with the received latent representation; for each latent representation in the trajectory: determining a predicted reward; and processing the state latent representation using a value neural network to generate a predicted state value; determining a corresponding target state value for each latent representation in the trajectory; determining, based on the target state values, an update to the current values of the policy neural network parameters; and determining an update to the current values of the value neural network parameters.Type: ApplicationFiled: November 24, 2020Publication date: May 27, 2021Inventors: Danijar Hafner, Mohammad Norouzi, Timothy Paul Lillicrap
-
Publication number: 20200380023Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for classifying data objects. One of the methods includes obtaining data that associates each term in a vocabulary of terms with a respective high-dimensional representation of the term; obtaining classification data for a data object, wherein the classification data includes a respective score for each of a plurality of categories, and wherein each of the categories is associated with a respective category label; computing an aggregate high-dimensional representation for the data object from high-dimensional representations for the category labels associated with the categories and the respective scores; identifying a first term in the vocabulary of terms having a high-dimensional representation that is closest to the aggregate high-dimensional representation; and selecting the first term as a category label for the data object.Type: ApplicationFiled: August 20, 2020Publication date: December 3, 2020Inventors: Gregory Sean Corrado, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea L. Frome, Jeffrey Adgate Dean, Mohammad Norouzi
-
Publication number: 20200320372Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a policy neural network used to select actions to be performed by a reinforcement learning agent interacting with an environment. In one aspect, a method includes obtaining path data defining a path through the environment traversed by the agent. A consistency error is determined for the path from a combined reward, first and last soft-max state values, and a path likelihood. A value update for the current values of the policy neural network parameters is determined from at least the consistency error. The value update is used to adjust the current values of the policy neural network parameters.Type: ApplicationFiled: June 18, 2020Publication date: October 8, 2020Inventors: Ofir Nachum, Mohammad Norouzi, Dale Eric Schuurmans, Kelvin Xu
-
Patent number: 10769191Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for classifying data objects. One of the methods includes obtaining data that associates each term in a vocabulary of terms with a respective high-dimensional representation of the term; obtaining classification data for a data object, wherein the classification data includes a respective score for each of a plurality of categories, and wherein each of the categories is associated with a respective category label; computing an aggregate high-dimensional representation for the data object from high-dimensional representations for the category labels associated with the categories and the respective scores; identifying a first term in the vocabulary of terms having a high-dimensional representation that is closest to the aggregate high-dimensional representation; and selecting the first term as a category label for the data object.Type: GrantFiled: December 19, 2014Date of Patent: September 8, 2020Assignee: Google LLCInventors: Gregory Sean Corrado, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea L. Frome, Jeffrey Adgate Dean, Mohammad Norouzi
-
Publication number: 20200279163Abstract: A method for determining a placement for machine learning model operations across multiple hardware devices is described.Type: ApplicationFiled: May 20, 2020Publication date: September 3, 2020Inventors: Samuel Bengio, Mohammad Norouzi, Benoit Steiner, Jeffrey Adgate Dean, Hieu Hy Pham, Azalia Mirhoseini, Quoc V. Le, Naveen Kumar, Yuefeng Zhou, Rasmus Munk Larsen
-
Patent number: 10733502Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a policy neural network used to select actions to be performed by a reinforcement learning agent interacting with an environment. In one aspect, a method includes obtaining path data defining a path through the environment traversed by the agent. A consistency error is determined for the path from a combined reward, first and last soft-max state values, and a path likelihood. A value update for the current values of the policy neural network parameters is determined from at least the consistency error. The value update is used to adjust the current values of the policy neural network parameters.Type: GrantFiled: July 8, 2019Date of Patent: August 4, 2020Assignee: Google LLCInventors: Ofir Nachum, Mohammad Norouzi, Dale Eric Schuurmans, Kelvin Xu