Patents by Inventor Timo Oskari Aila

Timo Oskari Aila has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Patent number: 11315018
    Abstract: A method, computer readable medium, and system are disclosed for neural network pruning. The method includes the steps of receiving first-order gradients of a cost function relative to layer parameters for a trained neural network and computing a pruning criterion for each layer parameter based on the first-order gradient corresponding to the layer parameter, where the pruning criterion indicates an importance of each neuron that is included in the trained neural network and is associated with the layer parameter. The method includes the additional steps of identifying at least one neuron having a lowest importance and removing the at least one neuron from the trained neural network to produce a pruned neural network.
    Type: Grant
    Filed: October 17, 2017
    Date of Patent: April 26, 2022
    Assignee: NVIDIA Corporation
    Inventors: Pavlo Molchanov, Stephen Walter Tyree, Tero Tapani Karras, Timo Oskari Aila, Jan Kautz
  • Publication number: 20220121958
    Abstract: A generative adversarial neural network (GAN) learns a particular task by being shown many examples. In one scenario, a GAN may be trained to generate new images including specific objects, such as human faces, bicycles, etc. Rather than training a complex GAN having a predetermined topology of features and interconnections between the features to learn the task, the topology of the GAN is modified as the GAN is trained for the task. The topology of the GAN may be simple in the beginning and become more complex as the GAN learns during the training, eventually evolving to match the predetermined topology of the complex GAN. In the beginning the GAN learns large-scale details for the task (bicycles have two wheels) and later, as the GAN becomes more complex, learns smaller details (the wheels have spokes).
    Type: Application
    Filed: January 3, 2022
    Publication date: April 21, 2022
    Inventors: Tero Tapani Karras, Timo Oskari Aila, Samuli Matias Laine, Jaakko T. Lehtinen
  • Patent number: 11263525
    Abstract: A neural network learns a particular task by being shown many examples. In one scenario, a neural network may be trained to label an image, such as cat, dog, bicycle, chair, etc. In other scenario, a neural network may be trained to remove noise from videos or identify specific objects within images, such as human faces, bicycles, etc. Rather than training a complex neural network having a predetermined topology of features and interconnections between the features to learn the task, the topology of the neural network is modified as the neural network is trained for the task, eventually evolving to match the predetermined topology of the complex neural network. In the beginning the neural network learns large-scale details for the task (bicycles have two wheels) and later, as the neural network becomes more complex, learns smaller details (the wheels have spokes).
    Type: Grant
    Filed: January 18, 2019
    Date of Patent: March 1, 2022
    Assignee: NVIDIA Corporation
    Inventors: Tero Tapani Karras, Timo Oskari Aila, Samuli Matias Laine, Jaakko T. Lehtinen, Janne Hellsten
  • Publication number: 20220051481
    Abstract: A three-dimensional (3D) model of an object is recovered from two-dimensional (2D) images of the object. Each image in the set of 2D images includes the object captured from a different camera position and deformations of a base mesh that defines the 3D model may be computed corresponding to each image. The 3D model may also include a texture map that represents the lighting and material properties of the 3D model. Recovery of the 3D model relies on analytic antialiasing to provide a link between pixel colors in the 2D images and geometry of the 3D model. A modular differentiable renderer design yields high performance by leveraging existing, highly optimized hardware graphics pipelines to reconstruct the 3D model. The differential renderer renders images of the 3D model and differences between the rendered images and reference images are propagated backwards through the rendering pipeline to iteratively adjust the 3D model.
    Type: Application
    Filed: February 15, 2021
    Publication date: February 17, 2022
    Inventors: Samuli Matias Laine, Janne Johannes Hellsten, Tero Tapani Karras, Yeongho Seol, Jaakko T. Lehtinen, Timo Oskari Aila
  • Patent number: 11250329
    Abstract: A generative adversarial neural network (GAN) learns a particular task by being shown many examples. In one scenario, a GAN may be trained to generate new images including specific objects, such as human faces, bicycles, etc. Rather than training a complex GAN having a predetermined topology of features and interconnections between the features to learn the task, the topology of the GAN is modified as the GAN is trained for the task. The topology of the GAN may be simple in the beginning and become more complex as the GAN learns during the training, eventually evolving to match the predetermined topology of the complex GAN. In the beginning the GAN learns large-scale details for the task (bicycles have two wheels) and later, as the GAN becomes more complex, learns smaller details (the wheels have spokes).
    Type: Grant
    Filed: October 10, 2018
    Date of Patent: February 15, 2022
    Assignee: NVIDIA Corporation
    Inventors: Tero Tapani Karras, Timo Oskari Aila, Samuli Matias Laine, Jaakko T. Lehtinen
  • Patent number: 11244226
    Abstract: A method, computer readable medium, and system are disclosed for training a neural network model. The method includes the step of selecting an input vector from a set of training data that includes input vectors and sparse target vectors, where each sparse target vector includes target data corresponding to a subset of samples within an output vector of the neural network model. The method also includes the steps of processing the input vector by the neural network model to produce output data for the samples within the output vector and adjusting parameter values of the neural network model to reduce differences between the output vector and the sparse target vector for the subset of the samples.
    Type: Grant
    Filed: January 26, 2018
    Date of Patent: February 8, 2022
    Assignee: NVIDIA Corporation
    Inventors: Carl Jacob Munkberg, Jon Niklas Theodor Hasselgren, Jaakko T. Lehtinen, Timo Oskari Aila
  • Publication number: 20210383241
    Abstract: Embodiments of the present disclosure relate to a technique for training neural networks, such as a generative adversarial neural network (GAN), using a limited amount of data. Training GANs using too little example data typically leads to discriminator overfitting, causing training to diverge and produce poor results. An adaptive discriminator augmentation mechanism is used that significantly stabilizes training with limited data providing the ability to train high-quality GANs. An augmentation operator is applied to the distribution of inputs to a discriminator used to train a generator, representing a transformation that is invertible to ensure there is no leakage of the augmentations into the images generated by the generator. Reducing the amount of training data that is needed to achieve convergence has the potential to considerably help many applications and may the increase use of generative models in fields such as medicine.
    Type: Application
    Filed: March 24, 2021
    Publication date: December 9, 2021
    Inventors: Tero Tapani Karras, Miika Samuli Aittala, Janne Johannes Hellsten, Samuli Matias Laine, Jaakko T. Lehtinen, Timo Oskari Aila
  • Publication number: 20210329306
    Abstract: Apparatuses, systems, and techniques to perform compression of video data using neural networks to facilitate video streaming, such as video conferencing. In at least one embodiment, a sender transmits to a receiver a key frame from video data and one or more keypoints identified by a neural network from said video data, and a receiver reconstructs video data using said key frame and one or more received keypoints.
    Type: Application
    Filed: October 13, 2020
    Publication date: October 21, 2021
    Inventors: Ming-Yu Liu, Ting-Chun Wang, Arun Mohanray Mallya, Tero Tapani Karras, Samuli Matias Laine, David Patrick Luebke, Jaakko Lehtinen, Miika Samuli Aittala, Timo Oskari Aila
  • Patent number: 11113800
    Abstract: A method, computer readable medium, and system are disclosed for performing spatiotemporal filtering. The method includes identifying image data to be rendered, reconstructing the image data to create reconstructed image data, utilizing a filter including a neural network having one or more skip connections and one or more recurrent layers, and returning the reconstructed image data.
    Type: Grant
    Filed: January 16, 2018
    Date of Patent: September 7, 2021
    Assignee: NVIDIA CORPORATION
    Inventors: Anton S. Kaplanyan, Chakravarty Reddy Alla Chaitanya, Timo Oskari Aila, Aaron Eliot Lefohn, Marco Salvi
  • Patent number: 11068781
    Abstract: A method, computer readable medium, and system are disclosed for implementing a temporal ensembling model for training a deep neural network. The method for training the deep neural network includes the steps of receiving a set of training data for a deep neural network and training the deep neural network utilizing the set of training data by: analyzing the plurality of input vectors by the deep neural network to generate a plurality of prediction vectors, and, for each prediction vector in the plurality of prediction vectors corresponding to the particular input vector, computing a loss term associated with the particular input vector by combining a supervised component and an unsupervised component according to a weighting function and updating the target prediction vector associated with the particular input vector.
    Type: Grant
    Filed: September 29, 2017
    Date of Patent: July 20, 2021
    Assignee: NVIDIA Corporation
    Inventors: Samuli Matias Laine, Timo Oskari Aila
  • Publication number: 20210150187
    Abstract: A latent code defined in an input space is processed by the mapping neural network to produce an intermediate latent code defined in an intermediate latent space. The intermediate latent code may be used as appearance vector that is processed by the synthesis neural network to generate an image. The appearance vector is a compressed encoding of data, such as video frames including a person's face, audio, and other data. Captured images may be converted into appearance vectors at a local device and transmitted to a remote device using much less bandwidth compared with transmitting the captured images. A synthesis neural network at the remote device reconstructs the images for display.
    Type: Application
    Filed: January 7, 2021
    Publication date: May 20, 2021
    Inventors: Tero Tapani Karras, Samuli Matias Laine, David Patrick Luebke, Jaakko T. Lehtinen, Miika Samuli Aittala, Timo Oskari Aila, Ming-Yu Liu, Arun Mohanray Mallya, Ting-Chun Wang
  • Publication number: 20210150357
    Abstract: A style-based generative network architecture enables scale-specific control of synthesized output data, such as images. During training, the style-based generative neural network (generator neural network) includes a mapping network and a synthesis network. During prediction, the mapping network may be omitted, replicated, or evaluated several times. The synthesis network may be used to generate highly varied, high-quality output data with a wide variety of attributes. For example, when used to generate images of people's faces, the attributes that may vary are age, ethnicity, camera viewpoint, pose, face shape, eyeglasses, colors (eyes, hair, etc.), hair style, lighting, background, etc. Depending on the task, generated output data may include images, audio, video, three-dimensional (3D) objects, text, etc.
    Type: Application
    Filed: January 28, 2021
    Publication date: May 20, 2021
    Inventors: Tero Tapani Karras, Samuli Matias Laine, Jaakko T. Lehtinen, Miika Samuli Aittala, Janne Johannes Hellsten, Timo Oskari Aila
  • Publication number: 20210150369
    Abstract: A style-based generative network architecture enables scale-specific control of synthesized output data, such as images. During training, the style-based generative neural network (generator neural network) includes a mapping network and a synthesis network. During prediction, the mapping network may be omitted, replicated, or evaluated several times. The synthesis network may be used to generate highly varied, high-quality output data with a wide variety of attributes. For example, when used to generate images of people's faces, the attributes that may vary are age, ethnicity, camera viewpoint, pose, face shape, eyeglasses, colors (eyes, hair, etc.), hair style, lighting, background, etc. Depending on the task, generated output data may include images, audio, video, three-dimensional (3D) objects, text, etc.
    Type: Application
    Filed: January 28, 2021
    Publication date: May 20, 2021
    Inventors: Tero Tapani Karras, Samuli Matias Laine, Jaakko T. Lehtinen, Miika Samuli Aittala, Janne Johannes Hellsten, Timo Oskari Aila
  • Publication number: 20210150354
    Abstract: A latent code defined in an input space is processed by the mapping neural network to produce an intermediate latent code defined in an intermediate latent space. The intermediate latent code may be used as appearance vector that is processed by the synthesis neural network to generate an image. The appearance vector is a compressed encoding of data, such as video frames including a person's face, audio, and other data. Captured images may be converted into appearance vectors at a local device and transmitted to a remote device using much less bandwidth compared with transmitting the captured images. A synthesis neural network at the remote device reconstructs the images for display.
    Type: Application
    Filed: January 7, 2021
    Publication date: May 20, 2021
    Inventors: Tero Tapani Karras, Samuli Matias Laine, David Patrick Luebke, Jaakko T. Lehtinen, Miika Samuli Aittala, Timo Oskari Aila, Ming-Yu Liu, Arun Mohanray Mallya, Ting-Chun Wang
  • Publication number: 20210117795
    Abstract: A style-based generative network architecture enables scale-specific control of synthesized output data, such as images. During training, the style-based generative neural network (generator neural network) includes a mapping network and a synthesis network. During prediction, the mapping network may be omitted, replicated, or evaluated several times. The synthesis network may be used to generate highly varied, high-quality output data with a wide variety of attributes. For example, when used to generate images of people's faces, the attributes that may vary are age, ethnicity, camera viewpoint, pose, face shape, eyeglasses, colors (eyes, hair, etc.), hair style, lighting, background, etc. Depending on the task, generated output data may include images, audio, video, three-dimensional (3D) objects, text, etc.
    Type: Application
    Filed: December 28, 2020
    Publication date: April 22, 2021
    Inventors: Tero Tapani Karras, Timo Oskari Aila, Samuli Matias Laine
  • Publication number: 20210049468
    Abstract: A latent code defined in an input space is processed by the mapping neural network to produce an intermediate latent code defined in an intermediate latent space. The intermediate latent code may be used as appearance vector that is processed by the synthesis neural network to generate an image. The appearance vector is a compressed encoding of data, such as video frames including a person's face, audio, and other data. Captured images may be converted into appearance vectors at a local device and transmitted to a remote device using much less bandwidth compared with transmitting the captured images. A synthesis neural network at the remote device reconstructs the images for display.
    Type: Application
    Filed: October 13, 2020
    Publication date: February 18, 2021
    Inventors: Tero Tapani Karras, Samuli Matias Laine, David Patrick Luebke, Jaakko T. Lehtinen, Miika Samuli Aittala, Timo Oskari Aila, Ming-Yu Liu, Arun Mohanray Mallya, Ting-Chun Wang
  • Publication number: 20210042503
    Abstract: A latent code defined in an input space is processed by the mapping neural network to produce an intermediate latent code defined in an intermediate latent space. The intermediate latent code may be used as appearance vector that is processed by the synthesis neural network to generate an image. The appearance vector is a compressed encoding of data, such as video frames including a person's face, audio, and other data. Captured images may be converted into appearance vectors at a local device and transmitted to a remote device using much less bandwidth compared with transmitting the captured images. A synthesis neural network at the remote device reconstructs the images for display.
    Type: Application
    Filed: October 13, 2020
    Publication date: February 11, 2021
    Inventors: Tero Tapani Karras, Samuli Matias Laine, David Patrick Luebke, Jaakko T. Lehtinen, Miika Samuli Aittala, Timo Oskari Aila, Ming-Yu Liu, Arun Mohanray Mallya, Ting-Chun Wang
  • Patent number: 10866990
    Abstract: An apparatus, computer readable medium, and method are disclosed for decompressing compressed geometric data stored in a lossless compression format. The compressed geometric data resides within a compression block sized according to a system cache line. An indirection technique maps a global identifier value in a linear identifier space to corresponding variable rate compressed data. The apparatus may include decompression circuitry within a graphics processing unit configured to perform ray-tracing.
    Type: Grant
    Filed: July 3, 2019
    Date of Patent: December 15, 2020
    Assignee: NVIDIA Corporation
    Inventors: Jaakko Lehtinen, Timo Oskari Aila, Tero Tapani Karras, Alexander Keller, Nikolaus Binder, Carsten Alexander Waechter, Samuli Matias Laine
  • Publication number: 20200242739
    Abstract: A neural network architecture is disclosed for restoring noisy data. The neural network is a blind-spot network that can be trained according to a self-supervised framework. In an embodiment, the blind-spot network includes a plurality of network branches. Each network branch processes a version of the input data using one or more layers associated with kernels that have a receptive field that extends in a particular half-plane relative to the output value. In one embodiment, the versions of the input data are offset in a particular direction and the convolution kernels are rotated to correspond to the particular direction of the associated network branch. In another embodiment, the versions of the input data are rotated and the convolution kernel is the same for each network branch. The outputs of the network branches are composited to de-noise the image. In some embodiments, Bayesian filtering is performed to de-noise the input data.
    Type: Application
    Filed: December 20, 2019
    Publication date: July 30, 2020
    Inventors: Samuli Matias Laine, Tero Tapani Karras, Jaakko T. Lehtinen, Timo Oskari Aila
  • Publication number: 20200151559
    Abstract: A style-based generative network architecture enables scale-specific control of synthesized output data, such as images. During training, the style-based generative neural network (generator neural network) includes a mapping network and a synthesis network. During prediction, the mapping network may be omitted, replicated, or evaluated several times. The synthesis network may be used to generate highly varied, high-quality output data with a wide variety of attributes. For example, when used to generate images of people's faces, the attributes that may vary are age, ethnicity, camera viewpoint, pose, face shape, eyeglasses, colors (eyes, hair, etc.), hair style, lighting, background, etc. Depending on the task, generated output data may include images, audio, video, three-dimensional (3D) objects, text, etc.
    Type: Application
    Filed: May 21, 2019
    Publication date: May 14, 2020
    Inventors: Tero Tapani Karras, Timo Oskari Aila, Samuli Matias Laine