Patents by Inventor Jon Niklas Theodor Hasselgren

Jon Niklas Theodor Hasselgren has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Patent number: 11657571
    Abstract: Systems and methods enable optimization of a 3D model representation comprising the shape and appearance of a particular 3D scene or object. The opaque 3D mesh (e.g., vertex positions and corresponding topology) and spatially varying material attributes are jointly optimized based on image space losses to match multiple image observations (e.g., reference images of the reference 3D scene or object). A geometric topology defines faces and/or cells in the opaque 3D mesh that are visible and may be randomly initialized and optimized through training based on the image space losses. Applying the geometry topology to an opaque 3D mesh for learning the shape improves accuracy of silhouette edges and performance compared with using transparent mesh representations. In contrast with approaches that require an initial guess for the topology and/or an exhaustive testing of possible geometric topologies, the 3D model representation is learned based on image space differences without requiring an initial guess.
    Type: Grant
    Filed: December 13, 2022
    Date of Patent: May 23, 2023
    Assignee: NVIDIA Corporation
    Inventors: Jon Niklas Theodor Hasselgren, Carl Jacob Munkberg
  • Publication number: 20230140460
    Abstract: A technique is described for extracting or constructing a three-dimensional (3D) model from multiple two-dimensional (2D) images. In an embodiment, a foreground segmentation mask or depth field may be provided as an additional supervision input with each 2D image. In an embodiment, the foreground segmentation mask or depth field is automatically generated for each 2D image. The constructed 3D model comprises a triangular mesh topology, materials, and environment lighting. The constructed 3D model is represented in a format that can be directly edited and/or rendered by conventional application programs, such as digital content creation (DCC) tools. For example, the constructed 3D model may be represented as a triangular surface mesh (with arbitrary topology), a set of 2D textures representing spatially-varying material parameters, and an environment map. Furthermore, the constructed 3D model may be included in 3D scenes and interacts realistically with other objects.
    Type: Application
    Filed: May 30, 2022
    Publication date: May 4, 2023
    Inventors: Carl Jacob Munkberg, Jon Niklas Theodor Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex John Bauld Evans, Thomas Müller-Höhne, Sanja Fidler
  • Publication number: 20230105025
    Abstract: Systems and methods enable optimization of a 3D model representation comprising the shape and appearance of a particular 3D scene or object. The opaque 3D mesh (e.g., vertex positions and corresponding topology) and spatially varying material attributes are jointly optimized based on image space losses to match multiple image observations (e.g., reference images of the reference 3D scene or object). A geometric topology defines faces and/or cells in the opaque 3D mesh that are visible and may be randomly initialized and optimized through training based on the image space losses. Applying the geometry topology to an opaque 3D mesh for learning the shape improves accuracy of silhouette edges and performance compared with using transparent mesh representations. In contrast with approaches that require an initial guess for the topology and/or an exhaustive testing of possible geometric topologies, the 3D model representation is learned based on image space differences without requiring an initial guess.
    Type: Application
    Filed: December 13, 2022
    Publication date: April 6, 2023
    Inventors: Jon Niklas Theodor Hasselgren, Carl Jacob Munkberg
  • Patent number: 11615602
    Abstract: Appearance driven automatic three-dimensional (3D) modeling enables optimization of a 3D model comprising the shape and appearance of a particular 3D scene or object. Triangle meshes and shading models may be jointly optimized to match the appearance of a reference 3D model based on reference images of the reference 3D model. Compared with the reference 3D model, the optimized 3D model is a lower resolution 3D model that can be rendered in less time. More specifically, the optimized 3D model may include fewer geometric primitives compared with the reference 3D model. In contrast with the conventional inverse rendering or analysis-by-synthesis modeling tools, the shape and appearance representations of the 3D model are automatically generated that, when rendered, match the reference images.
    Type: Grant
    Filed: August 15, 2022
    Date of Patent: March 28, 2023
    Assignee: NVIDIA Corporation
    Inventors: Carl Jacob Munkberg, Jon Niklas Theodor Hasselgren
  • Patent number: 11610370
    Abstract: Systems and methods enable optimization of a 3D model representation comprising the shape and appearance of a particular 3D scene or object. The opaque 3D mesh (e.g., vertex positions and corresponding topology) and spatially varying material attributes are jointly optimized based on image space losses to match multiple image observations (e.g., reference images of the reference 3D scene or object). A geometric topology defines faces and/or cells in the opaque 3D mesh that are visible and may be randomly initialized and optimized through training based on the image space losses. Applying the geometry topology to an opaque 3D mesh for learning the shape improves accuracy of silhouette edges and performance compared with using transparent mesh representations. In contrast with approaches that require an initial guess for the topology and/or an exhaustive testing of possible geometric topologies, the 3D model representation is learned based on image space differences without requiring an initial guess.
    Type: Grant
    Filed: August 27, 2021
    Date of Patent: March 21, 2023
    Assignee: NVIDIA Corporation
    Inventors: Jon Niklas Theodor Hasselgren, Carl Jacob Munkberg
  • Publication number: 20230014245
    Abstract: A neural network-based rendering technique increases temporal stability and image fidelity of low sample count path tracing by optimizing a distribution of samples for rendering each image in a sequence. A sample predictor neural network learns spatio-temporal sampling strategies such as placing more samples in dis-occluded regions and tracking specular highlights. Temporal feedback enables a denoiser neural network to boost the effective input sample count and increases temporal stability. The initial uniform sampling step typically present in adaptive sampling algorithms is not needed. The sample predictor and denoiser operate at interactive rates to achieve significantly improved image quality and temporal stability compared with conventional adaptive sampling techniques.
    Type: Application
    Filed: September 8, 2022
    Publication date: January 19, 2023
    Inventors: Carl Jacob Munkberg, Jon Niklas Theodor Hasselgren, Anjul Patney, Marco Salvi, Aaron Eliot Lefohn, Donald Lee Brittain
  • Patent number: 11557022
    Abstract: A neural network-based rendering technique increases temporal stability and image fidelity of low sample count path tracing by optimizing a distribution of samples for rendering each image in a sequence. A sample predictor neural network learns spatio-temporal sampling strategies such as placing more samples in dis-occluded regions and tracking specular highlights. Temporal feedback enables a denoiser neural network to boost the effective input sample count and increases temporal stability. The initial uniform sampling step typically present in adaptive sampling algorithms is not needed. The sample predictor and denoiser operate at interactive rates to achieve significantly improved image quality and temporal stability compared with conventional adaptive sampling techniques.
    Type: Grant
    Filed: December 18, 2019
    Date of Patent: January 17, 2023
    Assignee: NVIDIA Corporation
    Inventors: Carl Jacob Munkberg, Jon Niklas Theodor Hasselgren, Anjul Patney, Marco Salvi, Aaron Eliot Lefohn, Donald Lee Brittain
  • Publication number: 20220405582
    Abstract: A method, computer readable medium, and system are disclosed for training a neural network model. The method includes the step of selecting an input vector from a set of training data that includes input vectors and sparse target vectors, where each sparse target vector includes target data corresponding to a subset of samples within an output vector of the neural network model. The method also includes the steps of processing the input vector by the neural network model to produce output data for the samples within the output vector and adjusting parameter values of the neural network model to reduce differences between the output vector and the sparse target vector for the subset of the samples.
    Type: Application
    Filed: February 4, 2022
    Publication date: December 22, 2022
    Inventors: Carl Jacob Munkberg, Jon Niklas Theodor Hasselgren, Jaakko T. Lehtinen, Timo Oskari Aila
  • Publication number: 20220392160
    Abstract: Systems and methods enable optimization of a 3D model representation comprising the shape and appearance of a particular 3D scene or object. The opaque 3D mesh (e.g., vertex positions and corresponding topology) and spatially varying material attributes are jointly optimized based on image space losses to match multiple image observations (e.g., reference images of the reference 3D scene or object). A geometric topology defines faces and/or cells in the opaque 3D mesh that are visible and may be randomly initialized and optimized through training based on the image space losses. Applying the geometry topology to an opaque 3D mesh for learning the shape improves accuracy of silhouette edges and performance compared with using transparent mesh representations. In contrast with approaches that require an initial guess for the topology and/or an exhaustive testing of possible geometric topologies, the 3D model representation is learned based on image space differences without requiring an initial guess.
    Type: Application
    Filed: August 27, 2021
    Publication date: December 8, 2022
    Inventors: Jon Niklas Theodor Hasselgren, Carl Jacob Munkberg
  • Publication number: 20220392179
    Abstract: Appearance driven automatic three-dimensional (3D) modeling enables optimization of a 3D model comprising the shape and appearance of a particular 3D scene or object. Triangle meshes and shading models may be jointly optimized to match the appearance of a reference 3D model based on reference images of the reference 3D model. Compared with the reference 3D model, the optimized 3D model is a lower resolution 3D model that can be rendered in less time. More specifically, the optimized 3D model may include fewer geometric primitives compared with the reference 3D model. In contrast with the conventional inverse rendering or analysis-by-synthesis modeling tools, the shape and appearance representations of the 3D model are automatically generated that, when rendered, match the reference images.
    Type: Application
    Filed: August 15, 2022
    Publication date: December 8, 2022
    Inventors: Carl Jacob Munkberg, Jon Niklas Theodor Hasselgren
  • Patent number: 11475542
    Abstract: A neural network-based rendering technique increases temporal stability and image fidelity of low sample count path tracing by optimizing a distribution of samples for rendering each image in a sequence. A sample predictor neural network learns spatio-temporal sampling strategies such as placing more samples in dis-occluded regions and tracking specular highlights. Temporal feedback enables a denoiser neural network to boost the effective input sample count and increases temporal stability. The initial uniform sampling step typically present in adaptive sampling algorithms is not needed. The sample predictor and denoiser operate at interactive rates to achieve significantly improved image quality and temporal stability compared with conventional adaptive sampling techniques.
    Type: Grant
    Filed: December 17, 2019
    Date of Patent: October 18, 2022
    Assignee: NVIDIA Corporation
    Inventors: Carl Jacob Munkberg, Jon Niklas Theodor Hasselgren, Anjul Patney, Marco Salvi, Aaron Eliot Lefohn, Donald Lee Brittain
  • Patent number: 11450077
    Abstract: Appearance driven automatic three-dimensional (3D) modeling enables optimization of a 3D model comprising the shape and appearance of a particular 3D scene or object. Triangle meshes and shading models may be jointly optimized to match the appearance of a reference 3D model based on reference images of the reference 3D model. Compared with the reference 3D model, the optimized 3D model is a lower resolution 3D model that can be rendered in less time. More specifically, the optimized 3D model may include fewer geometric primitives compared with the reference 3D model. In contrast with the conventional inverse rendering or analysis-by-synthesis modeling tools, the shape and appearance representations of the 3D model are automatically generated that, when rendered, match the reference images.
    Type: Grant
    Filed: March 8, 2021
    Date of Patent: September 20, 2022
    Assignee: NVIDIA Corporation
    Inventors: Carl Jacob Munkberg, Jon Niklas Theodor Hasselgren
  • Publication number: 20220165040
    Abstract: Appearance driven automatic three-dimensional (3D) modeling enables optimization of a 3D model comprising the shape and appearance of a particular 3D scene or object. Triangle meshes and shading models may be jointly optimized to match the appearance of a reference 3D model based on reference images of the reference 3D model. Compared with the reference 3D model, the optimized 3D model is a lower resolution 3D model that can be rendered in less time. More specifically, the optimized 3D model may include fewer geometric primitives compared with the reference 3D model. In contrast with the conventional inverse rendering or analysis-by-synthesis modeling tools, the shape and appearance representations of the 3D model are automatically generated that, when rendered, match the reference images.
    Type: Application
    Filed: March 8, 2021
    Publication date: May 26, 2022
    Inventors: Carl Jacob Munkberg, Jon Niklas Theodor Hasselgren
  • Patent number: 11244226
    Abstract: A method, computer readable medium, and system are disclosed for training a neural network model. The method includes the step of selecting an input vector from a set of training data that includes input vectors and sparse target vectors, where each sparse target vector includes target data corresponding to a subset of samples within an output vector of the neural network model. The method also includes the steps of processing the input vector by the neural network model to produce output data for the samples within the output vector and adjusting parameter values of the neural network model to reduce differences between the output vector and the sparse target vector for the subset of the samples.
    Type: Grant
    Filed: January 26, 2018
    Date of Patent: February 8, 2022
    Assignee: NVIDIA Corporation
    Inventors: Carl Jacob Munkberg, Jon Niklas Theodor Hasselgren, Jaakko T. Lehtinen, Timo Oskari Aila
  • Publication number: 20210374384
    Abstract: Apparatuses, systems, and techniques to identify one or more layers of a three-dimensional graphical image to generate a two-dimensional representation. In at least one embodiment, one or more layers of a three-dimensional graphical image are identified to generate one or more two-dimensional representations.
    Type: Application
    Filed: June 2, 2020
    Publication date: December 2, 2021
    Inventors: Carl Jacob Munkberg, Jon Niklas Theodor Hasselgren
  • Patent number: 10970816
    Abstract: A neural network structure, namely a warped external recurrent neural network, is disclosed for reconstructing images with synthesized effects. The effects can include motion blur, depth of field reconstruction (e.g., simulating lens effects), and/or anti-aliasing (e.g., removing artifacts caused by sampling frequency). The warped external recurrent neural network is not recurrent at each layer inside the neural network. Instead, the external state output by the final layer of the neural network is warped and provided as a portion of the input to the neural network for the next image in a sequence of images. In contrast, in a conventional recurrent neural network, hidden state generated at each layer is provided as a feedback input to the generating layer. The neural network can be implemented, at least in part, on a processor. In an embodiment, the neural network is implemented on at least one parallel processing unit.
    Type: Grant
    Filed: May 24, 2019
    Date of Patent: April 6, 2021
    Assignee: Nvidia Corporation
    Inventors: Carl Jacob Munkberg, Jon Niklas Theodor Hasselgren, Marco Salvi
  • Publication number: 20200126191
    Abstract: A neural network-based rendering technique increases temporal stability and image fidelity of low sample count path tracing by optimizing a distribution of samples for rendering each image in a sequence. A sample predictor neural network learns spatio-temporal sampling strategies such as placing more samples in dis-occluded regions and tracking specular highlights. Temporal feedback enables a denoiser neural network to boost the effective input sample count and increases temporal stability. The initial uniform sampling step typically present in adaptive sampling algorithms is not needed. The sample predictor and denoiser operate at interactive rates to achieve significantly improved image quality and temporal stability compared with conventional adaptive sampling techniques.
    Type: Application
    Filed: December 17, 2019
    Publication date: April 23, 2020
    Inventors: Carl Jacob Munkberg, Jon Niklas Theodor Hasselgren, Anjul Patney, Marco Salvi, Aaron Eliot Lefohn, Donald Lee Brittain
  • Publication number: 20200126192
    Abstract: A neural network-based rendering technique increases temporal stability and image fidelity of low sample count path tracing by optimizing a distribution of samples for rendering each image in a sequence. A sample predictor neural network learns spatio-temporal sampling strategies such as placing more samples in dis-occluded regions and tracking specular highlights. Temporal feedback enables a denoiser neural network to boost the effective input sample count and increases temporal stability. The initial uniform sampling step typically present in adaptive sampling algorithms is not needed. The sample predictor and denoiser operate at interactive rates to achieve significantly improved image quality and temporal stability compared with conventional adaptive sampling techniques.
    Type: Application
    Filed: December 18, 2019
    Publication date: April 23, 2020
    Inventors: Carl Jacob Munkberg, Jon Niklas Theodor Hasselgren, Anjul Patney, Marco Salvi, Aaron Eliot Lefohn, Donald Lee Brittain
  • Patent number: 10565686
    Abstract: A method, computer readable medium, and system are disclosed for training a neural network. The method includes the steps of selecting an input sample from a set of training data that includes input samples and noisy target samples, where the input samples and the noisy target samples each correspond to a latent, clean target sample. The input sample is processed by a neural network model to produce an output and a noisy target sample is selected from the set of training data, where the noisy target samples have a distribution relative to the latent, clean target sample. The method also includes adjusting parameter values of the neural network model to reduce differences between the output and the noisy target sample.
    Type: Grant
    Filed: November 8, 2017
    Date of Patent: February 18, 2020
    Assignee: NVIDIA Corporation
    Inventors: Jaakko T. Lehtinen, Timo Oskari Aila, Jon Niklas Theodor Hasselgren, Carl Jacob Munkberg
  • Publication number: 20200051206
    Abstract: A neural network structure, namely a warped external recurrent neural network, is disclosed for reconstructing images with synthesized effects. The effects can include motion blur, depth of field reconstruction (e.g., simulating lens effects), and/or anti-aliasing (e.g., removing artifacts caused by sampling frequency). The warped external recurrent neural network is not recurrent at each layer inside the neural network. Instead, the external state output by the final layer of the neural network is warped and provided as a portion of the input to the neural network for the next image in a sequence of images. In contrast, in a conventional recurrent neural network, hidden state generated at each layer is provided as a feedback input to the generating layer. The neural network can be implemented, at least in part, on a processor. In an embodiment, the neural network is implemented on at least one parallel processing unit.
    Type: Application
    Filed: May 24, 2019
    Publication date: February 13, 2020
    Inventors: Carl Jacob Munkberg, Jon Niklas Theodor Hasselgren, Marco Salvi