Abstract: Methods are described for rendering a virtual object at a designated position in an input digital image corresponding to a perspective of a scene. In an embodiment, the method includes: estimating a set of lighting parameters using a lighting neural network; estimating a scene layout using a layout neural network; generating an environment texture map using a texture neural network using an input including the input digital image, the lighting parameters, and the scene layout; rendering the virtual object in a virtual scene constructed using the estimated lighting parameters, the scene layout, and the environment texture map; and compositing the rendered virtual object on the input digital image at the designated position. Corresponding systems and non-transitory computer-readable media are also described.

Abstract: The present disclosure provides computer-implemented methods and systems to generate high dynamic range (HDR) panoramas to render virtual objects. A low dynamic range (LDR) panorama from a prompt describing a desired aspect of a panorama. The LDR panorama is then converted to an HDR panorama. The process can include generating a backplate image using a first machine learning model, projecting the image onto a sphere and inpainting it using a second machine learning model to generate an LDR panorama, and converting the LDR panorama to an HDR panorama using a third machine learning model. Each of the 10 first, second and third machine learning models can be diffusion models. Alternatively, the third machine learning model can be a convolutional neural network model. A physics-based rendering engine can then be used to render a virtual object in the HDR panorama.

Abstract: Methods are described for rendering a virtual object at a designated position in an input digital image corresponding to a perspective of a scene. In an embodiment, the method includes: estimating a set of lighting parameters using a lighting neural network; estimating a scene layout using a layout neural network; generating an environment texture map using a texture neural network using an input including the input digital image, the lighting parameters, and the scene layout; rendering the virtual object in a virtual scene constructed using the estimated lighting parameters, the scene layout, and the environment texture map; and compositing the rendered virtual object on the input digital image at the designated position. Corresponding systems and non-transitory computer-readable media are also described.

Abstract: This disclosure relates to methods, non-transitory computer readable media, and systems that use a local-lighting-estimation-neural network to estimate lighting parameters for specific positions within a digital scene for augmented reality. For example, based on a request to render a virtual object in a digital scene, a system uses a local-lighting-estimation-neural network to generate location-specific-lighting parameters for a designated position within the digital scene. In certain implementations, the system also renders a modified digital scene comprising the virtual object at the designated position according to the parameters. In some embodiments, the system generates such location-specific-lighting parameters to spatially vary and adapt lighting conditions for different positions within a digital scene.

Abstract: This disclosure relates to methods, non-transitory computer readable media, and systems that use a local-lighting-estimation-neural network to estimate lighting parameters for specific positions within a digital scene for augmented reality. For example, based on a request to render a virtual object in a digital scene, a system uses a local-lighting-estimation-neural network to generate location-specific-lighting parameters for a designated position within the digital scene. In certain implementations, the system also renders a modified digital scene comprising the virtual object at the designated position according to the parameters. In some embodiments, the system generates such location-specific-lighting parameters to spatially vary and adapt lighting conditions for different positions within a digital scene.

Abstract: This disclosure relates to methods, non-transitory computer readable media, and systems that use a local-lighting-estimation-neural network to estimate lighting parameters for specific positions within a digital scene for augmented reality. For example, based on a request to render a virtual object in a digital scene, a system uses a local-lighting-estimation-neural network to generate location-specific-lighting parameters for a designated position within the digital scene. In certain implementations, the system also renders a modified digital scene comprising the virtual object at the designated position according to the parameters. In some embodiments, the system generates such location-specific-lighting parameters to spatially vary and adapt lighting conditions for different positions within a digital scene.

Abstract: This disclosure relates to methods, non-transitory computer readable media, and systems that use a local-lighting-estimation-neural network to render a virtual object in a digital scene by using a local-lighting-estimation-neural network to analyze both global and local features of the digital scene and generate location-specific-lighting parameters for a designated position within the digital scene. For example, the disclosed systems extract and combine such global and local features from a digital scene using global network layers and local network layers of the local-lighting-estimation-neural network. In certain implementations, the disclosed systems can generate location-specific-lighting parameters using a neural-network architecture that combines global and local feature vectors to spatially vary lighting for different positions within a digital scene.