Abstract: In implementation of techniques for generating volumetric representations from panorama images, a computing device implements a volumetric system to receive a two-dimensional panorama image. The volumetric system generates a feature map that indicates relationships between pixels of the two-dimensional panorama image. Based on the feature map, the volumetric system generates a volumetric representation by rearranging the pixels indicated by the feature map into a three-dimensional spherical map using a machine learning model.

Abstract: Methods are described for rendering a virtual object at a designated position in an input digital image corresponding to a perspective of a scene. In an embodiment, the method includes: estimating a set of lighting parameters using a lighting neural network; estimating a scene layout using a layout neural network; generating an environment texture map using a texture neural network using an input including the input digital image, the lighting parameters, and the scene layout; rendering the virtual object in a virtual scene constructed using the estimated lighting parameters, the scene layout, and the environment texture map; and compositing the rendered virtual object on the input digital image at the designated position. Corresponding systems and non-transitory computer-readable media are also described.

Abstract: The present disclosure provides computer-implemented methods and systems to generate high dynamic range (HDR) panoramas to render virtual objects. A low dynamic range (LDR) panorama from a prompt describing a desired aspect of a panorama. The LDR panorama is then converted to an HDR panorama. The process can include generating a backplate image using a first machine learning model, projecting the image onto a sphere and inpainting it using a second machine learning model to generate an LDR panorama, and converting the LDR panorama to an HDR panorama using a third machine learning model. Each of the 10 first, second and third machine learning models can be diffusion models. Alternatively, the third machine learning model can be a convolutional neural network model. A physics-based rendering engine can then be used to render a virtual object in the HDR panorama.

Abstract: Methods are described for rendering a virtual object at a designated position in an input digital image corresponding to a perspective of a scene. In an embodiment, the method includes: estimating a set of lighting parameters using a lighting neural network; estimating a scene layout using a layout neural network; generating an environment texture map using a texture neural network using an input including the input digital image, the lighting parameters, and the scene layout; rendering the virtual object in a virtual scene constructed using the estimated lighting parameters, the scene layout, and the environment texture map; and compositing the rendered virtual object on the input digital image at the designated position. Corresponding systems and non-transitory computer-readable media are also described.

Abstract: In some examples, a computing device processes depth images to capture an interaction of an object relative to an interaction plane.

Abstract: In some examples, a computing device processes depth images to capture an interaction of an object relative to an interaction plane.