Abstract: A method for generating, by an encoder-based model, a three-dimensional (3D) representation of a two-dimensional (2D) image is provided. The encoder-based model is trained to infer the 3D representation using a synthetic training data set generated by a pre-trained model. The pre-trained model is a 3D generative model that produces a 3D representation and a corresponding 2D rendering, which can be used to train a separate encoder-based model for downstream tasks like estimating a triplane representation, neural radiance field, mesh, depth map, 3D key points, or the like, given a single input image, using the pseudo ground truth 3D synthetic training data set. In a particular embodiment, the encoder-based model is trained to predict a triplane representation of the input image, which can then be rendered by a volume renderer according to pose information to generate an output image of the 3D scene from the corresponding viewpoint.

Abstract: Visual cue augmentation for virtual windows adds visual cues into a rendered virtual window that is displayed in a real environment. Visual cue contingent rendering reinforces connections between virtual and real objects and spaces, harmonizing a virtual window scene displayed within a real environment with the real environment. In an embodiment, a virtual window is rendered that includes the visual cues. In an embodiment, using head tracking, the virtual window changes as a viewer in the real environment changes their viewing direction. In another embodiment, a true light field display is used to present the virtual window without requiring head tracking. In an environment, a motion parallax display (MPD) is used to perform user head tracking and presenting the virtual window. In general, the virtual window that is displayed responds to a viewer's head position.

Abstract: Systems including a pelvis depth estimation model configured to generate a pelvis depth estimate of a person depicted in an image, a human mesh estimation model configured to generate a body mesh corresponding to the person depicted in the image given the estimated pelvis depth, and a camera solver configured to apply differentiable rasterization to derive camera parameters for the image.

Abstract: A remote device utilizes ray tracing to compute a light field for a scene to be rendered, where the light field includes information about light reflected off surfaces within the scene. This light field is then compressed utilizing one or more video compression techniques that implement temporal reuse, such that only differences between the light field for the scene and a light field for a previous scene are compressed. The compressed light field data is then sent to a client device that decompresses the light field data and uses such data to obtain the light field for the scene at the client device. This light field is then used by the client device to compute global illumination for the scene. The global illumination may be used to accurately render the scene at the mobile device, resulting in a realistic scene that is presented by the mobile device.

Abstract: Converting neural radiance fields to three-dimensional (3D) Gaussians includes receiving multi-view images, projecting camera rays through each of the multi-view images, generating first 3D points by sampling each of the camera rays, generating first five-dimensional (5D) input coordinates from the first 3D points and a corresponding two-dimensional (2D) viewing direction, processing the first 5D input coordinates using a neural network to generate first 3D Gaussians, generating second 3D points by sampling each of the camera rays, generating second 5D input coordinates from the second 3D points and the corresponding 2D viewing direction, processing the second 5D input coordinates using the neural network to generate second 3D Gaussians, and pruning the first 3D Gaussians summed with the second 3D Gaussians.

Abstract: Apparatuses, systems, and techniques to determine head poses of users and provide audio for the users. In at least one embodiment, a head pose is determined based, at least in part, on camera frame information, and an audio signal is generated, based at least in part, on the determined head pose.

Abstract: Systems and methods for fast light field rendering from a Neural Radiance Field (NeRF), for example, to visualize a three-dimensional (3D) scene represented by the NeRF on a 3D display. In at least one embodiment, fast light field rendering exploits intersection of sampling points in a ray pattern corresponding to an orthographic imaging array, thereby enhancing computational efficiency during rendering.

Abstract: One embodiment of a method includes calculating one or more activation values of one or more neural networks trained to infer eye gaze information based, at least in part, on eye position of one or more images of one or more faces indicated by an infrared light reflection from the one or more images.

Abstract: At least one embodiment is directed towards a computer-implemented method for training generative artificial intelligence (AI) models. The computer-implemented method includes the steps of receiving a plurality of training images; rendering, via a generative AI model, a plurality of synthetic images based on the plurality of training images; generating triplane loss metrics for the plurality of synthetic images by comparing the plurality of synthetic images against the plurality of training images; generating total variation (TV) loss metrics based on the triplane loss metrics; generating triplane compression loss metrics based on the triplane loss metrics; generating total loss metrics based on the TV loss metrics and the triplane compression loss metrics; and performing at least one backpropagation operation based on the total loss metrics to update weights associated with the generative AI model to generate an updated generative AI model.

Abstract: At least one embodiment is directed towards a computer-implemented method for rendering video content. The computer-implemented method includes the steps of decompressing compressed video content to generate decompressed video content, wherein the decompressed video content includes a plurality of normalized triplanes; de-normalizing the plurality of normalized triplanes to generate a plurality of modified triplanes; performing neural rendering operations to generate a plurality of final images via ray tracing based on the plurality of modified triplanes; and displaying the plurality of final images as rendered video content via a display device. Another embodiment is directed towards a computer-method for generating compressed video content. Yet another embodiment is directed towards a computer-implemented method for training generative artificial intelligence (AI) models.

Abstract: At least one embodiment is directed towards a computer-implemented method for generating compressed video content. The computer-implemented method includes the steps of receiving a plurality of triplanes associated with video content; extracting channel range values from each triplane included in the plurality of triplanes; normalizing the plurality of triplanes based on the channel range values to generate a plurality of normalized triplanes; storing the channel range values with the plurality of normalized triplanes; generating a plurality of tiled triplanes based on the plurality of normalized triplanes; compressing the plurality of tiled triplanes to generate compressed video content; and transmitting the compressed video content to an endpoint device. Another embodiment is directed towards a computer-method for rendering video content. Yet another embodiment is directed towards a computer-implemented method for training generative artificial intelligence (AI) models.

Abstract: The disclosure provides a method for audio calibration that uses audio simulation and reconstructed surface information from images or video recordings along with recorded sound. The surface component of the method introduces knowledge that enables audio wave propagation simulation for a particular location allowing optimization of the sound distribution. For example, unwanted audio reflection and occlusion can be recognized and resolved. In one example, the disclosure provides a method for improving acoustics at a location that includes: (1) obtaining visual data of a location using one or more cameras of a mobile computing device, (2) obtaining audio measurements at the location using at least one microphone of the mobile computing device, wherein the location includes an audio system and the audio measurements correspond to audio generated by the audio system, and (3) sending the visual data and the audio measurements to a computing system for improving acoustics at the location.

Abstract: Visual cue augmentation for virtual windows adds visual cues into a rendered virtual window that is displayed in a real environment. Visual cue contingent rendering reinforces connections between virtual and real objects and spaces, harmonizing a virtual window scene displayed within a real environment with the real environment. In an embodiment, a virtual window is rendered that includes the visual cues. In an embodiment, using head tracking, the virtual window changes as a viewer in the real environment changes their viewing direction. In another embodiment, a true light field display is used to present the virtual window without requiring head tracking. In an environment, a motion parallax display (MPD) is used to perform user head tracking and presenting the virtual window. In general, the virtual window that is displayed responds to a viewer's head position.

Abstract: Apparatuses, systems, and techniques are presented to estimate user gaze. In at least one embodiment, one or more neural networks are used to determine coarse and fine gaze estimates for one or more users.

Abstract: The disclosure provides a method for audio calibration that uses audio simulation and reconstructed surface information from images or video recordings along with recorded sound. The surface component of the method introduces knowledge that enables audio wave propagation simulation for a particular location. Using the simulation results the sound distribution can be optimized. For example, unwanted audio reflection and occlusion can be recognized and resolved. In one example, the disclosure provides a method for improving acoustics at a location that includes: (1) generating a geometric model of a location using visual data obtained from the location, wherein the location includes an audio system, and (2) simulating, using the geometric model, movement of sound waves in the location that originate from the audio system. The disclosure also provides a computer system, a computer program product, and a mobile computing device that include features for improving acoustics at a location.

Abstract: Systems and methods are disclosed that utilize a triplane fusion architecture that achieves both state-of-the-art three-dimensional (3D) reconstruction accuracy and temporal consistency. For instance, recognizing the need to maintain both coherent identity and dynamic per-frame appearance to enable the best possible realism, the triplane fusion architecture may include an undistorter that removes view-dependent distortions from the raw triplane by using the triplane prior as a reference and a fuser that fuses the triplanes together to recover the occluded areas in the input frame by incorporating features from the triplane prior that is lifted from the frontal reference image. By using the undistorter, the fuser, and a volume renderer, the triplane fusion architecture allows for reconstruction of 3D portrait models from a monocular video stream of a user, which may be useful in various applications including 3D telepresence.

Abstract: In an embodiment, a modular augmented reality display is provided that incorporates prescription eyewear that can be used separately by the wearer. In an embodiment, an image is generated from a removable display attached to the eyewear and directed into the edge of a prescription lens, which acts as a waveguide. The image is internally reflected within the prescription lens, and is directed to the wearer by an image combiner embedded within the prescription lens. In an embodiment, the augmented reality display includes a wearable belt pouch that includes a battery and support electronics connected to the eyewear so that the weight on the eyewear is reduced.

Abstract: In an embodiment, a modular augmented reality display is provided that incorporates prescription eyewear that can be used separately by the wearer. In an embodiment, an image is generated from a removable display attached to the eyewear and directed into the edge of a prescription lens, which acts as a waveguide. The image is internally reflected within the prescription lens, and is directed to the wearer by an image combiner embedded within the prescription lens. In an embodiment, the augmented reality display includes a wearable belt pouch that includes a battery and support electronics connected to the eyewear so that the weight on the eyewear is reduced.

Abstract: The disclosure provides a method for audio calibration that uses audio simulation and reconstructed surface information from images or video recordings along with recorded sound. The surface component of the method introduces knowledge that enables audio wave propagation simulation for a particular location. Using the simulation results the sound distribution can be optimized. For example, unwanted audio reflection and occlusion can be recognized and resolved. In one example, the disclosure provides a method for improving acoustics at a location that includes: (1) generating a geometric model of a location using visual data obtained from the location, wherein the location includes an audio system, and (2) simulating, using the geometric model, movement of sound waves in the location that originate from the audio system. The disclosure also provides a computer system, a computer program product, and a mobile computing device that include features for improving acoustics at a location.

Abstract: A remote device utilizes ray tracing to compute a light field for a scene to be rendered, where the light field includes information about light reflected off surfaces within the scene. This light field is then compressed utilizing one or more video compression techniques that implement temporal reuse, such that only differences between the light field for the scene and a light field for a previous scene are compressed. The compressed light field data is then sent to a client device that decompresses the light field data and uses such data to obtain the light field for the scene at the client device. This light field is then used by the client device to compute global illumination for the scene. The global illumination may be used to accurately render the scene at the mobile device, resulting in a realistic scene that is presented by the mobile device.