Abstract: In various examples, image-based three-dimensional occupant assessment for in-cabin monitoring systems and applications are provided. An evaluation function may determine a 3D representation of an occupant of a machine by evaluating sensor data comprising an image frame from an optical image sensor. The 3D representation may comprise at least one characteristic representative of a size of the occupant, (e.g., a 3D pose and/or 3D shape), which may be used to derive other characteristics such as, but not limited to weight, height, and/or age. A first processing path may generate a representation of one or more features corresponding to at least a portion of the occupant based on optical image data, and a second processing path may determine a depth corresponding to the one or more features based on depth data derived from the optical image data and ground truth depth data corresponding to the interior of the machine.

Abstract: Systems and methods are disclosed that improve performance of synthesized motion generated by a diffusion neural network model. A physics-guided motion diffusion model incorporates physical constraints into the diffusion process to model the complex dynamics induced by forces and contact. Specifically, a physics-based motion projection module uses motion imitation in a physics simulator to project the denoised motion of a diffusion step to a physically plausible motion. The projected motion is further used in the next diffusion iteration to guide the denoising diffusion process. The use of physical constraints in the physics-guided motion diffusion model iteratively pulls the motion toward a physically-plausible space, reducing artifacts such as floating, foot sliding, and ground penetration.

Abstract: In various examples, systems and methods are disclosed relating to generating animatable characters or avatars. The system can assign a plurality of first elements of a three-dimensional (3D) model of a subject to a plurality of locations on a surface of the subject in an initial pose. Further, the system can assign a plurality of second elements to the plurality of first elements, each second element of the plurality of second elements having an opacity corresponding to a distance between the second element and the surface of the subject. Further, the system can update the plurality of second elements based at least on a target pose for the subject and one or more attributes of the subject to determine a plurality of updated second elements. Further, the system can render a representation of the subject based at least on the plurality of updated second elements.

Abstract: Systems and methods are disclosed that animate a source portrait image with motion (i.e., pose and expression) from a target image. In contrast to conventional systems, given an unseen single-view portrait image, an implicit three-dimensional (3D) head avatar is constructed that not only captures photo-realistic details within and beyond the face region, but also is readily available for animation without requiring further optimization during inference. In an embodiment, three processing branches of a system produce three tri-planes representing coarse 3D geometry for the head avatar, detailed appearance of a source image, as well as the expression of a target image. By applying volumetric rendering to a combination of the three tri-planes, an image of the desired identity, expression and pose is generated.

Abstract: In various examples, systems and methods are disclosed relating to generating physics-plausible whole body motion, including determining a mesh sequence corresponding to a motion of at least one dynamic character of one or more dynamic characters and a mesh of a terrain using a video sequence, determining using a generative model and based at least one the mesh sequence and the mesh of the terrain, an occlusion-free motion of the at least one dynamic character by infilling physics-plausible character motions in the mesh sequence for at least one frame of the video sequence that includes an occlusion of at least a portion of the at least one dynamic character, and determining physics-plausible whole body motion of the at least one dynamic character by applying physics-based imitation upon the occlusion-free motion.

Abstract: One embodiment of a method for controlling a character includes receiving one or more conditions, based on the one or more conditions, generating or estimating a motion using a trained machine learning model, wherein the trained machine learning model is configured to estimate the motion via one denoising diffusion step or generate the motion via a plurality of denoising diffusion steps, and causing the character to move based on the motion.

Abstract: One embodiment of a method for training a machine learning model to predict motion includes estimating, using an untrained machine learning model, a first motion based on at least one first condition; generating, using the untrained machine learning model, a second motion based on at least one second condition; computing a loss based on the first motion, the second motion, and one or more ground truth motions; and updating one or more parameters of the untrained machine learning model based on the loss to generate a trained machine learning model.

Abstract: In various examples, systems and methods are disclosed relating to generating animatable characters or avatars. The system can assign a plurality of first elements of a three-dimensional (3D) model of a subject to a plurality of locations on a surface of the subject in an initial pose. Further, the system can assign a plurality of second elements to the plurality of first elements, each second element of the plurality of second elements having an opacity corresponding to a distance between the second element and the surface of the subject. Further, the system can update the plurality of second elements based at least on a target pose for the subject and one or more attributes of the subject to determine a plurality of updated second elements. Further, the system can render a representation of the subject based at least on the plurality of updated second elements.

Abstract: Apparatuses, systems, and techniques are presented to generate one or more images. In at least one embodiment, one or more neural networks are used to generate one or more images of one or more objects including one or more portions of one or more other objects.

Abstract: Systems and methods are disclosed for training and using a digital human foundational model (DHFM) comprising a generative adversarial network (GAN) generator. For instance, the method may include processing training inputs using the GAN generator to generate texel-aligned Gaussian maps that align Gaussian attributes to a coarse mesh template of the human and rendering a synthetic human representation of the human based on the texel-aligned Gaussian maps. The synthetic human representation comprises a full-bodied representation of the human indicating facial and hand features of the human. The method also includes processing the synthetic human representation using one or more discriminators to generate one or more discriminator outputs, computing one or more losses based on the texel-aligned Gaussian maps and the one or more discriminator outputs, and training the GAN generator using the one or more losses.

Abstract: Systems and methods are disclosed for generating and curating a training dataset for training one or more machine learning-artificial intelligence (ML-AI) models. For instance, the method may include extracting 2D landmarks of a human from an obtained image that is within the training dataset and extracting 3D poses of the human from the obtained image. The method may further include using camera coordinates associated with the obtained image to project the 3D poses of the human into 2D space and fine-tuning the 3D poses of the human based on comparing the projected 3D poses in 2D space with the extracted 2D landmarks. The method may also include generating labels for the obtained image within the training dataset, augmenting the training dataset with a plurality of generated synthetic images of humans, and training the one or more ML-AI models.

Abstract: Systems and methods are disclosed for training and using a digital human foundational model (DHFM) comprising a generative adversarial network (GAN) generator. For instance, the method may include obtaining one or more inputs comprising pose information indicating a three-dimensional (3D) pose representation of a human and processing the one or more inputs using a mapping network to generate intermediate latent code. The method may further include processing the intermediate latent code using the trained generator to generate texel-aligned Gaussian maps that align Gaussian attributes to a coarse mesh template of the human and performing linear blend skinning and deformation on the texel-aligned Gaussian maps to obtain modified texel-aligned Gaussian maps. The method may also include processing the modified texel-aligned Gaussian maps using a multi-part renderer to generate a synthetic human representation of the human indicating facial and hand features of the human.

Abstract: The disclosed method for training machine learning models for object generation includes performing, based on object data, one or more operations to train an untrained machine learning model to generate a trained machine learning model that comprises a trained encoder and a trained decoder, wherein the trained machine learning model is trained to generate an object surface representation, performing, based on the object data and natural language data, one or more operations to train an untrained diffusion model to generate a trained diffusion model, where the trained diffusion model is trained to generate an object geometry embedding, and where the trained diffusion model and the trained decoder are used to generate a virtual object based on natural language input.

Abstract: The disclosed method of generating an animatable representation of a character includes generating, using a trained diffusion model, one or more predicted target image latents and a diffusion timestep, generating, using a trained machine learning model and based on the diffusion timestep and the one or more predicted target image latents, a first global representation of the character at the diffusion timestep, determining, based on the first global representation of the character and the diffusion timestep, a second global representation of the character, and generating, based on the second global representation of the character, the animatable representation of the character.

Abstract: The disclosed method of generating an animatable representation of a character includes generating, based on a global representation of the character, one or more local views, generating, based on the global representation of the character and the one or more local views, one or more local ray maps, generating, using a trained diffusion model and a trained machine learning model and based on the one or more local views and the one or more local ray maps, one or more multi-part local views, and generating, based on the global representation of the character and the one or more multi-part local views, a refined representation of the character.

Abstract: The disclosed method for generating a virtual object includes processing a language embedding associated with a natural language description of an object using a trained diffusion model to generate a first object geometry embedding, processing the first object geometry embedding using a trained decoder to generate an object surface representation, and converting the object surface representation into a first object geometry of the virtual object.

Abstract: The disclosed method of training a machine learning model and a diffusion model includes generating, based on multi-camera video data, one or more first input views and one or more target views, the first input view(s) comprising a first input image of a first character and the first target view(s) comprising a first target image of the first character; and performing, based on the first input view(s) and the first target view(s), training operations to train an untrained diffusion model and an untrained machine learning model to generate a trained diffusion model and a trained machine learning model, the trained diffusion model being trained to generate one or more predicted target image latents and the trained machine learning model being trained to generate a global representation of the first character. An animatable representation of a second character is generated using the trained diffusion model and the trained machine learning model.

Abstract: Approaches presented herein provide for the use of reinforcement learning to fine-tune a generative model, such as a motion diffusion model, for a specific objective, such as to generate representations of human motion corresponding to provided text input. A discriminator can be used to guide the training of the generative model. In at least one embodiment, the discriminator can compare the input text and generated motion representation (or embeddings of each) to determine an alignment value or match score, for example, which can then be used to adjust the network parameters or weights of the generative model to improve the alignment between input text and generated motion.

Abstract: Estimating motion of a human or other object in video is a common computer task with applications in robotics, sports, mixed reality, etc. However, motion estimation becomes difficult when the camera capturing the video is moving, because the observed object and camera motions are entangled. The present disclosure provides for joint estimation of the motion of a camera and the motion of articulated objects captured in video by the camera.

Abstract: Apparatuses, systems, and techniques to identify orientations of objects within images. In at least one embodiment, one or more neural networks are trained to identify an orientations of one or more objects based, at least in part, on one or more characteristics of the object other than the object's orientation.