Abstract: One embodiment of a method for training a machine learning model to predict motion includes estimating, using an untrained machine learning model, a first motion based on at least one first condition; generating, using the untrained machine learning model, a second motion based on at least one second condition; computing a loss based on the first motion, the second motion, and one or more ground truth motions; and updating one or more parameters of the untrained machine learning model based on the loss to generate a trained machine learning model.

Abstract: One embodiment of a method for controlling a character includes receiving one or more conditions, based on the one or more conditions, generating or estimating a motion using a trained machine learning model, wherein the trained machine learning model is configured to estimate the motion via one denoising diffusion step or generate the motion via a plurality of denoising diffusion steps, and causing the character to move based on the motion.

Abstract: In various examples, systems and methods are disclosed relating to generating animatable characters or avatars. The system can assign a plurality of first elements of a three-dimensional (3D) model of a subject to a plurality of locations on a surface of the subject in an initial pose. Further, the system can assign a plurality of second elements to the plurality of first elements, each second element of the plurality of second elements having an opacity corresponding to a distance between the second element and the surface of the subject. Further, the system can update the plurality of second elements based at least on a target pose for the subject and one or more attributes of the subject to determine a plurality of updated second elements. Further, the system can render a representation of the subject based at least on the plurality of updated second elements.

Abstract: Apparatuses, systems, and techniques are presented to generate one or more images. In at least one embodiment, one or more neural networks are used to generate one or more images of one or more objects including one or more portions of one or more other objects.

Abstract: Systems and methods are disclosed for training and using a digital human foundational model (DHFM) comprising a generative adversarial network (GAN) generator. For instance, the method may include processing training inputs using the GAN generator to generate texel-aligned Gaussian maps that align Gaussian attributes to a coarse mesh template of the human and rendering a synthetic human representation of the human based on the texel-aligned Gaussian maps. The synthetic human representation comprises a full-bodied representation of the human indicating facial and hand features of the human. The method also includes processing the synthetic human representation using one or more discriminators to generate one or more discriminator outputs, computing one or more losses based on the texel-aligned Gaussian maps and the one or more discriminator outputs, and training the GAN generator using the one or more losses.

Abstract: Systems and methods are disclosed for generating and curating a training dataset for training one or more machine learning-artificial intelligence (ML-AI) models. For instance, the method may include extracting 2D landmarks of a human from an obtained image that is within the training dataset and extracting 3D poses of the human from the obtained image. The method may further include using camera coordinates associated with the obtained image to project the 3D poses of the human into 2D space and fine-tuning the 3D poses of the human based on comparing the projected 3D poses in 2D space with the extracted 2D landmarks. The method may also include generating labels for the obtained image within the training dataset, augmenting the training dataset with a plurality of generated synthetic images of humans, and training the one or more ML-AI models.

Abstract: Systems and methods are disclosed for training and using a digital human foundational model (DHFM) comprising a generative adversarial network (GAN) generator. For instance, the method may include obtaining one or more inputs comprising pose information indicating a three-dimensional (3D) pose representation of a human and processing the one or more inputs using a mapping network to generate intermediate latent code. The method may further include processing the intermediate latent code using the trained generator to generate texel-aligned Gaussian maps that align Gaussian attributes to a coarse mesh template of the human and performing linear blend skinning and deformation on the texel-aligned Gaussian maps to obtain modified texel-aligned Gaussian maps. The method may also include processing the modified texel-aligned Gaussian maps using a multi-part renderer to generate a synthetic human representation of the human indicating facial and hand features of the human.

Abstract: The disclosed method for training machine learning models for object generation includes performing, based on object data, one or more operations to train an untrained machine learning model to generate a trained machine learning model that comprises a trained encoder and a trained decoder, wherein the trained machine learning model is trained to generate an object surface representation, performing, based on the object data and natural language data, one or more operations to train an untrained diffusion model to generate a trained diffusion model, where the trained diffusion model is trained to generate an object geometry embedding, and where the trained diffusion model and the trained decoder are used to generate a virtual object based on natural language input.

Abstract: The disclosed method of generating an animatable representation of a character includes generating, using a trained diffusion model, one or more predicted target image latents and a diffusion timestep, generating, using a trained machine learning model and based on the diffusion timestep and the one or more predicted target image latents, a first global representation of the character at the diffusion timestep, determining, based on the first global representation of the character and the diffusion timestep, a second global representation of the character, and generating, based on the second global representation of the character, the animatable representation of the character.

Abstract: The disclosed method of generating an animatable representation of a character includes generating, based on a global representation of the character, one or more local views, generating, based on the global representation of the character and the one or more local views, one or more local ray maps, generating, using a trained diffusion model and a trained machine learning model and based on the one or more local views and the one or more local ray maps, one or more multi-part local views, and generating, based on the global representation of the character and the one or more multi-part local views, a refined representation of the character.

Abstract: The disclosed method for generating a virtual object includes processing a language embedding associated with a natural language description of an object using a trained diffusion model to generate a first object geometry embedding, processing the first object geometry embedding using a trained decoder to generate an object surface representation, and converting the object surface representation into a first object geometry of the virtual object.

Abstract: The disclosed method of training a machine learning model and a diffusion model includes generating, based on multi-camera video data, one or more first input views and one or more target views, the first input view(s) comprising a first input image of a first character and the first target view(s) comprising a first target image of the first character; and performing, based on the first input view(s) and the first target view(s), training operations to train an untrained diffusion model and an untrained machine learning model to generate a trained diffusion model and a trained machine learning model, the trained diffusion model being trained to generate one or more predicted target image latents and the trained machine learning model being trained to generate a global representation of the first character. An animatable representation of a second character is generated using the trained diffusion model and the trained machine learning model.

Abstract: Approaches presented herein provide for the use of reinforcement learning to fine-tune a generative model, such as a motion diffusion model, for a specific objective, such as to generate representations of human motion corresponding to provided text input. A discriminator can be used to guide the training of the generative model. In at least one embodiment, the discriminator can compare the input text and generated motion representation (or embeddings of each) to determine an alignment value or match score, for example, which can then be used to adjust the network parameters or weights of the generative model to improve the alignment between input text and generated motion.

Abstract: Estimating motion of a human or other object in video is a common computer task with applications in robotics, sports, mixed reality, etc. However, motion estimation becomes difficult when the camera capturing the video is moving, because the observed object and camera motions are entangled. The present disclosure provides for joint estimation of the motion of a camera and the motion of articulated objects captured in video by the camera.

Abstract: Apparatuses, systems, and techniques to identify orientations of objects within images. In at least one embodiment, one or more neural networks are trained to identify an orientations of one or more objects based, at least in part, on one or more characteristics of the object other than the object's orientation.

Abstract: Apparatuses, systems, and techniques for generating a clothed three-dimensional (3D) avatar character from a text prompt and enabling smooth animation through physics or neural simulators. In at least one embodiment, a clothed 3D avatar is generated through body layer modeling and garment layer modeling based on text descriptions. The outputs from the body layer and garment layer modeling are combined to generate an animation-ready, clothed 3D avatar.

Abstract: Systems and methods are disclosed that perform global human and camera motion estimation using a motion diffusion model that is attached to a control branch. For instance, using a controlled motion denoiser that comprises the motion diffusion model and the control branch, global human motions and the corresponding camera motions from “in-the-wild” videos may be estimated. Initially, SLAM may be used to initialize the camera motion and a pose estimation model may be used to estimate the local human motion. Combining the two, embodiments of the present disclosure initialize the global human motion. Then, during optimization and using a COIN system that includes the controlled motion denoiser and/or using a COIN algorithm, embodiments of the present disclosure enforce the global human and camera motion to satisfy a two-dimensional (2D) projection on videos and the motion distribution from the motion diffusion model.

Abstract: Estimating motion of a human or other object in video is a common computer task with applications in robotics, sports, mixed reality, etc. However, motion estimation becomes difficult when the camera capturing the video is moving, because the observed object and camera motions are entangled. The present disclosure provides for joint estimation of the motion of a camera and the motion of articulated objects captured in video by the camera.

Abstract: A method and system are provided for tracking instances within a sequence of video frames. The method includes the steps of processing an image frame by a backbone network to generate a set of feature maps, processing the set of feature maps by one or more prediction heads, and analyzing the embedding features corresponding to a set of instances in two or more image frames of the sequence of video frames to establish a one-to-one correlation between instances in different image frames. The one or more prediction heads includes an embedding head configured to generate a set of embedding features corresponding to one or more instances of an object identified in the image frame. The method may also include training the one or more prediction heads using a set of annotated image frames and/or a plurality of sequences of unlabeled video frames.

Abstract: Apparatuses, systems, and techniques to generate 3D models. In at least one embodiment, a 3D model, generated by a second neural network, is refined by a first neural network. In at least one embodiment, the first neural network is adjusted based on a determination made by the first neural network.