Abstract: Systems and methods are disclosed for training and using a digital human foundational model (DHFM) comprising a generative adversarial network (GAN) generator. For instance, the method may include processing training inputs using the GAN generator to generate texel-aligned Gaussian maps that align Gaussian attributes to a coarse mesh template of the human and rendering a synthetic human representation of the human based on the texel-aligned Gaussian maps. The synthetic human representation comprises a full-bodied representation of the human indicating facial and hand features of the human. The method also includes processing the synthetic human representation using one or more discriminators to generate one or more discriminator outputs, computing one or more losses based on the texel-aligned Gaussian maps and the one or more discriminator outputs, and training the GAN generator using the one or more losses.

Abstract: Systems and methods are disclosed for generating and curating a training dataset for training one or more machine learning-artificial intelligence (ML-AI) models. For instance, the method may include extracting 2D landmarks of a human from an obtained image that is within the training dataset and extracting 3D poses of the human from the obtained image. The method may further include using camera coordinates associated with the obtained image to project the 3D poses of the human into 2D space and fine-tuning the 3D poses of the human based on comparing the projected 3D poses in 2D space with the extracted 2D landmarks. The method may also include generating labels for the obtained image within the training dataset, augmenting the training dataset with a plurality of generated synthetic images of humans, and training the one or more ML-AI models.

Abstract: Systems and methods are disclosed for training and using a digital human foundational model (DHFM) comprising a generative adversarial network (GAN) generator. For instance, the method may include obtaining one or more inputs comprising pose information indicating a three-dimensional (3D) pose representation of a human and processing the one or more inputs using a mapping network to generate intermediate latent code. The method may further include processing the intermediate latent code using the trained generator to generate texel-aligned Gaussian maps that align Gaussian attributes to a coarse mesh template of the human and performing linear blend skinning and deformation on the texel-aligned Gaussian maps to obtain modified texel-aligned Gaussian maps. The method may also include processing the modified texel-aligned Gaussian maps using a multi-part renderer to generate a synthetic human representation of the human indicating facial and hand features of the human.

Abstract: The disclosed method for training machine learning models for object generation includes performing, based on object data, one or more operations to train an untrained machine learning model to generate a trained machine learning model that comprises a trained encoder and a trained decoder, wherein the trained machine learning model is trained to generate an object surface representation, performing, based on the object data and natural language data, one or more operations to train an untrained diffusion model to generate a trained diffusion model, where the trained diffusion model is trained to generate an object geometry embedding, and where the trained diffusion model and the trained decoder are used to generate a virtual object based on natural language input.

Abstract: The disclosed method of generating an animatable representation of a character includes generating, using a trained diffusion model, one or more predicted target image latents and a diffusion timestep, generating, using a trained machine learning model and based on the diffusion timestep and the one or more predicted target image latents, a first global representation of the character at the diffusion timestep, determining, based on the first global representation of the character and the diffusion timestep, a second global representation of the character, and generating, based on the second global representation of the character, the animatable representation of the character.

Abstract: The disclosed method of generating an animatable representation of a character includes generating, based on a global representation of the character, one or more local views, generating, based on the global representation of the character and the one or more local views, one or more local ray maps, generating, using a trained diffusion model and a trained machine learning model and based on the one or more local views and the one or more local ray maps, one or more multi-part local views, and generating, based on the global representation of the character and the one or more multi-part local views, a refined representation of the character.

Abstract: The disclosed method for generating a virtual object includes processing a language embedding associated with a natural language description of an object using a trained diffusion model to generate a first object geometry embedding, processing the first object geometry embedding using a trained decoder to generate an object surface representation, and converting the object surface representation into a first object geometry of the virtual object.

Abstract: The disclosed method of training a machine learning model and a diffusion model includes generating, based on multi-camera video data, one or more first input views and one or more target views, the first input view(s) comprising a first input image of a first character and the first target view(s) comprising a first target image of the first character; and performing, based on the first input view(s) and the first target view(s), training operations to train an untrained diffusion model and an untrained machine learning model to generate a trained diffusion model and a trained machine learning model, the trained diffusion model being trained to generate one or more predicted target image latents and the trained machine learning model being trained to generate a global representation of the first character. An animatable representation of a second character is generated using the trained diffusion model and the trained machine learning model.

Abstract: Approaches presented herein provide for the use of reinforcement learning to fine-tune a generative model, such as a motion diffusion model, for a specific objective, such as to generate representations of human motion corresponding to provided text input. A discriminator can be used to guide the training of the generative model. In at least one embodiment, the discriminator can compare the input text and generated motion representation (or embeddings of each) to determine an alignment value or match score, for example, which can then be used to adjust the network parameters or weights of the generative model to improve the alignment between input text and generated motion.

Abstract: Estimating motion of a human or other object in video is a common computer task with applications in robotics, sports, mixed reality, etc. However, motion estimation becomes difficult when the camera capturing the video is moving, because the observed object and camera motions are entangled. The present disclosure provides for joint estimation of the motion of a camera and the motion of articulated objects captured in video by the camera.

Abstract: Apparatuses, systems, and techniques to identify orientations of objects within images. In at least one embodiment, one or more neural networks are trained to identify an orientations of one or more objects based, at least in part, on one or more characteristics of the object other than the object's orientation.

Abstract: Apparatuses, systems, and techniques for generating a clothed three-dimensional (3D) avatar character from a text prompt and enabling smooth animation through physics or neural simulators. In at least one embodiment, a clothed 3D avatar is generated through body layer modeling and garment layer modeling based on text descriptions. The outputs from the body layer and garment layer modeling are combined to generate an animation-ready, clothed 3D avatar.

Abstract: Systems and methods are disclosed that perform global human and camera motion estimation using a motion diffusion model that is attached to a control branch. For instance, using a controlled motion denoiser that comprises the motion diffusion model and the control branch, global human motions and the corresponding camera motions from “in-the-wild” videos may be estimated. Initially, SLAM may be used to initialize the camera motion and a pose estimation model may be used to estimate the local human motion. Combining the two, embodiments of the present disclosure initialize the global human motion. Then, during optimization and using a COIN system that includes the controlled motion denoiser and/or using a COIN algorithm, embodiments of the present disclosure enforce the global human and camera motion to satisfy a two-dimensional (2D) projection on videos and the motion distribution from the motion diffusion model.

Abstract: Estimating motion of a human or other object in video is a common computer task with applications in robotics, sports, mixed reality, etc. However, motion estimation becomes difficult when the camera capturing the video is moving, because the observed object and camera motions are entangled. The present disclosure provides for joint estimation of the motion of a camera and the motion of articulated objects captured in video by the camera.

Abstract: A method and system are provided for tracking instances within a sequence of video frames. The method includes the steps of processing an image frame by a backbone network to generate a set of feature maps, processing the set of feature maps by one or more prediction heads, and analyzing the embedding features corresponding to a set of instances in two or more image frames of the sequence of video frames to establish a one-to-one correlation between instances in different image frames. The one or more prediction heads includes an embedding head configured to generate a set of embedding features corresponding to one or more instances of an object identified in the image frame. The method may also include training the one or more prediction heads using a set of annotated image frames and/or a plurality of sequences of unlabeled video frames.

Abstract: Apparatuses, systems, and techniques to generate 3D models. In at least one embodiment, a 3D model, generated by a second neural network, is refined by a first neural network. In at least one embodiment, the first neural network is adjusted based on a determination made by the first neural network.

Abstract: In various examples, systems and methods are disclosed relating to receive at least one of a text prompt or a kinematic constraint and determine first human motion data using a motion model by applying the at least one of the text prompt or the kinematic constraint to the motion model. The motion model is updated by generating, using the motion model, second human motion data by applying motion capture (mocap) data and video reconstruction data as inputs to the motion model, receiving user feedback information for the second human motion data, and updating the motion model based on the user feedback information. The video reconstruction data is generated by reconstructing human motions from a plurality of videos. Physically implausible artifacts are filtered from the video reconstruction data using a motion imitation controller. The motion imitation controller is updated using at least one of Reinforced Learning (RL) or physics-based character simulations.

Abstract: Systems and methods are disclosed relating to receiving at least one of a text prompt or a kinematic constraint, generating, by a motion model including a first model and a second model, human motion data of a human character by applying a random noise and the at least one of the text prompt or the kinematic constraint into the motion model. Generating the human motion data includes, for each iteration of diffusion determining, using the first model, global root motion by applying noisy global root motion and noisy local joint motion as inputs into the first model and determining, using the second model, local joint motion by applying the noisy local joint motion and local root motion as inputs into the second model. The local root motion is determined based on the global root motion. The human motion data includes the local joint motion and the global root motion.

Abstract: In various examples, a timeline of text prompt(s) specifying any number of (e.g., sequential and/or simultaneous) actions may be specified or generated, and the timeline may be used to drive a diffusion model to generate compositional human motion that implements the arrangement of action(s) specified by the timeline. For example, at each denoising step, a pre-trained motion diffusion model may be used to denoise a motion segment corresponding to each text prompt independently of the others, and the resulting denoised motion segments may be temporally stitched, and/or spatially stitched based on body part labels associated with each text prompt. As such, the techniques described herein may be used to synthesize realistic motion that accurately reflects the semantics and timing of the text prompt(s) specified in the timeline.

Abstract: The present document describes a pharmaceutical composition comprising a) a lipid nanoparticle operable to encapsulate a therapeutic agent, comprising a core and an external surface, said therapeutic agent being encapsulated within said core; said lipid nanoparticle having a size of said lipid nanoparticle of from about 30 to about 80 nm, or a pegylated lipid comprising a distearoyl-rac-glycerol (DSG)-PEG and 1,2-distearoyl-sn-glycero-3-phosphoethanolamine-N-(DSPE)-PEG-DBCO; or a combination of: a size of from about 30 to about 80 nm and a pegylated lipid comprising a DSG-PEG and DSPE-PEG-DBCO; and b) an antibody or antigen-binding fragment thereof operable to transmigrate the blood-brain barrier (BBB), wherein the antibody or antigen-binding fragment thereof comprises complementarity determining regions (CDR1, CDR2 and CDR3), operably linked to said external surface of said lipid nanoparticle.