Abstract: A method for controlling an artificial intelligence (AI) device can include receiving a set of reference images of a subject and encoding them into reference latents. The method estimates 3D morphable model (3DMM) parameters to derive pose and expression conditioning signals. A plurality of synthetic images are generated from different viewpoints using a morphable multi-view diffusion model. Generating the synthetic images includes executing an iterative reverse diffusion process on generated latents and applying stochastic conditioning by randomly sampling a subset of the reference latents and a subset of the generated latents to condition the model. The method further includes decoding the generated latents and training a 4D avatar model based on the reference and synthetic images. The 4D avatar model can utilize 3D Gaussian splatting augmented with expression-dependent appearance model information to enable real-time animation.

Abstract: A method for controlling a device can include receiving an input two-dimensional (2D) image, receiving a hybrid three-dimensional (3D) model including a first set of triangles forming a triangular mesh, and a second set of triangles with associated alpha map and neural feature maps, the vertices of both of the first and second sets of triangles including rigging information, and deforming the first and second sets of triangles of the hybrid 3D model based on 3D animation parameters and the rigging information, to generate deformed triangles. The method can further include rendering the deformed triangles based on rendering the first set of triangles using a texture mapping technique and rendering the second set of triangles using deferred neural rendering based on the neural feature maps and the alpha map to generate rendered triangles, and displaying an animated 3D object based on the rendered triangles and the input 2D image.

Abstract: A method for speech-driven three dimensional (3D) facial animation can include receiving an input speech audio signal, generating a speaker style vector from the input speech audio signal based on a speaker style embedding model, inputting the input speech audio signal and the speaker style vector into a mesh generation model and generating vertex position information for a 3D facial animation based on the input speech audio signal and the speaker style vector, and outputting the vertex position information.

Abstract: A method for controlling an artificial intelligence (AI) device can include receiving, by a processor, an input two dimensional (2D) image, encoding the 2D image to generate an image feature map, obtaining UV positional encoding information based on a three-dimensional (3D) face model where the UV positional encoding information includes a UV feature map, generating, by the processor, a correlation volume based on the image feature map and the UV feature map, generating flow map information and uncertainty information based on the correlation volume and the UV positional encoding information, and generating probabilistic 2D alignment information based on the flow map information and uncertainty information and outputting the probabilistic 2D alignment information. Also, the method can further include generating 3D reconstruction information based on the probabilistic 2D alignment information for animating a 3D face based on the input 2D image.

Abstract: A method for neural network driven vertex animation can include receiving, by an encoder component of a trained neural network, an input driving signal including audio data, processing, by the encoder component, the input driving signal to generate blendshape coefficient information based on the input driving signal, transmitting, by the encoder component, the blendshape coefficient information to a decoder component of the trained neural network, receiving, by the decoder component, the blendshape coefficient information from the encoder component, and generating, by the decoder component, vertex position information based on the blendshape coefficient information for animating a 3D model.

Abstract: A method for controlling an artificial intelligence (AI) device for implementing a digital avatar can include receiving an audio signal corresponding to a user query, converting, by a speech-to-text neural network model, the audio signal into a text query, inputting the text query into a large language gesture instruction model to generate high level movement instructions, inputting the text query and the high level movement instructions into an information retrieval model to generate a text response including at least one sentence and digital avatar control information, and inputting the text response into a text-to-speech neural network model to generate an audio response.