Abstract: Systems and methods of the present disclosure include animating virtual avatars or agents according to input audio and one or more selected or determined emotions and/or styles. For example, a deep neural network can be trained to output motion or deformation information for a character that is representative of the character uttering speech contained in audio input. The character can have different facial components or regions (e.g., head, skin, eyes, tongue) modeled separately, such that the network can output motion or deformation information for each of these different facial components. During training, the network can use a transformer-based audio encoder with locked parameters to train an associated decoder using a weighted feature vector. The network output can be provided to a renderer to generate audio-driven facial animation that is emotion-accurate.

Abstract: In various examples, determining emotion sequences for speech in conversational AI systems and applications is described herein. Systems and methods are disclosed that use one or more first machine learning models to determine a sequence of emotional states associated with audio data representing speech. To use the first machine learning model(s), the systems and methods may train the first machine learning model(s) using one or more second machine learning models, where the second machine learning model(s) is trained to determine scores indicating accuracies associated with sequences of emotional states. For instance, the second machine learning model(s) may be trained to determine the scores using audio data representing speech, sequences of emotional states associated with the speech, and indications of which sequences of emotional states better represent the speech as compared to other sequences of emotional states.

Abstract: In various examples, a technique for audio-driven facial animation with adaptive speech includes determining that a rate of speech associated with an audio segment exceeds a threshold. The technique also includes based at least on the rate of speech exceeding the threshold, upsampling a first set of features associated with the audio segment into a second set of features that is different in size than the first set of features. The technique further includes generating, using one or more machine learning models and based at least on at least a subset of the second set of features, a facial animation output corresponding to the audio segment.

Abstract: In various examples, techniques are described for animating characters by decoupling portions of a face from other portions of the face. Systems and methods are disclosed that use one or more neural networks to generate high-fidelity facial animation using inputted audio data. In order to generate the high-fidelity facial animations, the systems and methods may decouple effects of implicit emotional states from effects of audio on the facial animations during training of the neural network(s). For instance, the training may cause the audio to drive the lower face animations while the implicit emotional states drive the upper face animations. In some examples, in order to encourage more expressive expressions, adversarial training is further used to learn a discriminator that predicts if generated emotional states are from real distribution.

Abstract: Approaches in accordance with various embodiments provide for fluid simulation with substantially reduced time and memory requirements with respect to conventional approaches. In particular, various embodiments can perform time and energy efficient, large scale fluid simulation on processing hardware using a method that does not solve for the Navier-Stokes equations to enforce incompressibility. Instead, various embodiments generate a density tensor and rigid body map tensor for a large number of particles contained in a sub-domain. Collectively, the density tensor and rigid body map may represent input channels of a network with three spatial-dimensions. The network may apply a series of operations to the input channels to predict an updated position and updated velocity for each particle at the end of a frame. Such approaches can handle tens of millions of particles within a virtually unbounded simulation domain, as compared to classical approaches that solve for the Navier-Stokes equations.

Abstract: Approaches in accordance with various embodiments provide for fluid simulation with substantially reduced time and memory requirements with respect to conventional approaches. In particular, various embodiments can perform time and energy efficient, large scale fluid simulation on processing hardware using a method that does not solve for the Navier-Stokes equations to enforce incompressibility. Instead, various embodiments generate a density tensor and rigid body map tensor for a large number of particles contained in a sub-domain. Collectively, the density tensor and rigid body map may represent input channels of a network with three spatial-dimensions. The network may apply a series of operations to the input channels to predict an updated position and updated velocity for each particle at the end of a frame. Such approaches can handle tens of millions of particles within a virtually unbounded simulation domain, as compared to classical approaches that solve for the Navier-Stokes equations.

Abstract: Approaches in accordance with various embodiments provide for fluid simulation with substantially reduced time and memory requirements with respect to conventional approaches. In particular, various embodiments can perform time and energy efficient, large scale fluid simulation on processing hardware using a method that does not solve for the Navier-Stokes equations to enforce incompressibility. Instead, various embodiments generate a density tensor and rigid body map tensor for a large number of particles contained in a sub-domain. Collectively, the density tensor and rigid body map may represent input channels of a network with three spatial-dimensions. The network may apply a series of operations to the input channels to predict an updated position and updated velocity for each particle at the end of a frame. Such approaches can handle tens of millions of particles within a virtually unbounded simulation domain, as compared to classical approaches that solve for the Navier-Stokes equations.

Abstract: Approaches in accordance with various embodiments provide for fluid simulation with substantially reduced time and memory requirements with respect to conventional approaches. In particular, various embodiments can perform time and energy efficient, large scale fluid simulation on processing hardware using a method that does not solve for the Navier-Stokes equations to enforce incompressibility. Instead, various embodiments generate a density tensor and rigid body map tensor for a large number of particles contained in a sub-domain. Collectively, the density tensor and rigid body map may represent input channels of a network with three spatial-dimensions. The network may apply a series of operations to the input channels to predict an updated position and updated velocity for each particle at the end of a frame. Such approaches can handle tens of millions of particles within a virtually unbounded simulation domain, as compared to classical approaches that solve for the Navier-Stokes equations.