Abstract: This specification provides a computer-implemented method for generating facial animation data. The facial animation data animates a face in a video game in accordance with speech sounds and speech emotion of speech audio. Speech audio data representing the speech audio is processed using a machine-learned speech emotion recognition model. One or more emotion representations that represent the emotion of the speech audio are generated. For each of the one or more emotion representations, a respective conditioning input for a facial animation generative model is determined. Facial animation data for the speech audio data is generated, comprising processing, using the facial animation generative model: (i) input data derived from the speech audio data, and (ii) the one or more conditioning inputs.

Abstract: This specification provides a system comprising: or more computing devices; and one or more storage devices communicatively coupled to the one or more computing devices. The one or more storage devices store instructions that, when executed by the one or more computing devices, cause the one or more computing devices to perform operations comprising: receiving input data derived from speech audio; generating facial animation data, comprising processing the input data and a conditioning input using a machine-learned generative model; generating further animation data, comprising processing the input data using a further machine-learned generative model; and generating animation data for at least a face in a video game using the facial animation data and the further animation data, wherein the animation data animates at least the face in the video game in accordance with speech sounds of the speech audio.