Abstract: A method of generating an image for use in a conversation taking place in a messaging application is disclosed. Conversation input text is received from a user of a portable device that includes a display. Model input text is generated from the conversation input text, which is processed with a text-to-image model to generate an image based on the model input text. The coordinates of a face in the image are determined, and the face of the user or another person is added to the image at the location. The final image is displayed on the portable device, and user input is received to transmit the image to a remote recipient.

Abstract: Provided are systems and methods for face reenactment. An example method includes receiving a target video that includes at least one target frame, where the at least one target frame includes a target face, receiving a scenario including a series of source facial expressions, determining, based on the target face, a target facial expression of the target face, synthesizing, based on a parametric face model and a texture model, an output face including the target face, where the target facial expression of the target face is modified to imitate a source facial expression of the series of source facial expressions, and generating, based on the output face, a frame of an output video. The parametric face model includes a template mesh pre-generated based on historical images of faces of a plurality of individuals, where the template mesh includes a pre-determined number of vertices.

Abstract: Provided are systems and methods for realistic head turns and face animation synthesis. An example method includes receiving a source frame of a source video, where the source frame includes a head and a face of a source actor, generating source pose parameters corresponding to a pose of the head and a facial expression of the source actor; receiving a target image including a target head and a target face of a target person, determining target identity information associated with the target head and the target face of the target person, replacing source identity information in the source pose parameters with the target identity information to obtain further source pose parameters, and generating an output frame of an output video that includes a modified image of the target face and the target head adopting the pose of the head and the facial expression of the source actor.

Abstract: A method of generating an image for use in a conversation taking place in a messaging application is disclosed. Conversation input text is received from a user of a portable device that includes a display. Model input text is generated from the conversation input text, which is processed with a text-to-image model to generate an image based on the model input text. The generated image is displayed on the portable device, and user input is received to transmit the image to a remote recipient.

Abstract: Provided are systems and methods for face reenactment. An example method includes receiving visual data including a visible portion of a source face, determining, based on the visible portion of the source face, a first portion of source face parameters associated with a parametric face model, where the first portion corresponds to the visible portion, predicting, based partially on the visible portion of the source face, a second portion of the source face parameters, where the second portion corresponds to the rest of the source face, receiving a target video that includes a target face, determining, based on the target video, target face parameters associated with the parametric face model and corresponding to the target face, and synthesizing, using the parametric face model, based on the source face parameters and the target face parameters, an output face that includes the source face imitating a facial expression of the target face.

Abstract: Systems and methods for text and audio-based real-time face reenactment are provided. An example method includes receiving an input text and a target image, where the target image including a target face, generating, based on the input text, a sequence of sets of acoustic features corresponding to the input text, generating, based on the sequence of sets of acoustic features, a sequence of sets of mouth key points, generating, based on the sequence of sets of mouth key points, a sequence of sets of facial key points, determining, based on the set sequence of sets of facial key points, a sequence of deformations of the target face, and applying the sequence of deformations to the target image, thereby generating a sequence of frames of an output video.

Abstract: Provided are systems and methods for text and audio-based real-time face reenactment. An example method includes receiving an input text and a target image, the target image including a target face; generating, based on the input text, a sequence of sets of acoustic features representing the input text; generating, based on the sequence of sets of acoustic features, a sequence of sets of mouth key points; generating, based on the sequence of sets of mouth key points, a sequence of sets of facial key points; generating, by the computing device and based on the sequence of sets of the facial key points and the target image, a sequence of frames; and generating, based on the sequence of frames, an output video. Each of the frames includes the target face modified based on at least one set of mouth key points of the sequence of sets of mouth key points.

Abstract: Provided are systems and methods for portrait animation. An example method includes receiving, by a computing device, a scenario video, where the scenario video includes information concerning a first face, receiving, by the computing device, a target image, where the target image includes a second face, determining, by the computing device and based on the target image and the information concerning the first face, two-dimensional (2D) deformations of the second face in the target image, and applying, by the computing device, the 2D deformations to the target image to obtain at least one output frame of an output video.

Abstract: Disclosed are systems and methods for portrait animation. An example method includes receiving, by a computing device, a scenario video, where the scenario video includes at least one input frame and the at least one input frame includes a first face, receiving, by the computing device, a target image, where the target image includes a second face, determining, by the computing device and based on the at least one input frame and the target image, two-dimensional (2D) deformations of the second face in the target image, where the 2D deformations, when applied to the second face, modify the second face to imitate at least a facial expression of the first face, and applying, by the computing device, the 2D deformations to the target image to obtain at least one output frame of an output video.

Abstract: Provided are systems and methods for realistic head turns and face animation synthesis. An example method includes receiving a source frame of a source video, where the source frame includes a head and a face of a source actor, generating source pose parameters corresponding to a pose of the head and a facial expression of the source actor; receiving a target image including a target head and a target face of a target person, determining target identity information associated with the target head and the target face of the target person, replacing source identity information in the source pose parameters with the target identity information to obtain further source pose parameters, and generating an output frame of an output video that includes a modified image of the target face and the target head adopting the pose of the head and the facial expression of the source actor.

Abstract: Provided are systems and methods for face reenactment. An example method includes receiving visual data including a visible portion of a source face, determining, based on the visible portion of the source face, a first portion of source face parameters associated with a parametric face model, where the first portion corresponds to the visible portion, predicting, based partially on the visible portion of the source face, a second portion of the source face parameters, where the second portion corresponds to the rest of the source face, receiving a target video that includes a target face, determining, based on the target video, target face parameters associated with the parametric face model and corresponding to the target face, and synthesizing, using the parametric face model, based on the source face parameters and the target face parameters, an output face that includes the source face imitating a facial expression of the target face.

Abstract: The subject technology receives frames of a source media content, the frames of the source media content including representations of a head and a face of a source actor. The subject technology generates, based at least in part on the frames of the source media content, sets of source pose parameters. The subject technology receives at least one target image, the at least one target image including representations of a target head and a target face of a target entity. The subject technology provides the sets of source pose parameters to a neural network to determine facial landmarks for head turns and facial expressions. The subject technology generates, based at least in part on the sets of source pose parameters and the facial landmarks for head turns and facial expressions, an output media content. The subject technology provides augmented reality content based at least in part on the output media content for display on a computing device.

Abstract: Disclosed are systems and methods for portrait animation. An example method includes receiving, by a computing device, a scenario video, where the scenario video includes at least one input frame and the at least one input frame includes a first face, receiving, by the computing device, a target image, where the target image includes a second face, determining, by the computing device and based on the at least one input frame and the target image, two-dimensional (2D) deformations of the second face in the target image, where the 2D deformations, when applied to the second face, modify the second face to imitate at least a facial expression of the first face, and applying, by the computing device, the 2D deformations to the target image to obtain at least one output frame of an output video.

Abstract: A methodology for training a machine learning model to generate color-neutral input face images is described. For each training face image from a training dataset that is used for training the model, the training system generates an input face image, which has the color and lighting of a randomly selected image from the set of color source images, and which has facial features and expression of a face object from the training face image. Because, during training, the machine learning model is “confused” by changing the color and lighting of a training face image to a randomly selected different color and lighting, the trained machine learning model generates a color neutral embedding representing facial features from the training face image.

Abstract: Provided are systems and methods for realistic head turns and face animation synthesis. An example method may include receiving frames of a source video with the head and the face of a source actor. The method may then proceed with generating sets of source pose parameters that represent positions of the head and facial expressions of the source actor. The method may further include receiving at least one target image including the target head and the target face of a target person, determining target identity information associated with the target face, and generating an output video based on the target identity information and the sets of source pose parameters. Each frame of the output video can include an image of the target face modified to mimic at least one of the positions of the head of the source actor and at least one of facial expressions of the source actor.

Abstract: Provided are systems and a method for photorealistic real-time face reenactment. An example method includes receiving a target video including a target face and a scenario including a series of source facial expressions, determining, based on the target face, one or more target facial expressions, and synthesizing, using the parametric face model, an output face. The output face includes the target face. The one or more target facial expressions are modified to imitate the source facial expressions. The method further includes generating, based on a deep neural network, a mouth region and an eyes region, and combining the output face, the mouth region, and the eyes region to generate a frame of an output video.

Abstract: Provided are systems and methods for photorealistic real-time portrait animation. An example method includes receiving a scenario video with at least one input frame. The input frame includes a first face of a first person. The method further includes receiving a target image with a second face of a second person. The method further includes determining, based on the at least one input frame and the target image, two-dimensional (2D) deformations of the second face and a background in the target image. The 2D deformations, when applied to the second face, modify the second face to imitate at least a facial expression and a head orientation of the first face. The method further includes applying the 2D deformations to the target image to obtain at least one output frame of an output video.

Abstract: In some embodiments, users' experience of engaging with augmented reality technology is enhanced by providing a process, referred to as face animation synthesis, that replaces an actor's face in the frames of a video with a user's face from the user's portrait image. The resulting face in the frames of the video retains the facial expressions, as well as color and lighting, of the actor's face but, at the same time, has the likeness of the user's face. An example face animation synthesis experience can be made available to uses of a messaging system by providing a face animation synthesis augmented reality component.

Abstract: In some embodiments, users' experience of engaging with augmented reality technology is enhanced by providing a process, referred to as face animation synthesis, that replaces an actor's face in the frames of a video with a user's face from the user's portrait image. The resulting face in the frames of the video retains the facial expressions, as well as color and lighting, of the actor's face but, at the same time, has the likeness of the user's face. An example face animation synthesis experience can be made available to uses of a messaging system by providing a face animation synthesis augmented reality component.

Abstract: Provided are systems and methods for text and audio-based real-time face reenactment. An example method includes receiving an input text and a target image, the target image including a target face; generating, based on the input text, a sequence of sets of acoustic features representing the input text; generating, based on the sequence of sets of acoustic features, a sequence of sets of mouth key points; generating, based on the sequence of sets of mouth key points, a sequence of sets of facial key points; generating, by the computing device and based on the sequence of sets of the facial key points and the target image, a sequence of frames; and generating, based on the sequence of frames, an output video. Each of the frames includes the target face modified based on at least one set of mouth key points of the sequence of sets of mouth key points.