Abstract: A system and method for translating audio, and video when desired. The translations include synthetic media and data generated using AI systems. Through unique processors and generators executing a unique sequence of steps, the system and method produces more accurate translations that can account for various speech characteristics (e.g., emotion, pacing, idioms, sarcasm, jokes, tone, phonemes, etc.). These speech characteristics are identified in the input media and synthetically incorporated into the translated outputs to mirror the characteristics in the input media. Some embodiments further include systems and methods that manipulate the input video such that the speakers' faces and/or lips appear as if they are natively speaking the generated audio.

Abstract: A system and method of processing media for facial manipulation. Input frames are downscaled to a lower resolution and facial detection identifies a target face in each input frame. The location of the target face is determined in the downscaled frames and the locations are projected to the input frames based on the pixel differences between the frames. Facial landmark detection is then performed on a cropped image from the original input frame. The facial landmarks are used to adjust the orientation of the input frames so that the target face in each input frame is in a standard orientation. Facial landmark detection is again performed on the target face in each frame while in the standard orientation to produce more accurate landmarks. Facial manipulation can then be executed based on the landmarks in the aligned input images. The orientation of the images with the manipulated faces are then reverted.

Abstract: An audio-driven lip reanimation GAN network and method of reanimating lips based on an input audio using a GAN network. The GAN network includes a 1st stage GAN generator configured to receive 1st stage audio inputs and 1st stage input frames. The 1st stage GAN generator is trained to produce 1st stage synthetic output frames in which a pair of lips in a target face in the 1st stage synthetic output frames has been reanimated in reference to the 1st stage input frames based on the 1st stage audio inputs. The GAN network also includes a 2nd stage GAN generator configured to receive the 1st stage synthetic output frames as inputs. The 2nd stage GAN generator is trained to generate 2nd stage output frames that improve on the realism of the 1st stage output frames.

Abstract: An object trajectory analysis system and method that is usable to determine a probable match between objects across subsequent or related input frames of an input media, thereby tracking a movement of an object, such as a target face, across the subsequent or related input frames. The system receives an input media including a plurality of input frames for analysis, such that the target face is determined to be present through facial landmark analyses and positional analyses. Calculating an overlap value between bounding boxes that are associated with detected target faces in subsequent frames, the system determines whether a consistent face is present within the subsequent frames. The system then performs the analyses across multiple frames to track a relative position of the target face, and can use landmark detection to predict a presence of and characteristics of the target face during intermediary frames lacking detection of the target face.

Abstract: A system and method for translating audio, and video when desired. The translations include synthetic media and data generated using AI systems. Through unique processors and generators executing a unique sequence of steps, the system and method produces more accurate translations that can account for various speech characteristics (e.g., emotion, pacing, idioms, sarcasm, jokes, tone, phonemes, etc.). These speech characteristics are identified in the input media and synthetically incorporated into the translated outputs to mirror the characteristics in the input media. Some embodiments further include systems and methods that manipulate the input video such that the speakers' faces and/or lips appear as if they are natively speaking the generated audio.

Abstract: A system and method for translating audio, and video when desired. The translations include synthetic media and data generated using AI systems. Through unique processors and generators executing a unique sequence of steps, the system and method produces more accurate translations that can account for various speech characteristics (e.g., emotion, pacing, idioms, sarcasm, jokes, tone, phonemes, etc.). These speech characteristics are identified in the input media and synthetically incorporated into the translated outputs to mirror the characteristics in the input media. Some embodiments further include systems and methods that manipulate the input video such that the speakers' faces and/or lips appear as if they are natively speaking the generated audio.