Abstract: A method for modifying an audio visual recording originally produced with an original audio track of an original speaker, using a second audio dub track of a second speaker, comprising analyzing the original audio track to convert it into a continuous time coded facial and acoustic symbol stream to identify corresponding visual facial motions of the original speaker to create continuous data sets of facial motion corresponding to speech utterance states and transformations, storing these continuous data sets in a database, analyzing the second audio dub track to convert it to a continuous tune coded facial and acoustic symbol stream, using the second audio dub track's continuous time coded facial and acoustic symbol stream to animate the original speaker's face, synchronized to the second audio dub track to create natural continuous facial speech expression by the original speaker of the second dub audio track.

Abstract: This invention comprises analyzing a speaker's speech in an audio visual recording to convert it into triphones and/or phonemes and then using a time coded phoneme stream to identify corresponding visual facial motions, to create single frame snapshots or multi-frame clips of facial motion corresponding to speech phoneme utterance states and transformations, which are stored in a database, and which are subsequently used to animate the original speaker's face, synchronized to a new voice track that has been converted into a time-coded, image frame-indexed phoneme stream.