Abstract: Implementations relate to a multimodal translation application that can provide an abridged version of a translation through an audio interface of a computing device, while simultaneously providing a verbatim textual translation at a display interface of the computing device. The application can provide these different versions of the translation in certain circumstances when, for example, the rate of speech of a person speaking to a user is relatively high compared to a preferred rate of speech of the user. For example, a comparison between phonemes of an original language speech and a translated language speech can be performed to determine whether the ratio satisfies a threshold for providing an audible abridged translation. A determination to provide the abridged translation can additionally or alternatively be based on a determined language of the speaker.

Abstract: A computer generated voice can automatically be adapted to be similar to a user's voice. Various implementations include processing audio data capturing a first language spoken utterance to identify one or more pitch characteristics. For example, the one or more pitch characteristics can include an estimated frequency range of the given user's voice. Additionally or alternatively, the system can process the audio data capturing the first language spoken utterance and a set of candidate computer generated voices using a computer generated voice selection model to select a candidate computer generated voice. Various implementations can include automatically modifying the selected candidate computer generated voice based on the one or more pitch characteristics to change the frequency range of the modified computer generated voice based on the user's voice.

Abstract: This document describes methods and systems of on-device real-time translation for media content on a mobile electronic device. The translation is managed and executed by an operating system of the electronic device rather than within a particular application executing on the electronic device. The operating system can translate media content, including visual content displayed on a display device of the electronic device or audio content output by the electronic device. Because the translation is at the OS level, the translation can be implemented, automatically or based on a user input, across a variety of (including all) applications and a variety of content on the electronic device to provide a consistent translation experience, which is provided via a system UI overlay that displays translated text as captions to video content or as a replacement to on-screen text.