Abstract: Representation learning for text and speech has improved many language-related tasks. However, existing methods only learn from one input modality, while a unified representation for both speech and text is needed for tasks such as end-to-end speech translation. Consequently, these methods cannot exploit various large-scale text and speech data and their performance is limited by the scarcity of parallel speech translation data. To address these problems, embodiments of a fused acoustic and text masked language model (FAT-MLM) are disclosed. FAT-MLM embodiments jointly learn a unified representation for both acoustic and text input from various types of corpora including parallel data for speech recognition and machine translation, and pure speech and text data. Within this cross-modal representation learning framework, an end-to-end model is further presented for fused acoustic and text speech translation.

Abstract: Presented herein are embodiments of a prefix-to-prefix framework for simultaneous translation that implicitly learns to anticipates in a single translation. Within these frameworks are effective “wait-k” policy model embodiments that may be trained to generate a target sentence concurrently with a source sentence but lag behind by a predefined number of words. Embodiments of the prefix-to-prefix framework achieve low latency and better quality when compared to full-sentence translation in four directions: Chinese?English and German?English. Also presented herein is a novel latency metric that addresses deficiencies of previous latency metrics.

Abstract: Presented herein are embodiments of a prefix-to-prefix framework for simultaneous translation that implicitly learns to anticipates in a single translation. Within these frameworks are effective “wait-k” policy model embodiments that may be trained to generate a target sentence concurrently with a source sentence but lag behind by a predefined number of words. Embodiments of the prefix-to-prefix framework achieve low latency and better quality when compared to full-sentence translation in four directions: Chinese?English and German?English. Also presented herein is a novel latency metric that addresses deficiencies of previous latency metrics.