Abstract: An automatic speech recognition (ASR) system recognizes speech expressed in different languages. The ASR system includes a language-agnostic encoding component and prediction component. A language-specific joint analysis system generates first-language probabilities for symbols of a first language and second-language probabilities for symbols of a second language, based on outputs generated by the encoding component and the prediction component. The ASR system then modifies the probabilities produced by the joint analysis system by language-specific weighting information that, in turn, is produced by an attention system. This yields modified first-language probabilities and modified second-language probabilities. Finally, the ASR system predicts an updated instance of label information based on the modified first-language probabilities and the modified second-language probabilities. The ASR system can be successfully applied to recognize an utterance that combines words in two or more languages.