Abstract: An end-to-end automatic speech recognition (ASR) system can be constructed by fusing a first ASR model with a transformer. The input of the transformer is a learned layer generated by the first ASR model. The fused ASR model and transformer can be treated as a single end-to-end model and trained as a single model. In some embodiments, the end-to-end speech recognition system can be trained using a teacher-student training technique by selectively truncating portions of the first ASR model and/or the transformer components and selectively freezing various layers during the training passes.