Abstract: An electronic apparatus and method for visual speech recognition based on connectionist temporal classification (CTC) loss is disclosed. The electronic apparatus receives a video that includes human speakers and generates a prediction corresponding to lip movements of the human speakers. The prediction is generated based on application of a Deep Neural Network (DNN) on the video and the DNN is trained using a CTC loss function. The electronic apparatus detects, based on the prediction, word boundaries in a sequence of characters that correspond to the lip movements and divides the video into a sequence of video clips based on the detection. Each video clip corresponds to a word spoken by the human speakers. The electronic apparatus generates a sequence of word predictions by processing the sequence of video clips and generates a sentence, or a phrase based on the generated sequence of word predictions.

Abstract: Implementations generally relate to correcting lip-reading predictions. In some implementations, a method includes receiving video input of a user, where the user is talking in the video input. The method further includes predicting one or more words from mouth movement of the user to provide one or more predicted words. The method further includes correcting one or more correction candidate words from the one or more predicted words. The method further includes predicting one or more sentences from the one or more predicted words.