Abstract: A computer system implements an artificial intelligence (AI) based assessment engine. In a video assessment process, the computer system receives video input including video of a human learner; extracts video features from the video input using tasks such as action detection, emotion detection, role identification, posture detection, head pose detection, person detection, or person identification. In an audio assessment process, the computer system receives audio input; feeds the audio input to a context-aware NLP processing engine; and extracts features from the audio input such as fluency score, pronunciation score, grammar score, coherence score, vocabulary score, sentiment score, or a combination thereof. The computer system obtains one or more automated scores from an AI scoring engine based on the extracted features and a scoring rubric previously learned by the AI scoring engine.