Abstract: Systems and methods of related to a voice-based system used to determine the severity of emotional distress within an audio recording of an individual is provided. In one non-limiting example, a system comprises a computing device that is configured to receive an audio sample that includes an utterance of a user. Feature extraction is performed on the audio sample to extract a plurality of acoustic emotion features using a base model. Emotion level predictions are generated for an emotion type based at least in part on the acoustic emotion features provided to an emotion specific model. An emotion classification for the audio sample is determined based on the emotion level predictions. The emotion classification comprises the emotion type and a level associated with the emotion type.