Abstract: Traditional speaker verification systems are vulnerable to voice spoofing attacks, such as voice-replay attack, voice-cloning attack, and cloned-replay attack. To overcome these vulnerabilities, a secure automatic speaker verification system based on a novel sign modified acoustic local ternary pattern (sm-ALTP) features and asymmetric bagging-based classifier-ensemble with enhanced attack vector is presented. The proposed audio representation approach clusters the high and low frequency components in audio frames by normally distributing them against a convex function. Afterwards, the neighborhood statistics are applied to capture the user specific vocal tract information.