Abstract: The various embodiments herein provide a system and method to track the pitch of a human being in real time using time varying model. According to one embodiment, the input voice is synthesised to obtain a lower order model. The lower model is down sampled and fitted to a time varying 2nd order model. The down sampled signal is passed through a pitch tracking filter, a fading filter and a gradient filter to obtain a pitch signal in real time. The noise included in the pitch signal is removed by passing the acquired pitch signal through a Kalman filter to obtain a smoothened pitch signal in real time.