Abstract: A method and system are disclosed for reducing perplexity in a speech recognition system within a telephonic network based upon determined caller identity. In a speech recognition system which processes input frames of speech against stored templates representing speech, a core library of speech templates is created and stored representing a basic vocabulary of speech. Multiple caller-specific libraries of speech templates are also created and stored, each library containing speech templates which represent a specialized vocabulary and pronunciations for a specific geographic location and a particular individual. Additionally, the caller-specific libraries of speech templates are preferably processed to reflect the reduced bandwidth, transmission channel variations and other signal variations introduced into the system via a telephonic network.