Abstract: A natural language understanding (NLU) framework includes a lookup source framework that enables the lookup sources to be created and applied to understanding utterances. Each lookup source is associated with a respective lookup source template that defines the compile-time and inference-time behavior of the lookup source. For example, a lookup source template indicates which plugins are used by the lookup source, and may define property values that determine the operational behavior of each of these plugins during compile-time and/or inference-time operation of the lookup source. The lookup source framework includes a template manager that manages lookup source templates and determines a suitable lookup source template for each lookup source.

Abstract: A natural language understanding (NLU) framework includes a lookup source framework, which enables a lookup source system to be defined having one or more lookup sources. Each lookup source of the lookup source system includes a respective source data representation that is compiled from respective source data. For example, a source data representation may include source data arranged in a finite state transducer (IFST) structure as a set of finite-state automata (FSA) states, wherein each state is associated with a token that represents underlying source data. Different producers can be applied during compilation of a source data representation to derive additional states within the source data representation from the source data. Certain states of the source data representation that contain sensitive data can be selectively protected through encryption and/or obfuscation, while other portions of the source data representation that are not sensitive may remain in clear-text form.

Abstract: A natural language understanding (NLU) framework includes a lookup source framework, which enables a lookup source system to be defined having one or more lookup sources. In particular, taxonomy lookup sources can be compiled from suitable taxonomy source data that represents relationships between various entities within a domain of a client. These taxonomy lookup sources can extract taxonomy segmentations from utterances, such as received user utterances and sample utterances of an intent-entity model. The taxonomy segmentations can then be leveraged by the NLU system to perform vocabulary injection to expand the number of meaning representations in the utterance meaning model and/or the understanding model, increasing the likelihood of matches being located during a meaning search operation. Additionally, the taxonomy lookup sources can be leveraged by the NLU system to enable validation of sample utterances submitted for inclusion in the intent-entity model.