Abstract: A method of constructing a dataset for identifying a plurality of latent concepts in a Natural Language Processing model is provided. The method includes executing a clustering process on a first dataset, preparing a second dataset, defining a hierarchical concept tag-set from the second dataset, and annotating the hierarchical concept tag-set.

Abstract: The presently disclosed method and system automatically diacritize written Arabic text for use with applications that require verbalizing Arabic text. A method may comprise converting a written sentence into a word sequence and identifying a target source word. The method then may comprise repeatedly overlaying and translating a context window at a plurality of positions in the word sequence to select a plurality of subsets of the word sequence contained within the context window. A diacritized form of the target source word may be generated in each of the word sequence subsets. A final diacritized form of the target source word may be selected from the plurality of diacritized forms based on a voting scheme. The voting scheme may include selecting the diacritized form that is generated the most or may be based on a system of weighting factors.

Abstract: A search system that automatically generates questions to refine an underspecified query. The system may generate questions even for queries against a database that contains unstructured textual descriptions of items, allowing the system to operate on a database of items that can be constructed inexpensively. The system extracts from the unstructured text combinations of words that may serve as a set of attribute values. The system uses a classifier to filter out attribute values from the set that would generate unanswerable questions. The remaining attribute values are ranked on their ability to narrow the search results and the highest ranking attribute value is used to generate a question to the user who submitted the query. The response to the question narrows the search results, and the process can be repeated iteratively until the search results are sufficiently narrow.

Abstract: A search system that automatically generates questions to refine an underspecified query. The system may generate questions even for queries against a database that contains unstructured textual descriptions of items, allowing the system to operate on a database of items that can be constructed inexpensively. The system extracts from the unstructured text combinations of words that may serve as a set of attribute values. The system uses a classifier to filter out attribute values from the set that would generate unanswerable questions. The remaining attribute values are ranked on their ability to narrow the search results and the highest ranking attribute value is used to generate a question to the user who submitted the query. The response to the question narrows the search results, and the process can be repeated iteratively until the search results are sufficiently narrow.