Abstract: Systems and methods for pre-training and fine-tuning of neural-network-based language models are disclosed in which a neural-network-based textual knowledge retriever is trained along with the language model. In some examples, the knowledge retriever obtains documents from an unlabeled pre-training corpus, generates its own training tasks, and learns to retrieve documents relevant to those tasks. In some examples, the knowledge retriever is further refined using supervised open-QA questions. The framework of the present technology provides models that can intelligently retrieve helpful information from a large unlabeled corpus, rather than requiring all potentially relevant information to be stored implicitly in the parameters of the neural network. This framework may thus reduce the storage space and complexity of the neural network, and also enable the model to more effectively handle new tasks that may be different than those on which it was pre-trained.

Abstract: Semantic entity relation detection classifier training implementations are presented that are generally used to train a semantic entity relation detection classifier to identify relations expressed in a natural language query. In one general implementation, queries are found in a search query click log that exhibit relations and entity types found in a semantic knowledge graph. Explicit relations are inferred from the found queries and an explicit relations data set is generated that includes queries associated with the inferred explicit relations. In addition, implicit relations are inferred from the found queries and an implicit relations data set is generated that includes queries associated with the inferred implicit relations. A semantic entity relation detection classifier is then trained using the explicit and implicit data sets.

Abstract: Semantic entity relation detection classifier training implementations are presented that are generally used to train a semantic entity relation detection classifier to identify relations expressed in a natural language query. In one general implementation, queries are found in a search query click log that exhibit relations and entity types found in a semantic knowledge graph. Explicit relations are inferred from the found queries and an explicit relations data set is generated that includes queries associated with the inferred explicit relations. In addition, implicit relations are inferred from the found queries and an implicit relations data set is generated that includes queries associated with the inferred implicit relations. A semantic entity relation detection classifier is then trained using the explicit and implicit data sets.