Abstract: Implementations are described herein for identifying potentially false information in generative model output by performing entailment evaluation of generative model output. In various implementations, data indicative of a query may be processed to generate generative model output. Textual fragments may be extracted from the generative model output, and a subset of the textual fragments may be classified as being suitable for textual entailment analysis. Textual entailment analysis may be performed on each textual fragment of the subset, including formulating a search query based on the textual fragment, retrieving document(s) responsive to the search query, and processing the textual fragment and the document(s) using entailment machine learning model(s) to generate prediction(s) of whether the at least one document corroborates or contradicts the textual fragment. When natural language (NL) responsive to the query is rendered at a client device, annotation(s) may be rendered to express the prediction(s).