Abstract: An AI query response processed by an artificial intelligence (AI) module using a neural model is received as an input. The query response is compared to one or more tokenized facts of the compressed blocks at a data level. Comparing includes computing a similarity score of a vector derived from the tokenized query response to one or more vectors derived from the one or more tokenized facts. Responsive to a verification result failing to meet a similarity score threshold, it can be determined that a hallucination exists in the AI query response and a policy-based action, such as blocking the AI query response, can be taken. Responsive to the verification result meeting the similarity threshold, the AI query response can be allowed to proceed.

Abstract: An AI query response processed by an artificial intelligence (AI) module using a neural model is received as an input. The query response is compared to one or more tokenized facts of the compressed blocks at a data level. Comparing includes computing a similarity score of a vector derived from the tokenized query response to one or more vectors derived from the one or more tokenized facts. Responsive to a verification result failing to meet a similarity score threshold, it can be determined that a hallucination exists in the AI query response and a policy-based action, such as blocking the AI query response, can be taken. Responsive to the verification result meeting the similarity threshold, the AI query response can be allowed to proceed.

Abstract: A database of tokenized data is provided. The tokenized database has been trained with a chunk of original text with words that have been compressed with tokens corresponding to the words. The text chunk is assigned a chunkID and at least some of the words are assigned a tokenID. The tokenized database can be filtered based on the tokenIDs for the one or more tokenized words from a search query. Each tokenID exposes a list of blocksIDs. A chunk of original text corresponds to each of the chunkIDs. The one or more sentences are compared to each sentence of the list of tokenized sentences to rank sentences.