Abstract: Processor(s) can receive a first natural language (NL) based input as part of a dialog between a user of a client device and an automated assistant that is accessible at the client device; process, using a first machine learning (ML) model, the first NL based input to generate a first inference associated with the user and a confidence indicator for the first inference, the confidence indicator for the first inference indicative of a degree of confidence that the first inference is true; store the first inference and the confidence indicator for the first inference in a database; process, using a second ML model, the first inference and additional data to generate a modified confidence indicator for the first inference and store the modified confidence indicator in the database; and determine, based at least on the modified confidence indicator for the first inference, whether to process, using a first large language model (LLM), the first inference with a second NL based input to generate a first NL based re