Abstract: Techniques are provided for dynamic context-based evaluation of language models. One method comprises accessing a data structure comprising information characterizing a user input and a corresponding context of the user input; dynamically identifying, in response to the user input, evaluation criteria, from a pool of evaluation criteria, based on the context of the user input; dynamically generating evaluation metrics based on the identified evaluation criteria; applying the user input to at least one language model, wherein the at least one language model generates a response based on the user input; evaluating the response from the at least one language model using the dynamically generated evaluation metrics to assign a score to the response; and initiating an automated action based on the score assigned to the response. A weight may be assigned to the evaluation metrics based on the context of the user input.

Abstract: Techniques are provided for dynamic context-based evaluation of language models. One method comprises accessing a data structure comprising information characterizing a user input and a corresponding context of the user input; dynamically identifying, in response to the user input, evaluation criteria, from a pool of evaluation criteria, based on the context of the user input; dynamically generating evaluation metrics based on the identified evaluation criteria; applying the user input to at least one language model, wherein the at least one language model generates a response based on the user input; evaluating the response from the at least one language model using the dynamically generated evaluation metrics to assign a score to the response; and initiating an automated action based on the score assigned to the response. A weight may be assigned to the evaluation metrics based on the context of the user input.