Abstract: A system and computer implemented method for generating validated prompt-templates for generating prompts for instructing large language models (LLMs) to perform specific tasks. An initial prompt is generated instructing an LLM to produce a plurality of candidate prompt-templates. The initial prompt includes an input data type to be included with a prompt generated using a candidate prompt-template, and output data to be produced by an LLM that has processed a prompt generated using a candidate prompt-template. The initial prompt is passed through an LLM to generate candidate prompt-templates. Test prompts are generated, each constructed from a candidate prompt-templates using input data from a set of pre-labelled input data having items of input data and corresponding labels. Each test prompt is passed through a further LLM to generate an output. The output is assessed, and candidate-prompt-templates are selected for subsequent generation of prompts.

Abstract: A computer implemented method of detecting hallucination in a large language model (LLM) output. A message is received. A prompt is generated for an LLM including the message and an instruction to generate an output identifying predetermined content in the message. The prompt is passed through an LLM to generate the output. The output is processed in accordance with a hallucination detection process to identify if any predetermined content identified by the LLM in the output is potentially hallucinated.

Abstract: A system and computer implemented method for detecting hallucination in output of a generative AI system. User input is received specifying a query or task relating to information contained in a data object. A first vector representation of the user input and a second vector representation of the data object are generated. The first and second vector representations are compared to identify parts of the data object which match the query or task. An input is generated for a generative AI system with the user input and the parts of the data object. The input is input to a generative AI system. An output produced by the generative AI system is analysed to determine if the output contains information also present in the data object. If not, an error process is initiated. If so, the output is produced by the generative AI system.

Abstract: A system and method are described for improved training and testing of machine learning systems by curating user-generated scenarios and using them to validate model behavior post-training. In various embodiments, the system and method generate a dataset that contains records having relevant attribute(s), pattern(s), and/or signal(s) that can be used for model validation and training. Particular datasets that model business-relevant and real-world scenarios are selected and identified, so as to improve the effectiveness of the testing process. The selected data can be used on its own, or it can be used to augment existing training data.

Abstract: Some embodiments of the invention provide a novel method for generating benchmarking metrics for applications that execute on computing devices (e.g., mobile devices). In some embodiments, the method collects data from numerous devices regarding the execution of numerous applications on these devices. For each particular application in a set of applications, the method of some embodiments identifies a group of applications that are similar to the particular application based on a set of criteria. This identification is without human intervention in some embodiments. From the collected data, the method generates a first set of reporting metrics for the identified group of applications, and provides the generated first set of reporting metrics to the developer of the particular application. In some embodiments, the method also provides to the developer a similar set of reporting metrics that the method generates from the collected data for the particular application.