Abstract: Methods and systems are described herein for evaluating SFT processes for large language models (LLMs). For example, a request to evaluate a performance of a trained large language model to execute a computing task may be received. The request may indicate an evaluation data set including a plurality of evaluation samples. A prompt template may be retrieved based on the request. For each evaluation sample of the plurality of evaluation samples, the system may generate a sample prompt by incorporating a query and context from an evaluation sample into the prompt template. The sample prompt may be input into the large language model to obtain a sample evaluation result. An evaluation metric score may be determined based on the sample evaluation result and a reference evaluation result associated with the evaluation sample. A model performance score may be generated based on the evaluation score for each evaluation sample.

Abstract: A device may obtain a plurality of responses of a language model to a prompt, where each of the plurality of responses comprises a binary response portion and a reasoning portion. The device may determine an uncertainty quantification for the language model based on the plurality of responses. The uncertainty quantification may be based on variations across reasoning portions of the plurality of responses, or variations in binary option confidences in binary response portions of the plurality of responses. The device may output the binary response portion of one of the plurality of responses based on the uncertainty quantification indicating certainty in the plurality of responses, or an uncertainty indication based on the uncertainty quantification indicating uncertainty in the plurality of responses.

Abstract: Methods and systems are described herein for managing prompts used for executing and evaluating supervised fine tuning (SFT) processes. For example, a model identifier and a first description of a first computing task may be extracted from a first request to execute an SFT process to perform the first computing task. A candidate prompt template and associated metadata including a candidate prompt name may be generated, and the candidate prompt template may be uploaded to a prompt database with the associated metadata. A second request to execute an SFT process indicating the candidate prompt name may be received. The candidate prompt template may be used to execute the SFT process on a large language model associated with the second request, resulting in a trained first large language model. The trained first large language model may be evaluated using a test data set and the candidate prompt template.

Abstract: Methods and systems are described herein for optimizing the fine-tuning and evaluation thereof for large language models (LLMs). For example, a request to (i) execute a supervised fine tuning (SFT) process to train a large language model and (ii) evaluate a performance of the large language model after execution can be received. The request can indicate (a) a training data set to be used for the SFT process, (b) a test data set to be used to evaluate the performance of the large language model, and (c) a set of evaluation metrics to evaluate the performance of the large language model after the SFT process has been executed. A trained model can be obtained based on executing the SFT process using the training data set and a prompt template. A set of evaluation metric scores can be obtained based on evaluating the trained model using the test data set.

Abstract: Systems, apparatuses, and methods are described for data labeling for training artificial intelligence systems. A candidate dataset comprising data samples and corresponding labels may be used to update an incumbent dataset comprise data samples and corresponding labels. The integrity of a data sample-label pair in the candidate dataset may be determined before the data sample-label pair is added to the incumbent dataset. For determining labeling integrity, a plurality of machine classifiers may be trained based on the incumbent dataset and portions of the candidate dataset. The plurality of machine classifiers as trained may be used to generate predicted labels for data samples in the candidate dataset. The integrity of the data sample-label pair in the candidate dataset may be measured based on the predicted labels for the data sample.

Abstract: Methods and systems are described herein for minimizing resource expenditure during model training using user-defined constraints in sample selection. A system may obtain user-defined target parameter values for data labeling, a user input indicative of a value added per unit of model performance improvement, and a dataset (e.g., unlabeled samples). The system may select a first subset of the dataset and may transmit a request for labeling the samples. The system may receive a first training dataset comprising label data and the samples of the first subset. The system may train a machine learning model using the first training dataset and generate a margin curve. Based on the margin curve, the system may determine whether an amount of resource usage exceeds value added and responsive to determining that it does not exceed the amount of resource usage, select a second subset of the dataset.

Abstract: Disclosed embodiments pertain to protecting sensitive information. A browser extension associated with a web browser can detect a user entering information associated with the user into an electronic form. The browser extension can monitor the user entering sensitive information into the electronic form and detect that the user has entered sensitive information incorrectly. In response, the browser extension can provide a warning to the user that sensitive information has been incorrectly entered. Instructions can be displayed to a user on how incorrectly entered sensitive information is to be corrected. The incorrectly entered sensitive information is corrected based on a response from the user before the sensitive information propagates beyond the electronic form.

Abstract: Disclosed embodiments pertain to identifying sensitive data using redacted data. Data entry into electronic form fields can be monitored and analyzed to detect improperly entered sensitive data. The type of sensitive data can be determined, and the sensitive data can be removed or redacted from the electronic form field. Surrounding context data, including text associated with the sensitive data, can be identified and captured. The context data and type of sensitive data can be utilized to train or update a machine learning model configured to identify sensitive data. In one instance, the machine learning model can be employed to detect improperly entered sensitive data, and context and type can be utilized to improve the performance and predictive power of the machine learning model.

Abstract: Systems, apparatuses, and methods are described for data labeling for training artificial intelligence systems. A candidate dataset comprising data samples and corresponding labels may be used to update an incumbent dataset comprise data samples and corresponding labels. The integrity of a data sample-label pair in the candidate dataset may be determined before the data sample-label pair is added to the incumbent dataset. For determining labeling integrity, a plurality of machine classifiers may be trained based on the incumbent dataset and portions of the candidate dataset. The plurality of machine classifiers as trained may be used to generate predicted labels for data samples in the candidate dataset. The integrity of the data sample-label pair in the candidate dataset may be measured based on the predicted labels for the data sample.