Abstract: Mechanisms are provided for selecting and annotating test examples for artificial intelligence (AI) model selection. First and second AI computer models process a test example of a test set to generate a corresponding first AI computer model output and second AI computer model output. A corresponding first vector embedding and second vector embedding are generated and a difference vector is generated based on a difference between the first vector embedding and the second vector embedding. The difference vectors are clustered into a plurality of clusters of difference vectors for test examples of the test set. Representative difference vectors are selected from each cluster and their corresponding test examples are annotated by an oracle for use in selecting an AI model for a task.

Abstract: Methods, systems, and computer program products for implementing active learning in NLG tasks are provided herein. A computer-implemented method includes generating multiple natural language annotations associated with multiple items of unlabeled data by processing the unlabeled data using at least one artificial intelligence model; determining at least one quality score attributed to at least a portion of the multiple generated natural language annotations based at least in part on at least one quality metric; selecting at least one of the multiple natural language annotations and at least one corresponding item of the multiple items of unlabeled data based at least in part on the at least one determined quality score; and performing one or more automated actions based at least in part on the at least one selected natural language annotation.

Abstract: Methods, systems, and computer program products for implementing active learning in NLG tasks are provided herein. A computer-implemented method includes generating multiple natural language annotations associated with multiple items of unlabeled data by processing the unlabeled data using at least one artificial intelligence model; determining at least one quality score attributed to at least a portion of the multiple generated natural language annotations based at least in part on at least one quality metric; selecting at least one of the multiple natural language annotations and at least one corresponding item of the multiple items of unlabeled data based at least in part on the at least one determined quality score; and performing one or more automated actions based at least in part on the at least one selected natural language annotation.

Abstract: Techniques for autocontrastive decoding of a machine learning model are provided. In one aspect, a system for machine learning includes: a multi-layer machine learning model; and an autocontrastive decoding module configured to obtain prediction probabilities from multiple, different layers of the multi-layer machine learning model as data propagates through the multi-layer machine learning model, and aggregate in a contrastive manner the prediction probabilities from the multiple, different layers of the multi-layer machine learning model to provide a final output from the multi-layer machine learning model. The multi-layer machine learning model can be a transformer-based machine learning model. The autocontrastive decoding module can be configured to redistribute a prediction probability distribution of the transformer-based machine learning model by maximizing a difference between log-probabilities of a final layer and one or more intermediate layers of the transformer-based machine learning model.

Abstract: An example system includes a processor to pre-train a transformer-based language model on a general domain. The processor can inter-train the pre-trained transformer-based language model using partitioning and classification to generate an inter-trained transformer-based pre-trained language model. The processor can then fine-tune the inter-trained transformer-based pre-trained language model on a target task to generate a fine-tuned transformer-based language model.

Abstract: An example system includes a processor to pre-train a transformer-based language model on a general domain. The processor can inter-train the pre-trained transformer-based language model using partitioning and classification to generate an inter-trained transformer-based pre-trained language model. The processor can then fine-tune the inter-trained transformer-based pre-trained language model on a target task to generate a fine-tuned transformer-based language model.