Abstract: Disclosed is a preference model that improves the alignment of large language model (LLM) responses. For a particular scenario, a set of principles are identified that, when adhered to by the LLM, improve the quality of LLM responses. In a long form question and answer scenario, better answers tend to be useful, relevant, grounded, thorough, and true, although more and different principles are similarly contemplated. In a scenario that generates a movie script, better responses may include a relatable protagonist, a character arc, and a satisfying denouement. The disclosed axiomatic preference model is trained to understand when a response does or does not adhere to these principles. Once trained, the preference model may be used as a drop-in replacement of existing preference models used to train an LLM.

Abstract: This document relates to training of machine learning models. One example method involves providing a machine learning model having a first classification layer, a second classification layer, and an encoder that feeds into the first classification layer and the second classification layer. The example method also involves obtaining first training examples having explicit labels and second training examples having inferred labels. The inferred labels are based at least on actions associated with the second training examples. The example method also involves training the machine learning model using the first training examples and the second training examples using a training objective that considers first training loss of the first classification layer for the explicit labels and second training loss of the second classification layer for the inferred labels. The method also involves outputting a trained machine learning model having the encoder and the first classification layer.

Abstract: This document relates to training of machine learning models. One example method involves providing a machine learning model having a first classification layer, a second classification layer, and an encoder that feeds into the first classification layer and the second classification layer. The example method also involves obtaining first training examples having explicit labels and second training examples having inferred labels. The inferred labels are based at least on actions associated with the second training examples. The example method also involves training the machine learning model using the first training examples and the second training examples using a training objective that considers first training loss of the first classification layer for the explicit labels and second training loss of the second classification layer for the inferred labels. The method also involves outputting a trained machine learning model having the encoder and the first classification layer.

Abstract: This document relates to training of machine learning models. One example method involves providing a machine learning model having a first classification layer, a second classification layer, and an encoder that feeds into the first classification layer and the second classification layer. The example method also involves obtaining first training examples having explicit labels and second training examples having inferred labels. The inferred labels are based at least on actions associated with the second training examples. The example method also involves training the machine learning model using the first training examples and the second training examples using a training objective that considers first training loss of the first classification layer for the explicit labels and second training loss of the second classification layer for the inferred labels. The method also involves outputting a trained machine learning model having the encoder and the first classification layer.

Abstract: This document relates to training of machine learning models. One example method involves providing a machine learning model having a first classification layer, a second classification layer, and an encoder that feeds into the first classification layer and the second classification layer. The example method also involves obtaining first training examples having explicit labels and second training examples having inferred labels. The inferred labels are based at least on actions associated with the second training examples. The example method also involves training the machine learning model using the first training examples and the second training examples using a training objective that considers first training loss of the first classification layer for the explicit labels and second training loss of the second classification layer for the inferred labels. The method also involves outputting a trained machine learning model having the encoder and the first classification layer.