Abstract: Embodiments described herein provide systems and techniques for training large language models. In one aspect, a process for performing in-context training of a language model is disclosed. This process may begin by receiving a language model that includes a context window of a predetermined size, as well as receiving a set of in-context prompt/completion pairs prepared for a target task. The process then constructs a first token sequence based on the set of in-context prompt/completion pairs. Next, the process fits the first token sequence into the context window. The process subsequently performs a first in-context training pass using the first token sequence to train the language model to generate a next token in accordance with the target task.

Abstract: Embodiments described herein provide systems and techniques for training large language models. In one aspect, a process for performing in-context few-shot training for a transformer-based language model is disclosed. This process may begin by receiving the transformer-based language model having a context window of a predetermined size, as well as a training dataset comprising a set of prompt/completion examples. The process then constructs a training sequence based on the training dataset. Next, the process performs a single forward pass using the training sequence as input. The process subsequently performs a set of backward passes from a subset of examples in the training sequence, wherein each backward pass is conditioned on a selected subset of prompt/completion examples in the training sequence.