Abstract: An aspect of the disclosed technology comprises a test-time adaptation (“TTA”) technique for visual document understanding (“VDU”) tasks that uses self-supervised learning on different modalities (e.g., text and layout) by applying masked visual language modeling (“MVLM”) along with pseudo-labeling. In accordance with an aspect of the disclosed technology, the TTA technique enables a document model to adapt to domain or distribution shifts that are detected.

Abstract: A method for rehearsal-free continual learning includes obtaining a set of training samples where training sample in the set of training samples is associated with a respective task of a plurality of different tasks. The method includes obtaining a task-invariant prompt representative of learned knowledge common to each respective task of the plurality of different tasks. The method includes, for each respective task of the plurality of different tasks, obtaining a respective task-specific prompt representative of learned knowledge specific to the respective task. The method includes, during each of one or more training iterations, for each respective training sample in the set of training samples, selecting the respective task-specific prompt representative of the respective task of the respective training sample and training a model using the task-invariant prompt and the selected respective task-specific prompt.