Abstract: Certain aspects of the disclosure provide systems and methods for training an information extraction transformer model architecture directed to pre-training a first multimodal transformer model on an unlabeled dataset, training a second multimodal transformer model on a first labeled dataset to perform a key information extraction task processing the unlabeled dataset with the second multimodal transformer model to generate pseudo-labels for the unlabeled dataset, training the first multimodal transformer model based on a second labeled dataset comprising one or more labels, the pseudo-labels generated, or combinations thereof to generate a third multimodal transformer model, generating updated pseudo-labels based on label completion predictions from the third multimodal transformer model, and training the third multimodal transformer model using a noise-aware loss function and the updated pseudo-labels to generate an updated third multimodal transformer model.

Abstract: Aspects of the present disclosure provide techniques for image-based document search. Embodiments include receiving an image of a document and providing the image of the document as input to a machine learning model, where the machine learning model generates separate embeddings of a plurality of patches of the image of the document and the machine learning model generates an embedding of the image of the document based on the separate embeddings of the plurality of patches. Embodiments include determining a compact embedding of the image of the document based on applying a dimensionality reduction technique to the embedding of the image of the document generated by the machine learning model. Embodiments include performing a search for relevant documents based on the compact embedding of the image of the document. Embodiments include performing one or more actions based on one or more relevant documents identified through the search.

Abstract: Multimodal multitask machine learning system for document intelligence tasks includes a feature extractor processing token values obtained from a document to obtain features, and a token extraction head classifying, using the features, the token values to obtain classified tokens. The classified tokens are aggregated into entities. A document classification model is executed on the features to classify the document and obtain a document label prediction. Further a confidence head model applying the document label prediction processes the entities to obtain a result.