Abstract: In some embodiments, a multimodal computing system receives a query and identifies, from source documents, text passages and images that are relevant to the query. The multimodal computing system accesses a multimodal question-answering model that includes a textual stream of language models and a visual stream of language models. Each of the textual stream and the visual stream contains a set of transformer-based models and each transformer-based model includes a cross-attention layer using data generated by both the textual stream and visual stream of language models as an input. The multimodal computing system identifies text relevant to the query by applying the textual stream to the text passages and computes, using the visual stream, relevance scores of the images to the query, respectively. The multimodal computing system further generates a response to the query by including the text and/or an image according to the relevance scores.

Abstract: A method includes: determining a first bounding box in textual data; categorizing the first bounding box into a paragraph category among a list of categories including a header category, a section header category, the paragraph category, and a noise category; extracting a candidate noun from the text in the categorized first bounding box, as a candidate paragraph noun; extracting a candidate value from the text in the categorized first bounding box; generating a relationship between the candidate paragraph noun and the extracted candidate value; associating the candidate paragraph noun with a candidate header noun; and generating a key performance indicator using the candidate header noun, the candidate paragraph noun, the candidate value, and the generated relationship.

Abstract: A method comprises identifying entities in a form document; for each of the entities, generating a vector for the entity that comprises a plurality of values describing content of the entity and a position of the entity within the form document; identifying, by the computing system, pairs of the entities that satisfy a spatial relationship requirement; for each pair of entities satisfying the spatial relationship requirement, applying a classification machine-learned (ML) model that takes as input the vectors for the pair of entities and outputs a category for a first entity of the pair of entities and a category for a second entity of the pair of entities, wherein a combination of the category for the first entity and the category for the second entity represents a semantic relationship between the first entity and the second entity; and processing the form document based on the semantic relationships between the entities.

Abstract: Various embodiments of the present invention provide methods, apparatus, systems, computing devices, computing entities, and/or the like for performing natural language processing operations by generating semantic table representations for table data objects using a token-wise entity type classification mechanism whose output space is defined by a set of defined entity types characterized by an inter-related entity type taxonomy to generate a representation of a table data object that describes per-token semantic inferences and cross-token semantic inferences performed on the table data object in accordance with subject-matter-domain insights as described by the inter-related entity type taxonomy.

Abstract: In some embodiments, a multimodal computing system receives a query and identifies, from source documents, text passages and images that are relevant to the query. The multimodal computing system accesses a multimodal question-answering model that includes a textual stream of language models and a visual stream of language models. Each of the textual stream and the visual stream contains a set of transformer-based models and each transformer-based model includes a cross-attention layer using data generated by both the textual stream and visual stream of language models as an input. The multimodal computing system identifies text relevant to the query by applying the textual stream to the text passages and computes, using the visual stream, relevance scores of the images to the query, respectively. The multimodal computing system further generates a response to the query by including the text and/or an image according to the relevance scores.