Abstract: Systems and methods for document editing are provided. One aspect of the systems and methods includes obtaining a document and a natural language edit request. Another aspect of the systems and methods includes generating a structured edit command using a machine learning model based on the document and the natural language edit request. Yet another aspect of the systems and methods includes generating a modified document based on the document and the structured edit command, where the modified document includes a revision of the document that incorporates the natural language edit request.

Abstract: Adapting a machine learning model to process data that differs from training data used to configure the model for a specified objective is described. A domain adaptation system trains the model to process new domain data that differs from a training data domain by using the model to generate a feature representation for the new domain data, which describes different content types included in the new domain data. The domain adaptation system then generates a probability distribution for each discrete region of the new domain data, which describes a likelihood of the region including different content described by the feature representation. The probability distribution is compared to ground truth information for the new domain data to determine a loss function, which is used to refine model parameters. After determining that model outputs achieve a threshold similarity to the ground truth information, the model is output as a domain-agnostic model.

Abstract: Systems and methods for document classification are described. Embodiments of the present disclosure generate classification data for a plurality of samples using a neural network trained to identify a plurality of known classes; select a set of samples for annotation from the plurality of samples using an open-set metric based on the classification data, wherein the annotation includes an unknown class; and train the neural network to identify the unknown class based on the annotation of the set of samples.

Abstract: Systems and techniques for generating responsive documents are described. Digital content is organized into a structure that defines how content is presented when a document is displayed by a computing device. To generate the responsive document, relationships are defined among different digital content objects, such as groups of content objects to be presented together and content objects that are to be presented as alternatives of one another. Responsive patterns are assigned to grouped content objects, where each responsive pattern defines different layout configurations for displaying grouped content objects based on computing device display characteristics. In some implementations, multiple responsive patterns are assigned to a single content group and individual responsive patterns are associated with activation ranges for display characteristics that activate the responsive pattern.

Abstract: Embodiments provide systems, methods, and computer storage media for extracting semantic labels for field widgets of form fields in unfilled forms. In some embodiments, a processing device accesses a representation of a fillable widget of a form field of an unfilled form. The processing device generates an encoded input representing text and layout of a sequence of tokens in a neighborhood of the fillable widget. The processing device uses a machine learning model to extract a semantic label representing a field type of the fillable widget in view of the encoded input. The processing device causes execution of an action using the semantic label.

Abstract: Certain embodiments involve using a machine-learning tool to generate metadata identifying segments and topics for text within a document. For instance, in some embodiments, a text processing system obtains input text and applies a segmentation-and-labeling model to the input text. The segmentation-and-labeling model is trained to generate a predicted segment for the input text using a segmentation network. The segmentation-and-labeling model is also trained to generate a topic for the predicted segment using a pooling network of the model to the predicted segment. The output of the model is usable for generating metadata identifying the predicted segment and the associated topic.

Abstract: Embodiments are provided for facilitating multimodal extraction across multiple granularities. In one implementation, a set of features of a document for a plurality of granularities of the document is obtained. Via a machine learning model, the set of features of the document are modified to generate a set of modified features using a set of self-attention values to determine relationships within a first type of feature and a set of cross-attention values to determine relationships between the first type of feature and a second type of feature. Thereafter, the set of modified features are provided to a second machine learning model to perform a classification task.

Abstract: The technology described herein is directed to an adaptive sparse attention pattern that is learned during fine-tuning and deployed in a machine-learning model. In aspects, a row or a column in an attention matrix with an importance score for a task that is above a threshold importance score is identified. The important row or the column is included in an adaptive attention pattern used with a machine-learning model having a self-attention operation. In response to an input, a task-specific inference is generated for the input using the machine-learning model with the adaptive attention pattern.

Abstract: Certain embodiments involve using a machine-learning tool to generate metadata identifying segments and topics for text within a document. For instance, in some embodiments, a text processing system obtains input text and applies a segmentation-and-labeling model to the input text. The segmentation-and-labeling model is trained to generate a predicted segment for the input text using a segmentation network. The segmentation-and-labeling model is also trained to generate a topic for the predicted segment using a pooling network of the model to the predicted segment. The output of the model is usable for generating metadata identifying the predicted segment and the associated topic.

Abstract: Methods and systems are provided for facilitating identification of fillable regions and/or data associated therewith. In embodiments, a candidate fillable region indicating a region in a form that is a candidate for being fillable is obtained. Textual context indicating text from the form and spatial context indicating positions of the text within the form are also obtained. Fillable region data associated with the candidate fillable region is generated, via a machine learning model, using the candidate fillable region, the textual context, and the spatial context. Thereafter, a fillable form is generated using the fillable region data, the fillable form having one or more fillable regions for accepting input.

Abstract: The technology described includes methods for pretraining a document encoder model based on multimodal self cross-attention. One method includes receiving image data that encodes a set of pretraining documents. A set of sentences is extracted from the image data. A bounding box for each sentence is generated. For each sentence, a set of predicted features is generated by using an encoder machine-learning model. The encoder model performs cross-attention between a set of masked-textual features for the sentence and a set of masked-visual features for the sentence. The set of masked-textual features is based on a masking function and the sentence. The set of masked-visual features is based on the masking function and the corresponding bounding box. A document-encoder model is pretrained based on the set of predicted features for each sentence and pretraining tasks. The pretraining tasks includes masked sentence modeling, visual contrastive learning, or visual-language alignment.

Abstract: Various disclosed embodiments can resolve output inaccuracies produced by many machine learning models. Embodiments use content order as input to machine learning model systems so that they can process documents according to the position or rank of instances in a document or image. In this way, the model is less likely to misclassify or incorrectly detect instances or the ordering between predicted instances. The content order in various embodiments can be used as an additional signal to classify or make predictions.

Abstract: Systems and methods for natural language processing are described. One or more embodiments of the present disclosure identify a claim from a document, wherein the claim corresponds to a topic, create a graph comprising a plurality of nodes having a plurality of node types and a plurality of edges having a plurality of edge types, wherein one of the nodes represents the claim, and wherein each of the edges represents a relationship between a corresponding pair of the nodes, encode the claim based on the graph using a graph convolutional network (GCN) to obtain an encoded claim, classify the claim by decoding the encoded claim to obtain a stance label that indicates a stance of the claim towards the topic, and transmit information indicating a viewpoint of the document towards the topic based on the stance label.

Abstract: A domain alignment technique for cross-domain object detection tasks is introduced. During a preliminary pretraining phase, an object detection model is pretrained to detect objects in images associated with a source domain using a source dataset of images associated with the source domain. After completing the pretraining phase, a domain adaptation phase is performed using the source dataset and a target dataset to adapt the pretrained object detection model to detect objects in images associated with the target domain. The domain adaptation phase may involve the use of various domain alignment modules that, for example, perform multi-scale pixel/path alignment based on input feature maps or perform instance-level alignment based on input region proposals.

Abstract: Adapting a machine learning model to process data that differs from training data used to configure the model for a specified objective is described. A domain adaptation system trains the model to process new domain data that differs from a training data domain by using the model to generate a feature representation for the new domain data, which describes different content types included in the new domain data. The domain adaptation system then generates a probability distribution for each discrete region of the new domain data, which describes a likelihood of the region including different content described by the feature representation. The probability distribution is compared to ground truth information for the new domain data to determine a loss function, which is used to refine model parameters. After determining that model outputs achieve a threshold similarity to the ground truth information, the model is output as a domain-agnostic model.

Abstract: Various disclosed embodiments can resolve output inaccuracies produced by many machine learning models. Embodiments use content order as input to machine learning model systems so that they can process documents according to the position or rank of instances in a document or image. In this way, the model is less likely to misclassify or incorrectly detect instances or the ordering between predicted instances. The content order in various embodiments can be used as an additional signal to classify or make predictions.

Abstract: Adapting a machine learning model to process data that differs from training data used to configure the model for a specified objective is described. A domain adaptation system trains the model to process new domain data that differs from a training data domain by using the model to generate a feature representation for the new domain data, which describes different content types included in the new domain data. The domain adaptation system then generates a probability distribution for each discrete region of the new domain data, which describes a likelihood of the region including different content described by the feature representation. The probability distribution is compared to ground truth information for the new domain data to determine a loss function, which is used to refine model parameters. After determining that model outputs achieve a threshold similarity to the ground truth information, the model is output as a domain-agnostic model.

Abstract: Methods and systems disclosed herein relate generally to systems and methods for generating visual relationship graphs that identify relationships between objects depicted in an image. A vision-language application uses transformer encoders to generate a graph structure, in which the graph structure represents a dependency between a first region and a second region of an image. The dependency indicates that a contextual representation of the first region was derived, at least in part, by processing the second region. The contextual representation identifies a predicted identity of an image object depicted in the first region. The predicted identity is determined at least in part by identifying a relationship between the first region and other data objects associated with various modalities.

Abstract: Introduced here are computer programs and associated computer-implemented techniques for creating visualizations to explain the outputs produced by models designed for object detection. To accomplish this, a graphics editing platform can obtain a reference output that identifies a region of pixels in a digital image that allegedly contains an object. Then, the graphics editing platform can compute the similarity between the reference output and a series of outputs generated by a model upon being applied to masked versions of the digital image. A visualization component can be produced based on the similarity.

Abstract: Introduced here are computer programs and associated computer-implemented techniques for creating visualizations to explain the outputs produced by models designed for object detection. To accomplish this, a graphics editing platform can obtain a reference output that identifies a region of pixels in a digital image that allegedly contains an object. Then, the graphics editing platform can compute the similarity between the reference output and a series of outputs generated by a model upon being applied to masked versions of the digital image. A visualization component can be produced based on the similarity.