Abstract: A method for text-image integration is provided. The method may include receiving a question related to pairable data comprising text data and image data. Embeddings are generated from the text tokens and image encodings. Embeddings are generated from the text tokens and image encodings. The embeddings include text embeddings and image embeddings. A spectral conversion of the text embeddings and the image embeddings is performed to generate spectral data. The spectral data is processed to extract text-image features. The text-image features are processed to generate inferred answers to the question.

Abstract: Disclosed herein are system, method, and computer program product embodiments for utilizing non-RAM memory to implement machine learning configured with a meta-learning training set (small dataset), to create a common-sense predictive language model, thus boosting the performance for downstream tasks. An embodiment operates by receiving a base sentence and perturbation sentences as an input and tokenizing the input to generate a sequence of tokens. Tokens of the semantic perturbation sentences are embedded with tokens of the base sentence as contextually similar tokens pairs to generate training data and classified to capture relationships of the base sentence and the perturbation sentences to generate a classification, which is used to train a language model.

Abstract: Disclosed herein are various embodiments for pronoun-based natural language processing. An embodiment operates by receiving a plurality of text-based sentences each comprising a plurality of words, and each text-based sentence including a pronoun. A plurality of candidate nouns are identified amongst the plurality of words. A trigger word is identified from the plurality of words, wherein the trigger word is associated with both the pronoun and one of the plurality of candidate nouns. A score for each of the candidate nouns is received based on a relationship with the trigger word. The candidate noun with a highest score is selected as being associated with the pronoun.

Abstract: A machine learning model may be trained based on a training set including training images depicting various base objects. Each training images may be associated with a ground-truth segmentation corresponding to one or more pixel-wise labels. The machine learning model may be trained to learn base class prototypes corresponding to segmentations of classes of similar base objects. The machine learning model may be further trained based on a support image depicting a novel object. The support image may be associated with an image-level label corresponding to the novel object. The machine learning model may be trained to learn, based on a base class prototype identified as being similar to the support image, a novel class prototype corresponding to the novel object. The trained machine learning model to may be applied to segment a query image. Related systems and computer program products are also provided.

Abstract: In an example embodiment, a self-supervised learning task is used for training commonsense-aware representations in a minimally supervised fashion and a pair level mutual-exclusive loss is used to enforce commonsense knowledge during representation learning. This helps to exploit the mutual-exclusive nature of the training samples of commonsense reasoning corpora. Given two pieces of input where the only difference between them are trigger pieces of data, it may be postulated that the pairwise pronoun disambiguation is mutually exclusive. This idea is formulated using a contrastive loss and then this is used to update the language model.

Abstract: A machine learning model may be trained on a first task of puzzle solving before being tuned on a second task of image analysis. The training of the machine learning model may be self-supervised whereas the tuning of the machine learning model may be supervised. The training data may include a puzzle generated to include multiple imaging modalities. The puzzle may be generated by shuffling a position of the pieces forming an original image. The machine learning model may be trained to perform the first task by reassembling the pieces in the puzzle to generate a reconstruction of the original image. Upon being trained to perform the first task and tuned to perform the second task, the machine learning model may be deployed to perform the second task. The second task may be an image segmentation task such as tumor segmentation and a regression task such as survival prediction.

Abstract: Aspects of the current subject matter are directed to a variational encoder that takes into account group characteristics of data elements of a dataset. For example, a prior adjusted variational autoencoder takes into account that not all attributes in the dataset naturally follow a normal Gaussian distribution N(0,1). To illustrate by way of an example, data from the dataset may be separated into groups in which elements in a group share group characteristics; for each group, a group representation N(mu_g, sigma_g) is calculated. And, for example, other attributes of data in the dataset do not depend on the group, and the associated data elements continue to follow the normal Gaussian distribution N(0,1). The representation may introduce a flexibility in which encodings of group-related attributes will be encoded close together in the content part instead of being close to an arbitrarily chosen point.

Abstract: A method may include training a machine learning model to perform a first task before training the machine learning model to perform the second task. The machine learning model includes a generator network and a discriminator network. The training includes training, based on a first training sample associated with the first task, the discriminator network to perform the first task. The generator network may be trained to generate a first synthetic training sample emulating the first training sample. The discriminator network trained to perform the first task may be reinitialized in order for the discriminator network to be trained, based on a second training sample, to perform the second task. The reinitialized discriminator network may be further retrained, based on the first synthetic training sample, to perform the first task. Related systems and articles of manufacture, including computer program products, are also provided.

Abstract: Aspects of the current subject matter are directed to a system in which knowledge graphs are incorporated with visual question answering. A knowledge graph is integrated into a visual question answering system to provide additional knowledge from one or more sources to answer a question about an image. Aspects of the current subject matter are directed to a neural network approach that combines methods of image feature extraction and questions processing with a neural network, such as a graph neural network, that operates on knowledge graphs. The graph neural network takes input vector representations of the nodes as inputs and combines them according to their relationships into question-specific representations. The question-specific representations are then processed with the image features and the question features to generate an answer.

Abstract: Disclosed herein are various embodiments for pronoun-based natural language processing. An embodiment operates by receiving a plurality of text-based sentences each comprising a plurality of words, and each text-based sentence including a pronoun. A plurality of candidate nouns are identified amongst the plurality of words. A trigger word is identified from the plurality of words, wherein the trigger word is associated with both the pronoun and one of the plurality of candidate nouns. A score for each of the candidate nouns is received based on a relationship with the trigger word.

Abstract: Disclosed herein are various embodiments for training and enriching a natural language processing system. An embodiment operates by identifying a natural language processor (NLP) trained on a first set of documents, wherein the NLP is trained to perform a set of functionality based on the first set of documents. An industry, set of words corresponding to the industry, and set of sentences including at least a subset of the set of words in which the NLP is to be configured to perform the set of functionality are identified. A set of sentences that exceed a similarity threshold are identified. The NLP is trained with the subset of the set of sentences that exceed the similarity threshold, wherein the trained NLP with the subset is configured to perform the set of functionality within the industry with a greater accuracy than NLP trained on only the first set of documents.

Abstract: Disclosed herein are system, method, and computer program product embodiments for utilizing non-RAM memory to implement machine learning configured with a meta-learning training set (small dataset), to create a common-sense predictive language model, thus boosting the performance for downstream tasks. An embodiment operates by receiving a base sentence and perturbation sentences as an input and tokenizing the input to generate a sequence of tokens. Tokens of the semantic perturbation sentences are embedded with tokens of the base sentence as contextually similar tokens pairs to generate training data and classified to capture relationships of the base sentence and the perturbation sentences to generate a classification, which is used to train a language model.

Abstract: Disclosed herein are system, method, and computer program product embodiments for similarity scoring of sentences, while restricting distances between tokenized pairs in the sentences. An embodiment operates by determining a similarity of tokens between a first sequence of tokens and a second sequence of tokens to generate token pairs, determining a distance of relative positioning of token pairs in the first tokenized sequence and the second tokenized sequence and generating a score value that indicates the degree to which the first sentence matches the second sentence based on restricting matches to a maximum value of the distance of relative positions of the token pairs in the first tokenized sequence and the second tokenized sequence.

Abstract: A first machine learning model may be trained to generate a first representation of a first user data having private attributes and non-private attributes. The first representation may be generated to achieve a focal entropy by maximizing an entropy amongst similar private attributes. The first representation to preserve information associated with the non-private attributes but omit information associated with the private attributes. Moreover, the first user data may be classified based on a target portion of the first representation including the non-private attributes but not the residual portion of the first representation including the private attributes. The trained first machine learning model may be applied to generate a second representation of a second user data such that downstream tasks may be performed by applying a second machine learning model to the second representation of the second user data. Related systems and computer program products are also provided.

Abstract: A method may include applying a machine learning model, such as a bidirectional encoder representations from transformers model, trained to generate a representation of a word sequence including a reference word, a first candidate noun, and a second candidate noun. The representation may include a first attention map and a second attention map. The first attention map may include attention values indicative of a strength of various linguistic relationships between the reference word and the first candidate noun. The second attention map may include attention values indicative of a strength of various linguistic relationships between the reference word and the second candidate noun. A natural language processing task, such as determining whether the reference word refers to the first candidate noun or the second candidate noun, may be performed based on the first attention map and the second attention map. Related methods and articles of manufacture are also disclosed.

Abstract: An image is received for classification. Thereafter, features are extracted from the image which are used by a machine learning model to classify the image. Thereafter, data is provided that characterizes the classification. The machine learning model can be trained using a training data set labeled, in part, using a generative model conditioned on label attribute information in combination with a directed relation graph having a plurality of nodes in which each node without images at training time are given predefined probability distributions. Related apparatus, systems, techniques and articles are also described.

Abstract: A machine learning model may be trained based on a training set including training images depicting various base objects. Each training images may be associated with a ground-truth segmentation corresponding to one or more pixel-wise labels. The machine learning model may be trained to learn base class prototypes corresponding to segmentations of classes of similar base objects. The machine learning model may be further trained based on a support image depicting a novel object. The support image may be associated with an image-level label corresponding to the novel object. The machine learning model may be trained to learn, based on a base class prototype identified as being similar to the support image, a novel class prototype corresponding to the novel object. The trained machine learning model to may be applied to segment a query image. Related systems and computer program products are also provided.

Abstract: A method may include training a first machine learning model to perform a question generation task and a second machine learning model to perform a question answering task. The first machine learning model and the second machine learning model may be subjected to a collaborative training in which a first plurality of weights applied by the first machine learning model generating one or more questions are adjusted to minimize an error in an output of the second machine learning model answering the one or more questions. The first machine learning model and the second machine learning model may be deployed to perform a natural language processing task that requires the first machine learning model to generate a question and/or the second machine learning model to answer a question. Related methods and articles of manufacture are also disclosed.

Abstract: A machine learning model may be trained on a first task of puzzle solving before being tuned on a second task of image analysis. The training of the machine learning model may be self-supervised whereas the tuning of the machine learning model may be supervised. The training data may include a puzzle generated to include multiple imaging modalities. The puzzle may be generated by shuffling a position of the pieces forming an original image. The machine learning model may be trained to perform the first task by reassembling the pieces in the puzzle to generate a reconstruction of the original image. Upon being trained to perform the first task and tuned to perform the second task, the machine learning model may be deployed to perform the second task. The second task may be an image segmentation task such as tumor segmentation and a regression task such as survival prediction.

Abstract: In an example embodiment, a solution is provided to learn representations of a dataset in order to minimize the amount of information which could be revealed about the identity of each client. Specifically, one goal is to enable the system to learn relevant properties (e.g., regular labels that are non-privacy infringing) of a dataset as a whole while protecting the privacy of the individual contributors (private labels, which can identify a client). The database may be held by a trusted server that can learn privacy-preserving representations, such as by sanitizing the identity-related information from a latent representation.