Abstract: A training system produces a resource-efficient machine-trained model via a training architecture that employs plural processing paths. Some of the processing paths incorporate the use of auxiliary information that imparts external knowledge about source items being processed. The training architecture also employs contrastive learning that operates at different respective levels within the training architecture. For instance, the training architecture uses encoder-level contrastive learning to compare output information generated by different encoders within the training architecture. The training architecture uses decoder-level contrastive learning to compare output information produced by different decoders within the training architecture. An inference-stage system performs an application task using the model produced by the training system.

Abstract: Systems and methods are provided for training a data model based on training data. The training includes pre-training and fine-tuning the data model based on a combination of an autoregressive (AR) model and a non-autoregressive (NAR) model. Training data may be received and encoded into streams of tokens. A pre-trainer during decoding generates a continuum of data structures of the AR and NAR combined model including a main stream and a series of predicting streams. Masked tokens in predicting streams reference or attend to one or more preceding tokens in the main stream or the preceding predicting streams. A fine-tuner selects streams to generate a trained model according to a target data model. The target data model is determined based on balancing an accuracy constraint and an efficiency constraint for predicting tokens. The decoder acts as abridge between the AR and NAR models in generating a trained data model.

Abstract: A knowledge injection model for generative commonsense reasoning. In examples, an encoder-decoder model is used to generate a model output (204) a plausible description for a set of concepts. A prototype (218) is generated from an in-domain or out-of-domain knowledge corpus, which is further used as input (202) for the encoder-decoder model. Concept input tokens and prototype input tokens are scaled to limit potential skew that may be introduced by the prototype (218). Additionally, position indicators are generated for each input token, which indicate the relative position each respective input token as compared to other input tokens. As such, when decoding the scaled encoded input tokens, the decoder (214) may be more attuned to the scenario bias that is introduced by the prototype (218) when generating a model output (204). Thus, the encoder-decoder model need not rely solely on the set of concepts when generating the model output (204).

Abstract: Systems and methods are provided for generating a keyword sequence from an input query. A first text sequence corresponding to an input query may be received and encoded into a source sequence representation using an encoder of a machine learning model. A keyword sentence may then be generated from the source sequence representation using a decoder of the machine learning model. The decoder may generate a modified generation score for a plurality of prediction tokens, wherein the modified generation score is based on the respective prediction token generation score and a maximum generation score for a suffix of each prediction token. The decoder may then select the prediction token of the plurality of prediction tokens based on the modified generation score, and add the selected prediction token to the previously decoded partial hypothesis provided by the decoder.

Abstract: Described herein is a mechanism to identify user intent in requests submitted to a system such as a digital assistant or question-answer systems. Embodiments utilize a match methodology instead of a classification methodology. Features derived from a subgraph retrieved from a knowledge base based on the request are concatenated with pretrained word embeddings for both the request and a candidate predicate. The concatenated inputs for both the request and predicate are encoded using two independent LSTM networks and then a matching score is calculated using a match LSTM network. The result is identified based on the matching scores for a plurality of candidate predicates. The pretrained word embeddings allow for knowledge transfer since pretrained word embeddings in one intent domain can apply to another intent domain without retraining.

Abstract: A training system produces a resource-efficient machine-trained model via a training architecture that employs plural processing paths. Some of the processing paths incorporate the use of auxiliary information that imparts external knowledge about source items being processed. The training architecture also employs contrastive learning that operates at different respective levels within the training architecture. For instance, the training architecture uses encoder-level contrastive learning to compare output information generated by different encoders within the training architecture. The training architecture uses decoder-level contrastive learning to compare output information produced by different decoders within the training architecture. An inference-stage system performs an application task using the model produced by the training system.

Abstract: Computing technology is described herein that provides an attention mechanism, implemented by a neural network, that generates attention information based on head-specific query information and shared key and value (KV) information, without computing head-specific key information and head-specific value information, and without caching the head-specific key information and the head-specific value information in memory. This manner of operation allows the computing technology to make efficient use of processing and memory resources. In some implementations, the attention mechanism is part of decoder of an encoder-decoder system, or a standalone decoder system. In some implementations, the computing technology leverages the attention information to generate synthesized text based on input text.

Abstract: A transformer-based neural network includes at least one mask attention network (MAN). The MAN computes an original attention data structure that expresses influence between pairs of data items in a sequence of data items. The MAN then modifies the original data structure by mask values in a mask data structure, to produce a modified attention data structure. Compared to the original attention data structure, the modified attention data structure better accounts for the influence of neighboring data items in the sequence of data items, given a particular data item under consideration. The mask data structure used by the MAN can have static and/or machine-trained mask values. In one implementation, the transformer-based neural network includes at least one MAN in combination with at least one other attention network that does not use a mask data structure, and at least one feed-forward neural network.

Abstract: Described herein is a mechanism to identify user intent in requests submitted to a system such as a digital assistant or question-answer systems. Embodiments utilize a match methodology instead of a classification methodology. Features derived from a subgraph retrieved from a knowledge base based on the request are concatenated with pretrained word embeddings for both the request and a candidate predicate. The concatenated inputs for both the request and predicate are encoded using two independent LSTM networks and then a matching score is calculated using a match LSTM network. The result is identified based on the matching scores for a plurality of candidate predicates. The pretrained word embeddings allow for knowledge transfer since pretrained word embeddings in one intent domain can apply to another intent domain without retraining.