Abstract: A technique determines whether a target item is adequately supported by a source item, and therefore likely free of hallucinations. The technique operates by progressively expanding a scope of source content to be considered when determining whether an individual target part of the target item has support in the source item. For instance, the technique initially determines whether any individual source part in the source item supports the target part. If this stage fails to identify support, the technique next considers whether a larger portion of the source item supports the particular target item. In some implementations, the technique selects a scope of analysis at a particular stage by choosing a group of source parts that most closely match the target part under consideration. The technique concatenates those source parts in the same order in which they appear in the source item.

Abstract: This document relates to training of machine learning models such as neural networks. One example method involves providing a machine learning model having one or more layers and associated parameters and performing a pretraining stage on the parameters of the machine learning model to obtain pretrained parameters. The example method also involves performing a tuning stage on the machine learning model by using labeled training samples to tune the pretrained parameters. The tuning stage can include performing noise adjustment of the labeled training examples to obtain noise-adjusted training samples. The tuning stage can also include adjusting the pretrained parameters based at least on the labeled training examples and the noise-adjusted training examples to obtain adapted parameters. The example method can also include outputting a tuned machine learning model having the adapted parameters.

Abstract: A method for training a language model comprises (a) receiving vectorized training data as input to a multitask pretraining problem; (b) generating modified vectorized training data based on the vectorized training data, according to an upstream data embedding; (c) emitting pretraining output based on the modified vectorized training data, according to a downstream data embedding equivalent to the upstream data embedding; and (d) adjusting the upstream data embedding and the downstream data embedding by computing, based on the pretraining output, a gradient of the upstream data embedding disentangled from a gradient of the downstream data embedding, thereby advancing the multitask pretraining problem toward a pretrained state.

Abstract: A technique is described for compressing input information fed to a machine-trained generative model. The technique includes: receiving original input information having a plurality of sentences; performing word-level encoding of the original input information using a first part of a machine-trained transformer model, to provide word-level encoded information; performing sentence-level encoding of the word-level encoded information using a second part of the machine-trained transformer model, to provide scores associated with the first plurality of sentences; selecting a subset of the sentences in the original input information based on the scores, to provide modified input information; and providing the modified input information to the machine-trained generative model. The operation of word-level encoding performs parallel processing on portions of the original input information.

Abstract: A technique supplements a language model with knowledge information retrieved from external sources. The technique operates by: receiving a query; receiving knowledge information based on the query; generating original model-input information that includes the query and the knowledge information; and presenting the original model-input information to the language model. The technique further includes: receiving an original response from the language model; generating a usefulness measure that identifies usefulness of the original response; and determining whether the usefulness measure satisfies a prescribed test. Upon determining that the usefulness measure does not satisfy the test, the technique includes: generating revised model-input information that includes feedback information; presenting the revised model-input information to the language model; and receiving a revised response from the language model.

Abstract: This document relates to architectures and training procedures for multi-task machine learning models, such as neural networks. One example method involves providing a multi-task machine learning model having one or more shared layers and two or more task-specific layers. The method can also involve performing a pretraining stage on the one or more shared layers using one or more unsupervised prediction tasks.

Abstract: Systems and methods are provided for facilitating the building and use of natural language understanding models. The systems and methods identify a plurality of tokens and use them to generate one or more pre-trained natural language models using a transformer. The transformer disentangles the content embedding and positional embedding in the computation of its attention matrix. Systems and methods are also provided to facilitate self-training of the pre-trained natural language model by utilizing multi-step decoding to better reconstruct masked tokens and improve pre-training convergence.

Abstract: The disclosure herein describes training a text processing model to generate model output text data using input text data and a sentence count. A training data entry including input text data and output text data is obtained. A sentence count of the output text data is determined, and the output text data is labeled with a sentence count label and a sentence number label. Model output text data is generated with a text processing model using the input text data and determined sentence count as input data. Loss data associated with a difference between the generated model output text data and the labeled output text data is determined and the text processing model is adjusted using the determined loss data. The use of labeled output text data enables the model to be trained to produce output text data with a target sentence count in a computationally efficient manner.

Abstract: This document relates to architectures and training procedures for multi-task machine learning models, such as neural networks. One example method involves providing a multi-task machine learning model having one or more shared layers and two or more task-specific layers. The method can also involve performing a pretraining stage on the one or more shared layers using one or more unsupervised prediction tasks. The method can also involve performing a tuning stage on the one or more shared layers and the two or more task-specific layers using respective task-specific objectives.

Abstract: A split-type friction stir welding head with an adjustable stirring pin length includes a stirring head housing, where a clamping handle and a detachable stirring pin are successively mounted in the stirring head housing towards a welding direction, the clamping handle is provided with external threads on a periphery thereof and in drive connection with an adjusting plate through threads, the adjusting plate is limited, fixed and mounted in the stirring head housing, and a pore-diameter-adjustable aperture shoulder is mounted between a bottom of the stirring head housing and the detachable stirring pin in order to compensate for an outside gap between a stirring pin channel of the stirring head housing and the detachable stirring pin.

Abstract: A shoulder-angle-adjustable friction stir welding head suitable for a fillet joint includes a stirring head body. A front end of the stirring head body is mounted with a movable shoulder, a stirring pin channel is arranged on the movable shoulder, and the stirring pin channel may allow a stirring pin of the stirring head body to pass through. The present disclosure can respond to welding tasks of the fillet joint of different angles and enlarges an application scope of the friction stir welding head in a manner that the front end of the stirring head body is mounted with the movable shoulder, the stirring pin channel is arranged on the movable shoulder, the stirring pin channel is used for allowing the stirring pin of the stirring head body to pass through, and the angle of the movable shoulder is adjusted.

Abstract: Generally discussed herein are devices, systems, and methods for generating an embedding that is both local string dependent and global string dependent. The generated embedding can improve machine learning (ML) model performance. A method can include converting a string of words to a series of tokens, generating a local string-dependent embedding of each token of the series of tokens, generating a global string-dependent embedding of each token of the series of tokens, combining the local string dependent embedding the global string dependent embedding to generate an n-gram induced embedding of each token of the series of tokens, obtaining a masked language model (MLM) previously trained to generate a masked word prediction, and executing the MLM based on the n-gram induced embedding of each token to generate the masked word prediction.

Abstract: Systems and methods are provided for training and using a novel unified language foundation model. An encoder-decoder natural language model is obtained and various training data is obtained and used for training. The training process integrates a combination of replaced token detection, corrupted span reconstruction, and disentangled attention methodologies to produce a unified encoder-decoder model. The trained model is trained for performing both natural language understanding (NLU) tasks and natural language generation (NLG) tasks. Attention applied to the model is applied discretely to segmented chunks of encoded data during processing to improve the efficiency of applying attention by the model.

Abstract: Systems and methods are provided for training and using a novel unified language foundation model. An encoder-decoder natural language model is obtained and various training data is obtained and used for training. The training process integrates a combination of replaced token detection, corrupted span reconstruction, and disentangled attention methodologies to produce a unified encoder-decoder model. The trained model is trained for performing both natural language understanding (NLU) tasks and natural language generation (NLG) tasks. Attention applied to the model is applied discretely to segmented chunks of encoded data during processing to improve the efficiency of applying attention by the model.

Abstract: This document relates to training of machine learning models. One example method involves providing a machine learning model having one or more mapping layers. The one or more mapping layers can include at least a first mapping layer configured to map components of pretraining examples into first representations in a space. The example method also includes performing a pretraining stage on the one or more mapping layers using the pretraining examples. The pretraining stage can include adding noise to the first representations of the components of the pretraining examples to obtain noise-adjusted first representations. The pretraining stage can also include performing a self-supervised learning process to pretrain the one or more mapping layers using at least the first representations of the training data items and the noise-adjusted first representations of the training data items.

Abstract: Generally discussed herein are devices, systems, and methods for generating an embedding that is both local string dependent and global string dependent. The generated embedding can improve machine learning (ML) model performance. A method can include converting a string of words to a series of tokens, generating a local string-dependent embedding of each token of the series of tokens, generating a global string-dependent embedding of each token of the series of tokens, combining the local string dependent embedding the global string dependent embedding to generate an n-gram induced embedding of each token of the series of tokens, obtaining a masked language model (MLM) previously trained to generate a masked word prediction, and executing the MLM based on the n-based induced embedding of each token to generate the masked word prediction.

Abstract: This document relates to training of machine learning models. One example method involves providing a machine learning model having one or more mapping layers. The one or more mapping layers can include at least a first mapping layer configured to map components of pretraining examples into first representations in a space. The example method also includes performing a pretraining stage on the one or more mapping layers using the pretraining examples. The pretraining stage can include adding noise to the first representations of the components of the pretraining examples to obtain noise-adjusted first representations. The pretraining stage can also include performing a self-supervised learning process to pretrain the one or more mapping layers using at least the first representations of the training data items and the noise-adjusted first representations of the training data items.

Abstract: Methods, systems, apparatuses, and computer program products are provided for extracting an entity value from a sentence. An embedding set that may include one or more sentence embeddings is generated for at least part of a first sentence that is tagged to associate a first named entity in the sentence with an entity type. A plurality of candidate embeddings is also generated for at least part of a second sentence. The one or more sentence embeddings in the embedding set may be compared with each of the plurality of candidate embeddings, and a match score may be assigned to each comparison to generate a match score set. A particular match score of the match score set may be identified that exceeds a similarity threshold, and an entity value of the entity type may be extracted from the second sentence associated with the identified match score.

Abstract: Systems and methods are provided for facilitating the building and use of natural language understanding models. The systems and methods identify a plurality of tokens and use them to generate one or more pre-trained natural language models using a transformer. The transformer disentangles the content embedding and positional embedding in the computation of its attention matrix. Systems and methods are also provided to facilitate self-training of the pre-trained natural language model by utilizing multi-step decoding to better reconstruct masked tokens and improve pre-training convergence.