Abstract: A technique supplements a language model with knowledge information retrieved from external sources. The technique operates by: receiving a query; receiving knowledge information based on the query; generating original model-input information that includes the query and the knowledge information; and presenting the original model-input information to the language model. The technique further includes: receiving an original response from the language model; generating a usefulness measure that identifies usefulness of the original response; and determining whether the usefulness measure satisfies a prescribed test. Upon determining that the usefulness measure does not satisfy the test, the technique includes: generating revised model-input information that includes feedback information; presenting the revised model-input information to the language model; and receiving a revised response from the language model.

Abstract: Example methods, apparatuses, and storage media are provided. An example method includes receiving at least one satellite signal and at least one reference signal. The example method further includes sending first information and second information. The first information includes a phase measurement value determined based on the at least one satellite signal. The second information includes a phase measurement value determined based on the at least one reference signal.

Abstract: This document relates to architectures and training procedures for multi-task machine learning models, such as neural networks. One example method involves providing a multi-task machine learning model having one or more shared layers and two or more task-specific layers. The method can also involve performing a pretraining stage on the one or more shared layers using one or more unsupervised prediction tasks.

Abstract: A technique performs the task of knowledge-graph completion in a manner that is both scalable and resource efficient. In some implementations, the technique identifies a source entity having a source-target relation that connects the source entity to a yet-to-be-determined target entity. The technique also identifies a source-entity data item that provides a passage of source-entity text pertaining to the source entity. The technique uses a machine-trained encoder model to map the source-entity data item to source-entity encoded information. The technique then predicts an identity of the target entity based on the source-entity encoded information, and based on predicate encoded information that encodes the source-target relation. In some implementations, the technique also predicts the target entity based on a consideration of one or more neighboring entities that are connected to the source entity and their respective source-to-neighbor relations.

Abstract: The disclosed concepts relate to pretraining of machine learning models. One example method involves performing separate optimization of a first machine learning model and a second machine learning model. The first machine learning model can be optimized based at least on first predictions and the second machine learning model can be optimized based at least on second predictions. The first predictions can represent predictions of masked values in first sequences of values values, and the second predictions can represent whether or not the first values were replaced with different values predicted by the first machine learning model.

Abstract: Systems and methods are provided for facilitating the building and use of natural language understanding models. The systems and methods identify a plurality of tokens and use them to generate one or more pre-trained natural language models using a transformer. The transformer disentangles the content embedding and positional embedding in the computation of its attention matrix. Systems and methods are also provided to facilitate self-training of the pre-trained natural language model by utilizing multi-step decoding to better reconstruct masked tokens and improve pre-training convergence.

Abstract: Systems and methods are provided for determining a response to a query in a dialog. An entity extractor extracts rules and conditions associated with the query and determines a particular task. The disclosed technology generates a transformer-based dialog embedding by pre-training a transformer using dialog corpora including a plurality of tasks. A task-specific classifier generates a first set of candidate responses based on rules and conditions associated with the task. The transformer-based dialog embedding generates a second set of candidate responses to the query. The classifier accommodates changes made to a task by an interactive dialog editor as machine teaching. A response generator generates a response based on the first and second sets of candidate responses using an optimization function.

Abstract: A computing device is provided including a processor configured to execute a transformer including an encoder having a global layer configured to receive tokenized embeddings for each of a plurality of tokens in a local input sequence and compute a global self-attention vector for each of the tokenized embeddings. The encoder further includes a local layer configured to receive each global self-attention vector from the global layer and compute local self-attention for each local input sequence, and add and normalize the global self-attention vector with the local self-attention vector to thereby produce an encoder representation including a self-attention vector for each local input sequence that includes both global self-attention values and local self-attention values. The transformer is configured to output a prediction for the global input sequence based on the encoder representation of each of the local input sequences of the global input sequence.

Abstract: A method for manufacturing a semiconductor and a semiconductor. The method includes: providing a substrate, wherein an active region trench is on the substrate, and a channel stack of a gate-all-around transistor is formed in the active region trench, the active region trench is divided into a source trench and a drain trench by the channel stack; epitaxially growing a source crystal structure in the source trench and a drain crystal structure in the drain trench, and stopping epitaxial growth before crystal planes with different orientations of the source crystal structure intersect and crystal planes with different orientations of the drain crystal structure intersect; and filling gaps between the crystal planes with different orientations of the source crystal structure and the drain crystal structure by using an isotropic metal material, and forming a source and a drain of the gate-all-around transistor in the source trench and the drain trench, respectively.

Abstract: A method for manufacturing a nanostructure and a nanostructure are disclosed. The method for manufacturing the nanostructure includes first alternately and periodically stacking a first material layer and a second material layer on a substrate to form a stacked layer, then forming a slot pattern on an upper surface of the stacked layer and etching the stacked layer to an upper surface of the substrate to transfer the slot pattern to the stacked layer, filling the slot pattern in the stacked layer with a molding material, and removing the first material layer or the second material layer left in the stacked layer, so as to form nanopores arranged in an array in the stacked layer.

Abstract: A metal interconnection structure of a semiconductor device and a method for forming the same. The method includes: providing a substrate; forming a first dielectric layer on the substrate; forming a first conductive structure in the first dielectric layer; etching back part of the first conductive structure; forming an etch stop layer on the first conductive structure; forming a second dielectric layer on the etch stop layer and performing chemical mechanical polishing; and forming a second conductive structure in the second dielectric layer, where the second conductive structure is electrically connected to the first conductive structure.

Abstract: This document relates to architectures and training procedures for multi-task machine learning models, such as neural networks. One example method involves providing a multi-task machine learning model having one or more shared layers and two or more task-specific layers. The method can also involve performing a pretraining stage on the one or more shared layers using one or more unsupervised prediction tasks. The method can also involve performing a tuning stage on the one or more shared layers and the two or more task-specific layers using respective task-specific objectives.

Abstract: A method for computer question answering includes, at a retriever subsystem of a question answering computer system, identifying a plurality of relevant text evidence strings for an input text question. At a linker subsystem of the question answering computer system, one or more of the plurality of relevant text evidence strings are associated with a respective secondary text evidence string to form a plurality of evidence chains via a previously-trained entity-linking machine-learning model. At a chainer subsystem of the question answering computer system, a ranked set of the evidence chains is identified based at least in part on an output of a generative machine-learning model applied to each of the plurality of evidence chains. At a reader subsystem of the question answering computer system, an answer to the input text question is output based at least in part on the ranked set of evidence chains.

Abstract: A computing system includes a logic subsystem and a storage subsystem holding instructions executable by the logic subsystem to implement a transformer-based text encoder. The transformer-based text encoder includes a plurality of transformer blocks previously-trained to apply encoding operations to computer-readable text representations of input text strings, the computer-readable text representations including computer-readable question representations of input text questions, and computer-readable passage representations of input text passages. The plurality of transformer blocks include a shared transformer block trained for both the computer-readable question representations and the computer-readable passage representations and a specialized transformer block including two or more input-specific subnetworks, and a routing function to select an input-specific subnetwork of the two or more input-specific subnetworks for each of the computer-readable text representations.

Abstract: Methods and systems are disclosed for improving dialog management for task-oriented dialog systems. The disclosed dialog builder leverages machine teaching processing to improve development of dialog managers. In this way, the dialog builder combines the strengths of both rule-based and machine-learned approaches to allow dialog authors to: (1) import a dialog graph developed using popular dialog composers, (2) convert the dialog graph to text-based training dialogs, (3) continuously improve the trained dialogs based on log dialogs, and (4) generate a corrected dialog for retraining the machine learning.

Abstract: Generally discussed herein are devices, systems, and methods for generating an embedding that is both local string dependent and global string dependent. The generated embedding can improve machine learning (ML) model performance. A method can include converting a string of words to a series of tokens, generating a local string-dependent embedding of each token of the series of tokens, generating a global string-dependent embedding of each token of the series of tokens, combining the local string dependent embedding the global string dependent embedding to generate an n-gram induced embedding of each token of the series of tokens, obtaining a masked language model (MLM) previously trained to generate a masked word prediction, and executing the MLM based on the n-gram induced embedding of each token to generate the masked word prediction.

Abstract: Systems and methods are provided for training and using a novel unified language foundation model. An encoder-decoder natural language model is obtained and various training data is obtained and used for training. The training process integrates a combination of replaced token detection, corrupted span reconstruction, and disentangled attention methodologies to produce a unified encoder-decoder model. The trained model is trained for performing both natural language understanding (NLU) tasks and natural language generation (NLG) tasks. Attention applied to the model is applied discretely to segmented chunks of encoded data during processing to improve the efficiency of applying attention by the model.

Abstract: Systems and methods are provided for training and using a novel unified language foundation model. An encoder-decoder natural language model is obtained and various training data is obtained and used for training. The training process integrates a combination of replaced token detection, corrupted span reconstruction, and disentangled attention methodologies to produce a unified encoder-decoder model. The trained model is trained for performing both natural language understanding (NLU) tasks and natural language generation (NLG) tasks. Attention applied to the model is applied discretely to segmented chunks of encoded data during processing to improve the efficiency of applying attention by the model.

Abstract: This document relates to machine learning. One example includes a method or technique that can be performed on a computing device. The method or technique can include obtaining a task-semantically-conditioned generative model that has been pretrained based at least on a first training data set having unlabeled training examples and semantically conditioned based at least on a second training data set having dialog act-labeled utterances. The method or technique can also include inputting dialog acts into the semantically-conditioned generative model and obtaining synthetic utterances that are output by the semantically-conditioned generative model. The method or technique can also include outputting the synthetic utterances.

Abstract: This document relates to training of machine learning models. One example method involves providing a machine learning model having one or more mapping layers. The one or more mapping layers can include at least a first mapping layer configured to map components of pretraining examples into first representations in a space. The example method also includes performing a pretraining stage on the one or more mapping layers using the pretraining examples. The pretraining stage can include adding noise to the first representations of the components of the pretraining examples to obtain noise-adjusted first representations. The pretraining stage can also include performing a self-supervised learning process to pretrain the one or more mapping layers using at least the first representations of the training data items and the noise-adjusted first representations of the training data items.