Abstract: A technique executes a client machine-trained model (“client model”) on a client device. In operation, the client device submits a description of a task to be performed by the client device to a network-accessible main system. The main system uses a main-system machine-trained model (“main-system model”) to produce a task prompt based on the task description. The client device subsequently uses the task prompt to process queries pertaining to the task. The main-system is trained to increase the accuracy of responses produced by the client model, while reducing the sizes of task prompts produced by the main system. The training process is performed by holding weights of the client model constant.

Abstract: A data structure describes a machine-trained model using a data structure that includes a plurality paths between a root node and respective leaf nodes. One such path is a main root-to-leaf (RTL) path, while other paths are referred to as non-main-RTL paths. Each node along the RTL path is associated with a portion of base model weights. At least one node along a non-main-RTL path is associated with a portion of model-variance information. A training system trains the portions of model-variance information as variations of corresponding portions of base model weights, while keeping the portion of base model weights fixed. In some cases, a local system obtains portions of model weights described by the data structure from a source system on an as needed-basis. The above characteristics contribute to the efficient storage, transfer, and execution of the machine-trained model.

Abstract: A technique uses a machine-trained model to generate a response based on a prompt which expresses current input information and abstract token information. The abstract token information summarizes a full dialogue history of a dialogue, and is generated by the model itself. The technique reduces the size of the prompt by incorporating the abstract summary information in lieu of the full dialogue history. A training system trains the machine-trained model by successively improving the predictive accuracy of the machine-trained model, while rewarding the machine-trained model based on an extent to which the machine-trained model compresses instances of abstract token information.

Abstract: Machine learning techniques are leveraged to provide personalized assistance on a computing device. In some configurations a timeline of a user's interactions with the computing device is generated. For example, screenshots and audio streams may be saved as entries in the timeline. Context—the state of the computing device when the entry is created, such as which documents and websites are open—is also stored. Entries in the timeline are processed by a model to generate embedding vectors. The timeline may be searched by finding the embedding vector that is closest to an embedding vector derived from a search query. The user may select a query result, causing the associated context to be restored. For example, if the query is “show me all documents related to my upcoming trip to Japan”, the query result may open documents and websites that were open when booking a flight to Japan.

Abstract: A technique uses an encoder system to produce an index of target item embeddings. Each target item embedding is input-agnostic and universal in the sense that different expressions of a target concept, produced using different combinations of input modes, map to the same target item embedding in the index. The encoder system throttles the amount of computations it performs based on the assessed capabilities of an execution platform. A retrieval system processes a multimodal input query by first generating a candidate set of target item embeddings in the index that match the input query, and then using a filtering operation to identify those target item embeddings that are most likely to match the input query. The encoder system and the retrieval system rely on language-based components having weights that are held constant during a training operation. Other weights of these systems are updated during the training operation.

Abstract: In various embodiments, a method for processing video streams is described. A plurality of video streams for transmission to a display device are received. The plurality of video streams have respective initial image quality levels. An estimated gaze location of a user of the display device is estimated. At least one video stream of the plurality of video streams is processed to have a modified image quality level based on the estimated gaze location. The modified image quality level is less than a corresponding initial image quality level. The plurality of video streams are transmitted to the display device.

Abstract: The present disclosure relates to an image restoration system that efficiently and accurately produces high-quality images captured under low-light and/or low-quality environmental conditions. To illustrate, when a user is in a low-lit environment and participating in a video stream, the image restoration system enhances the quality of the image by dynamically re-lighting the user's face. Moreover, it significantly enhances the image quality to the extent that other users viewing the video stream are unaware of the poor environmental conditions of the user. In addition, the image restoration system creates and utilizes an image restoration machine-learning model to improve the quality of low-quality images by re-lighting and restoring them in real time. Various implementations combine an autoencoder model with a distortion classifier model to create the image restoration machine-learning model.

Abstract: A technique is described herein that interprets some frames in a stream of video content as key frames and other frames as predicted frames. The technique uses an image analysis system to produce feature information for each key frame. The technique uses a prediction model to produce feature information for each predicted frame. The prediction model operates on two inputs: (1) feature information that has been computed for an immediately-preceding frame; and (2) frame-change information. A motion-determining model produces the frame-change information by computing the change in video content between the current frame being predicted and the immediately-preceding frame. The technique reduces the amount of image-processing operations that are used to process the stream of video content compared to a base case of processing all of the frames using the image analysis system. As such, the technique uses less computing resources compared to the base case.

Abstract: A technique implements a machine-trained model using resources of a local system. The technique operates by successively obtaining portions of model weights on an as-needed basis. The local system obtains at least some of the portions by downloading them from a source system in a streaming operation. The technique further successively executes parts of the machine-trained model in the local system using the portions of model weights that have been obtained, to provide an output result. An entirety of the model weights used by the local system to provide the output result is less than an entirety of the model weights available for download at the source system. The technique enables the local system to locally execute the machine-trained model without overburdening its local resources, and with reduced consumption of network resources.

Abstract: Methods and systems for generating and using a semantic index are provided. In some examples, content data is received. The content data includes a plurality of subsets of content data. Each of the plurality of subsets of content data are labelled, based on a semantic context corresponding to the content data. The plurality of subsets of content data and their corresponding labels are stored. The plurality of subsets of content data are grouped, based on their labels, thereby generating one or more groups of subsets of content data. Further, a computing device is adapted to perform an action, based on the one or more groups of subsets of content data.

Abstract: Aspects of the present disclosure relate to adjusting an illumination of a user depicted in one or more images when using a video conferencing application. In one example, one or more images depicting the user may be received from an image sensor. Further, an illumination of the user depicted in the one or more images may be determined to be unsatisfactory. For example, the user's face may be too bright or too dim. Accordingly, content displayed at a display device may identified and then modified. The modified content may then be rendered to a display device thereby changing the illumination of the user depicted in subsequent images. In examples, the modified content may include a graphical element, such as a ring of a specific color at least partially surrounding content rendered to and displayed at the display device.

Abstract: Aspects of the present disclosure relate to systems and methods for controlling a function of a computing system using gaze detection. In examples, one or more images of a user are received and gaze information may be determined from the received one or more images. Non-gaze information may be received when the gaze information is determined to satisfy a condition. Accordingly, a function may be enabled based on the received non-gaze information. In examples, the gaze information may be determined by extracting a plurality of features from the received one or more images, providing the plurality of features to a neural network, and determining, utilizing the neural network, a location at a display device at which a gaze of the user is directed.

Abstract: A technique executes tasks using a data store of machine-trained models. The data store specifically includes a subset of encoder-type machine-trained models for converting input data items having different input data types into respective embeddings in a vector space, and a subset of decoder-type machine-trained models for converting embeddings in the same vector space into data items having respective different output data types. When executing a particular task that involves one or more data types, the technique selects one or more machine-trained models that match those data types. In some implementations, the technique provides a clipboard store for storing embeddings produced by the encoder-type machine-trained models and consumable by the decoder-type machine-trained models. The technique includes provisions for ensuring that any decoder-type machine-model is capable of processing embeddings produced by different versions of the encoder-type machine-trained models.

Abstract: Methods and systems for generating and using a semantic index are provided. In some examples, content data is received. The content data includes a plurality of subsets of content data. Each of the plurality of subsets of content data are labelled, based on a semantic context corresponding to the content data. The plurality of subsets of content data and their corresponding labels are stored. The plurality of subsets of content data are grouped, based on their labels, thereby generating one or more groups of subsets of content data. Further, a computing device is adapted to perform an action, based on the one or more groups of subsets of content data.

Abstract: Methods and systems for generating and using a semantic index are provided. In some examples, content data is received. The content data includes a plurality of subsets of content data. Each of the plurality of subsets of content data are labelled, based on a semantic context corresponding to the content data. The plurality of subsets of content data and their corresponding labels are stored. The plurality of subsets of content data are grouped, based on their labels, thereby generating one or more groups of subsets of content data. Further, a computing device is adapted to perform an action, based on the one or more groups of subsets of content data.

Abstract: An image-processing technique is described herein for removing a visual effect in a face region of an image caused, at least in part, by screen illumination provided by an electronic screen. The technique can perform this removal without advance knowledge of the nature of the screen illumination provided by the electronic screen. The technique improves the quality of the image and also protects the privacy of a user by removing the visual effect in the face region that may reveal the characteristics of display information presented on the electronic screen. In some implementations, the technique first adjusts a face region of the image, and then adjusts other regions in the image for consistency with the face region. In some implementations, the technique is applied by a videoconferencing application, and is performed by a local computing device.

Abstract: In various embodiments, a computer-implemented method of training a neural network for relighting an image is described. A first training set that includes source images and a target illumination embedding is generated, the source images having respective illuminated subjects. A second training set that includes augmented images and the target illumination embedding is generated, where the augmented images corresponding to the source images. A first autoencoder is trained using the first training set to generate a first output set that includes estimated source illumination embeddings and first reconstructed images that correspond to the source images, the reconstructed images having respective subjects that are i) from the corresponding source image, and ii) illuminated based on the target illumination embedding.

Abstract: In various embodiments, a computer-implemented method of training a neural network for creating an output signal of different modality from an input signal is described. In embodiments, the first modality may be a sound signal or a visual image and where the output signal would be a visual image or a sound signal, respectively. In embodiments a model is trained using a first pair of visual and audio networks to train a set of codebooks using known visual signals and the audio signals and using a second pair of visual and audio networks to further train the set of codebooks using the augmented visual signals and the augmented audio signals. Further, the first and the second visual networks are equally weighted and where the first and the second audio networks are equally weighted.

Abstract: Methods and systems for applying gaze adjustment techniques to participants in a video conference are disclosed. Some examples may include: receiving, at computing system, image adjustment information associated with a video stream including images of a first participant, identifying, for a display layout of a communication application, a location displaying the images of the first participant, determining, based on the received image adjustment information, a location displaying images of a second participant for the display layout, the received image adjustment information indicating that an eye gaze of the first participant being directed toward the second participant, computing an eye gaze direction of the first participant based on the location displaying images of the second participant, generating gaze-adjusted images based on the desired eye gaze direction of the first participant and replacing the images within the video stream with the gaze-adjusted images.

Abstract: In various embodiments, a method for processing video streams is described. A plurality of video streams for transmission to a display device are received. The plurality of video streams have respective initial image quality levels. An estimated gaze location of a user of the display device is estimated. At least one video stream of the plurality of video streams is processed to have a modified image quality level based on the estimated gaze location. The modified image quality level is less than a corresponding initial image quality level. The plurality of video streams are transmitted to the display device.