Abstract: In various embodiments, a method for processing video streams is described. A plurality of video streams for transmission to a display device are received. The plurality of video streams have respective initial image quality levels. An estimated gaze location of a user of the display device is estimated. At least one video stream of the plurality of video streams is processed to have a modified image quality level based on the estimated gaze location. The modified image quality level is less than a corresponding initial image quality level. The plurality of video streams are transmitted to the display device.

Abstract: In various embodiments, a computer-implemented method of training a neural network for creating an output signal of different modality from an input signal is described. In embodiments, the first modality may be a sound signal or a visual image and where the output signal would be a visual image or a sound signal, respectively. In embodiments a model is trained using a first pair of visual and audio networks to train a set of codebooks using known visual signals and the audio signals and using a second pair of visual and audio networks to further train the set of codebooks using the augmented visual signals and the augmented audio signals. Further, the first and the second visual networks are equally weighted and where the first and the second audio networks are equally weighted. In aspects of the present disclosure, the set of codebooks comprise a visual codebook, an audio codebook and a correlation codebook.

Abstract: Methods and systems for applying gaze adjustment techniques to participants in a video conference are disclosed. Some examples may include: receiving, at computing system, image adjustment information associated with a video stream including images of a first participant, identifying, for a display layout of a communication application, a location displaying the images of the first participant, determining, based on the received image adjustment information, a location displaying images of a second participant for the display layout, the received image adjustment information indicating that an eye gaze of the first participant being directed toward the second participant, computing an eye gaze direction of the first participant based on the location displaying images of the second participant, generating gaze-adjusted images based on the desired eye gaze direction of the first participant and replacing the images within the video stream with the gaze-adjusted images.

Abstract: In various embodiments, a method for processing video streams is described. A plurality of video streams for transmission to a display device are received. The plurality of video streams have respective initial image quality levels. An estimated gaze location of a user of the display device is estimated. At least one video stream of the plurality of video streams is processed to have a modified image quality level based on the estimated gaze location. The modified image quality level is less than a corresponding initial image quality level. The plurality of video streams are transmitted to the display device.

Abstract: A method of improving image quality of a stream of input images is described. The stream of input images, including a current input image, is received. One or more target objects, including a first target object, are identified spatio-temporally within the stream of input images. The one or more target objects are tracked spatio-temporally within the stream of input images. The current input image is segmented into i) a foreground including the first target object, and ii) a background. The foreground is processed to have improved image quality in the current input image. Processing of the foreground further comprises processing the first target object using a same processing technique as for a prior input image of the stream of input images based on the tracking of the first target object. The background is processed differently from the foreground. An output image is generated by merging the foreground with the background.

Abstract: Methods and systems for applying gaze adjustment techniques to participants in a video conference are disclosed. Some examples may include: receiving, at computing system, image adjustment information associated with a video stream including images of a first participant, identifying, for a display layout of a communication application, a location displaying the images of the first participant, determining, based on the received image adjustment information, a location displaying images of a second participant for the display layout, the received image adjustment information indicating that an eye gaze of the first participant being directed toward the second participant, computing an eye gaze direction of the first participant based on the location displaying images of the second participant, generating gaze-adjusted images based on the desired eye gaze direction of the first participant and replacing the images within the video stream with the gaze-adjusted images.

Abstract: In various embodiments, a computer-implemented method of training a neural network for relighting an image is described. A first training set that includes source images and a target illumination embedding is generated, the source images having respective illuminated subjects. A second training set that includes augmented images and the target illumination embedding is generated, where the augmented images corresponding to the source images. A first autoencoder is trained using the first training set to generate a first output set that includes estimated source illumination embeddings and first reconstructed images that correspond to the source images, the reconstructed images having respective subjects that are i) from the corresponding source image, and ii) illuminated based on the target illumination embedding.

Abstract: A computing system includes an encoder that receives an input image and encodes the input image into real image features, a decoder that decodes the real image features into a reconstructed image, a generator that receives first audio data corresponding to the input image and generates first synthetic image features from the first audio data, and receives second audio data and generates second synthetic image features from the second audio data, a discriminator that receives both the real and synthetic image features and determines whether a target feature is real or synthetic, and a classifier that classifies a scene of the second audio data based on the second synthetic image features.

Abstract: A transformer is described herein for using transformer-based technology to process data items (e.g., image items). The transformer increases the efficiency of the transformer-based technology by using a modified attention component. In operation, the modified attention component accepts embedding vectors that represent a plurality of item tokens, together with a classification token. A first stage of the modified attention component generates original attention information based on the embedding vectors. A second stage generates score information based on a portion of the original attention information that pertains to the classification token. A third stage produces modified attention information by removing attention values from the original attention information, as guided by a sampling operation that is performed on the score information. The second and third stages do not rely on machine-trained values, which expedites the deployment of these functions in existing transformers.

Abstract: Aspects of the present disclosure relate to adjusting an illumination of a user depicted in one or more images when using a video conferencing application. In one example, one or more images depicting the user may be received from an image sensor. Further, an illumination of the user depicted in the one or more images may be determined to be unsatisfactory. For example, the user's face may be too bright or too dim. Accordingly, content displayed at a display device may identified and then modified. The modified content may then be rendered to a display device thereby changing the illumination of the user depicted in subsequent images. In examples, the modified content may include a graphical element, such as a ring of a specific color at least partially surrounding content rendered to and displayed at the display device.

Abstract: In various embodiments, a computer-implemented method of training a neural network for relighting an image is described. A first training set that includes source images and a target illumination embedding is generated, the source images having respective illuminated subjects. A second training set that includes augmented images and the target illumination embedding is generated, where the augmented images corresponding to the source images. A first autoencoder is trained using the first training set to generate a first output set that includes estimated source illumination embeddings and first reconstructed images that correspond to the source images, the reconstructed images having respective subjects that are i) from the corresponding source image, and ii) illuminated based on the target illumination embedding.

Abstract: Aspects of the present disclosure relate to adjusting an illumination of a user depicted in one or more images when using a video conferencing application. In one example, one or more images depicting the user may be received from an image sensor. Further, an illumination of the user depicted in the one or more images may be determined to be unsatisfactory. For example, the user's face may be too bright or too dim. Accordingly, content displayed at a display device may identified and then modified. The modified content may then be rendered to a display device thereby changing the illumination of the user depicted in subsequent images. In examples, the modified content may include a graphical element, such as a ring of a specific color at least partially surrounding content rendered to and displayed at the display device.

Abstract: Methods and systems for applying gaze adjustment techniques to participants in a video conference are disclosed. Some examples may include: receiving, at computing system, image adjustment information associated with a video stream including images of a first participant, identifying, for a display layout of a communication application, a location displaying the images of the first participant, determining, based on the received image adjustment information, a location displaying images of a second participant for the display layout, the received image adjustment information indicating that an eye gaze of the first participant being directed toward the second participant, computing an eye gaze direction of the first participant based on the location displaying images of the second participant, generating gaze-adjusted images based on the desired eye gaze direction of the first participant and replacing the images within the video stream with the gaze-adjusted images.

Abstract: A method of improving image quality of a stream of input images is described. The stream of input images, including a current input image, is received. One or more target objects, including a first target object, are identified spatio-temporally within the stream of input images. The one or more target objects are tracked spatio-temporally within the stream of input images. The current input image is segmented into i) a foreground including the first target object, and ii) a background. The foreground is processed to have improved image quality in the current input image. Processing of the foreground further comprises processing the first target object using a same processing technique as for a prior input image of the stream of input images based on the tracking of the first target object. The background is processed differently from the foreground. An output image is generated by merging the foreground with the background.

Abstract: In various embodiments, a computer-implemented method of training a neural network for creating an output signal of different modality from an input signal is described. In embodiments, the first modality may be a sound signal or a visual image and where the output signal would be a visual image or a sound signal, respectively. In embodiments a model is trained using a first pair of visual and audio networks to train a set of codebooks using known visual signals and the audio signals and using a second pair of visual and audio networks to further train the set of codebooks using the augmented visual signals and the augmented audio signals. Further, the first and the second visual networks are equally weighted and where the first and the second audio networks are equally weighted. In aspects of the present disclosure, the set of codebooks comprise a visual codebook, an audio codebook and a correlation codebook.

Abstract: In various embodiments, a method for processing video streams is described. A plurality of video streams for transmission to a display device are received. The plurality of video streams have respective initial image quality levels. An estimated gaze location of a user of the display device is estimated. At least one video stream of the plurality of video streams is processed to have a modified image quality level based on the estimated gaze location. The modified image quality level is less than a corresponding initial image quality level. The plurality of video streams are transmitted to the display device.

Abstract: In various embodiments, a computer-implemented method of training a neural network for relighting an image is described. A first training set that includes source images and a target illumination embedding is generated, the source images having respective illuminated subjects. A second training set that includes augmented images and the target illumination embedding is generated, where the augmented images corresponding to the source images. A first autoencoder is trained using the first training set to generate a first output set that includes estimated source illumination embeddings and first reconstructed images that correspond to the source images, the reconstructed images having respective subjects that are i) from the corresponding source image, and ii) illuminated based on the target illumination embedding.

Abstract: Aspects of the present disclosure relate to adjusting an illumination of a user depicted in one or more images when using a video conferencing application. In one example, one or more images depicting the user may be received from an image sensor. Further, an illumination of the user depicted in the one or more images may be determined to be unsatisfactory. For example, the user's face may be too bright or too dim. Accordingly, content displayed at a display device may identified and then modified. The modified content may then be rendered to a display device thereby changing the illumination of the user depicted in subsequent images. In examples, the modified content may include a graphical element, such as a ring of a specific color at least partially surrounding content rendered to and displayed at the display device.

Abstract: Aspects of the present disclosure relate to systems and methods for controlling a function of a computing system using gaze detection. In examples, one or more images of a user are received and gaze information may be determined from the received one or more images. Non-gaze information may be received when the gaze information is determined to satisfy a condition. Accordingly, a function may be enabled based on the received non-gaze information. In examples, the gaze information may be determined by extracting a plurality of features from the received one or more images, providing the plurality of features to a neural network, and determining, utilizing the neural network, a location at a display device at which a gaze of the user is directed.

Abstract: Technology is described herein that uses an object-encoding system to convert an object image into a combined encoding. The object image depicts a reference object, while the combined encoding represents an environment image. The environment image, in turn, depicts an estimate of an environment that has produced the illumination effects exhibited by the reference object. The combined encoding includes: a first part that represents image content in the environment image within a high range of intensities values; and a second part that represents image content within a low range of intensity values. Also described herein is a training system that trains the object-encoding system based on combined encodings produced by a separately-trained environment-encoding system. Also described herein are various applications of the object-encoding system and environment-encoding system.