Abstract: The present disclosure relates to an image restoration system that efficiently and accurately produces high-quality images captured under low-light and/or low-quality environmental conditions. To illustrate, when a user is in a low-lit environment and participating in a video stream, the image restoration system enhances the quality of the image by dynamically re-lighting the user's face. Moreover, it significantly enhances the image quality to the extent that other users viewing the video stream are unaware of the poor environmental conditions of the user. In addition, the image restoration system creates and utilizes an image restoration machine-learning model to improve the quality of low-quality images by re-lighting and restoring them in real time. Various implementations combine an autoencoder model with a distortion classifier model to create the image restoration machine-learning model.

Abstract: A graph-forming process generates a graph having nodes that represent a plurality of previously captured screenshots. The graph-forming process relies on a plurality of machine-trained models to identify edges between pairs of the nodes. The edges represent relationships among the screenshots. The graph-forming process then trains a graph neural network (GNN) based on the graph. The training produces a plurality of target embeddings associated with respective nodes in the graph. A retrieval process retrieves a previously captured screenshot using the plurality of target embeddings. The retrieval process involves adding a new node to the graph that represents the query and using the GNN to produce a query embedding associated with the new node. The retrieval process then finds at least one target embedding that matches the query embedding and retrieves a screenshot associated with the matching target embedding.

Abstract: A technique uses a local language model, implemented by a local computing device, to customize original information based on a profile of a local entity (e.g., a user). A profile-generating system produces the profile by first using plural machine-trained local encoders to convert a set of content items to instances of local encoded information. The profile-generating system then uses a global encoder to convert the plural instances of local encoded information into profile information that expresses the profile. The technique passes a combination of the original information and the profile information to the local language model, optionally with level information that specifies an extent of customization to be applied to the original information. In some implementations, the original information originates from another language model that is larger than the local language model.

Abstract: In various embodiments, a computer-implemented method of training a neural network for creating an output signal of different modality from an input signal is described. In embodiments, the first modality may be a sound signal or a visual image and where the output signal would be a visual image or a sound signal, respectively. In embodiments a model is trained using a first pair of visual and audio networks to train a set of codebooks using known visual signals and the audio signals and using a second pair of visual and audio networks to further train the set of codebooks using the augmented visual signals and the augmented audio signals. Further, the first and the second visual networks are equally weighted and where the first and the second audio networks are equally weighted.

Abstract: The present disclosure relates to an image restoration system that efficiently and accurately produces high-quality images captured under low-light and/or low-quality environmental conditions. To illustrate, when a user is in a low-lit environment and participating in a video stream, the image restoration system enhances the quality of the image by dynamically re-lighting the user's face. Moreover, it significantly enhances the image quality to the extent that other users viewing the video stream are unaware of the poor environmental conditions of the user. In addition, the image restoration system creates and utilizes an image restoration machine-learning model to improve the quality of low-quality images by re-lighting and restoring them in real time. Various implementations combine an autoencoder model with a distortion classifier model to create the image restoration machine-learning model.

Abstract: The present disclosure relates identifying an intended viewer and an unintended viewer of a liquid crystal display (LCD) using face recognition technology. Once identified the system may determine a face position for the unintended viewer. The system may modulate the voltage applied at a third electrode on the color filter layer of the LCD to achieve a certain off-axis contrast that may reduce the unintended viewer's visibility of the LCD without restricting the visibility of the intended viewer. Ultimately, the present disclosure provides enhanced privacy options for the intended viewer with a lightweight, inexpensive, and highly transportable system.

Abstract: An image-processing technique is described herein for removing a visual effect in a face region of an image caused, at least in part, by screen illumination provided by an electronic screen. The technique can perform this removal without advance knowledge of the nature of the screen illumination provided by the electronic screen. The technique improves the quality of the image and also protects the privacy of a user by removing the visual effect in the face region that may reveal the characteristics of display information presented on the electronic screen. In some implementations, the technique first adjusts a face region of the image, and then adjusts other regions in the image for consistency with the face region. In some implementations, the technique is applied by a videoconferencing application, and is performed by a local computing device.

Abstract: In various embodiments, a computer-implemented method of training a neural network for relighting an image is described. A first training set that includes source images and a target illumination embedding is generated, the source images having respective illuminated subjects. A second training set that includes augmented images and the target illumination embedding is generated, where the augmented images corresponding to the source images. A first autoencoder is trained using the first training set to generate a first output set that includes estimated source illumination embeddings and first reconstructed images that correspond to the source images, the reconstructed images having respective subjects that are i) from the corresponding source image, and ii) illuminated based on the target illumination embedding.

Abstract: In various embodiments, a computer-implemented method of training a neural network for creating an output signal of different modality from an input signal is described. In embodiments, the first modality may be a sound signal or a visual image and where the output signal would be a visual image or a sound signal, respectively. In embodiments a model is trained using a first pair of visual and audio networks to train a set of codebooks using known visual signals and the audio signals and using a second pair of visual and audio networks to further train the set of codebooks using the augmented visual signals and the augmented audio signals. Further, the first and the second visual networks are equally weighted and where the first and the second audio networks are equally weighted.

Abstract: Methods and systems for applying gaze adjustment techniques to participants in a video conference are disclosed. Some examples may include: receiving, at computing system, image adjustment information associated with a video stream including images of a first participant, identifying, for a display layout of a communication application, a location displaying the images of the first participant, determining, based on the received image adjustment information, a location displaying images of a second participant for the display layout, the received image adjustment information indicating that an eye gaze of the first participant being directed toward the second participant, computing an eye gaze direction of the first participant based on the location displaying images of the second participant, generating gaze-adjusted images based on the desired eye gaze direction of the first participant and replacing the images within the video stream with the gaze-adjusted images.

Abstract: In various embodiments, a computer-implemented method of training a neural network for creating an output signal of different modality from an input signal is described. In embodiments, the first modality may be a sound signal or a visual image and where the output signal would be a visual image or a sound signal, respectively. In embodiments a model is trained using a first pair of visual and audio networks to train a set of codebooks using known visual signals and the audio signals and using a second pair of visual and audio networks to further train the set of codebooks using the augmented visual signals and the augmented audio signals. Further, the first and the second visual networks are equally weighted and where the first and the second audio networks are equally weighted. In aspects of the present disclosure, the set of codebooks comprise a visual codebook, an audio codebook and a correlation codebook.

Abstract: Methods and systems for applying gaze adjustment techniques to participants in a video conference are disclosed. Some examples may include: receiving, at computing system, image adjustment information associated with a video stream including images of a first participant, identifying, for a display layout of a communication application, a location displaying the images of the first participant, determining, based on the received image adjustment information, a location displaying images of a second participant for the display layout, the received image adjustment information indicating that an eye gaze of the first participant being directed toward the second participant, computing an eye gaze direction of the first participant based on the location displaying images of the second participant, generating gaze-adjusted images based on the desired eye gaze direction of the first participant and replacing the images within the video stream with the gaze-adjusted images.

Abstract: Aspects of the present disclosure relate to video stream refinement for a dynamic scene. In examples, a system is provided that includes at least one processor, and memory storing instructions that, when executed by the at least one processor, causes the system to perform a set of operations. The set of operations include receiving an input video stream, identifying, within the input video stream, a frame portion containing features of interest, enlarging the frame portion containing the features of interest, enhancing the frame portion of the input video stream to increase fidelity within the frame portion, and displaying the enhanced frame portion.

Abstract: A method of improving image quality of a stream of input images is described. The stream of input images, including a current input image, is received. One or more target objects, including a first target object, are identified spatio-temporally within the stream of input images. The one or more target objects are tracked spatio-temporally within the stream of input images. The current input image is segmented into i) a foreground including the first target object, and ii) a background. The foreground is processed to have improved image quality in the current input image. Processing of the foreground further comprises processing the first target object using a same processing technique as for a prior input image of the stream of input images based on the tracking of the first target object. The background is processed differently from the foreground. An output image is generated by merging the foreground with the background.

Abstract: Methods and systems for applying gaze adjustment techniques to participants in a video conference are disclosed. Some examples may include: receiving, at computing system, image adjustment information associated with a video stream including images of a first participant, identifying, for a display layout of a communication application, a location displaying the images of the first participant, determining, based on the received image adjustment information, a location displaying images of a second participant for the display layout, the received image adjustment information indicating that an eye gaze of the first participant being directed toward the second participant, computing an eye gaze direction of the first participant based on the location displaying images of the second participant, generating gaze-adjusted images based on the desired eye gaze direction of the first participant and replacing the images within the video stream with the gaze-adjusted images.

Abstract: In various embodiments, a computer-implemented method of training a neural network for relighting an image is described. A first training set that includes source images and a target illumination embedding is generated, the source images having respective illuminated subjects. A second training set that includes augmented images and the target illumination embedding is generated, where the augmented images corresponding to the source images. A first autoencoder is trained using the first training set to generate a first output set that includes estimated source illumination embeddings and first reconstructed images that correspond to the source images, the reconstructed images having respective subjects that are i) from the corresponding source image, and ii) illuminated based on the target illumination embedding.

Abstract: A computing system includes an encoder that receives an input image and encodes the input image into real image features, a decoder that decodes the real image features into a reconstructed image, a generator that receives first audio data corresponding to the input image and generates first synthetic image features from the first audio data, and receives second audio data and generates second synthetic image features from the second audio data, a discriminator that receives both the real and synthetic image features and determines whether a target feature is real or synthetic, and a classifier that classifies a scene of the second audio data based on the second synthetic image features.

Abstract: In various embodiments, a computer-implemented method of training a neural network for relighting an image is described. A first training set that includes source images and a target illumination embedding is generated, the source images having respective illuminated subjects. A second training set that includes augmented images and the target illumination embedding is generated, where the augmented images corresponding to the source images. A first autoencoder is trained using the first training set to generate a first output set that includes estimated source illumination embeddings and first reconstructed images that correspond to the source images, the reconstructed images having respective subjects that are i) from the corresponding source image, and ii) illuminated based on the target illumination embedding.

Abstract: Methods and systems for applying gaze adjustment techniques to participants in a video conference are disclosed. Some examples may include: receiving, at computing system, image adjustment information associated with a video stream including images of a first participant, identifying, for a display layout of a communication application, a location displaying the images of the first participant, determining, based on the received image adjustment information, a location displaying images of a second participant for the display layout, the received image adjustment information indicating that an eye gaze of the first participant being directed toward the second participant, computing an eye gaze direction of the first participant based on the location displaying images of the second participant, generating gaze-adjusted images based on the desired eye gaze direction of the first participant and replacing the images within the video stream with the gaze-adjusted images.

Abstract: A method of improving image quality of a stream of input images is described. The stream of input images, including a current input image, is received. One or more target objects, including a first target object, are identified spatio-temporally within the stream of input images. The one or more target objects are tracked spatio-temporally within the stream of input images. The current input image is segmented into i) a foreground including the first target object, and ii) a background. The foreground is processed to have improved image quality in the current input image. Processing of the foreground further comprises processing the first target object using a same processing technique as for a prior input image of the stream of input images based on the tracking of the first target object. The background is processed differently from the foreground. An output image is generated by merging the foreground with the background.