Abstract: One embodiment of the present invention sets forth a technique for performing face swapping. The technique includes converting a first input image that depicts a first facial identity from a first viewpoint at a first time into a first latent representation and converting a second input image that depicts the first facial identity from a second viewpoint at the first time into a second latent representation. The technique also includes generating, via a first machine learning model, a first output image that depicts a second facial identity from the first viewpoint based on the first latent representation. The technique further includes generating, via the first machine learning model, a second output image that depicts the second facial identity from the second viewpoint based on the second latent representation.

Abstract: A system includes a machine learning (ML) model-based video downsampler configured to receive an input video sequence having a first display resolution, and to map the input video sequence to a lower resolution video sequence having a second display resolution lower than the first display resolution. The system also includes a neural network-based (NN-based) proxy video codec configured to transform the lower resolution video sequence into a decoded proxy bitstream. In addition, the system includes an upsampler configured to produce an output video sequence using the decoded proxy bitstream.

Abstract: Techniques are disclosed for resampling images. In some embodiments, a resampling model includes (1) one or more feature extraction layers that extract features from an input image and a degradation map; (2) one or more resampling layers that generate warped features from the extracted features and a warp grid; and (3) one or more prediction layers that generate, from the warped features, an output image or resampling kernels that can be applied to the input image to generate an output image. In some embodiments, the resampling model can be trained by applying degradation maps to output images in a training data set to generate corresponding input images, and training the resampling model using the input images and the corresponding output images.

Abstract: One embodiment of the present invention sets forth a technique for performing identity-preserving image generation. The technique includes converting an identity image depicting a facial identity into an identity embedding. The technique further includes generating a combined embedding based on the identity embedding and a diffusion iteration identifier. The technique further includes converting, using a neural network and based on the combined embedding, a first input image that includes first noise into a first predicted image depicting one or more facial features that include one or more first facial identity features, wherein the one or more first facial identity features correspond to one or more respective second facial identity features of the identity image and are based at least on the identity embedding.

Abstract: One embodiment of the present invention sets forth a technique for performing face swapping. The technique includes generating a latent representation of a first facial identity included in an input image. The technique further includes identifying a first identity-specific neural network layer associated with a second facial identity from a plurality of identity-specific neural network layers, wherein each neural network layer included in the plurality of identity-specific neural network layers is associated with a different facial identity. The technique further includes executing the first identity-specific neural network layer and one or more other neural network layers to generate one or more decoder input values corresponding to the latent representation. The technique further includes executing a decoder neural network that converts the one or more decoder input values into an output image depicting the second facial identity.

Abstract: An image processing system includes a computing platform having processing hardware, a display, and a system memory storing a software code. The processing hardware is configured to execute the software code to receive a three-dimensional (3D) digital model, surround the 3D digital model with multiple virtual cameras oriented toward the 3D digital model, and generate, using the virtual cameras, a multiple renders of the 3D digital model. The processing hardware is further configured to execute the software code to generate a UV texture coordinate space for a surface projection of the 3D digital model, and to transfer, using the multiple renders, lighting color values for each of multiple surface portions of the 3D digital model to the UV texture coordinate space.

Abstract: Techniques are disclosed for denoising videos. In some embodiments, video frames are denoised using a denoising model that includes an encoder-decoder architecture and attention modules. During training of the denoising model, the attention modules learn weightings to upweight certain dimensions of input features to help pixel registration, remove ghosting artifacts, and improve temporal consistency when the frames of a video are being denoised. The denoising model can also be used to train a student denoising model that has a same architecture as, but is smaller and faster than, the denoising model. After training, noisy video frames can be input into the denoising model and/or the student denoising model to generate corresponding denoised video frames.

Abstract: A computer-implemented method of changing a face within an output image or video frame that includes: receiving an input image that includes a face presenting a facial expression in a pose; processing the image with a neural network encoder to generate a latent space point that is an encoded representation of the image; decoding the latent space point to generate an initial output image in accordance with a desired facial identity but with the facial expression and pose of the face in the input image; identifying a feature of the facial expression in the initial output image to edit; applying an adjustment vector to a latent space point corresponding to the initial output image to generate an adjusted latent space point; and decoding the adjusted latent space point to generate an adjusted output image in accordance with the desired facial identity but with the facial expression and pose of the face in the input image altered in accordance with the adjustment vector

Abstract: A computer-implemented method of changing a face within an output image or video frame includes: receiving an input image that includes a face presenting a facial expression in a pose; separately encoding different portions of the image by, for each separately encoded portion, generating a latent space point of the portion, thereby generating a plurality of multi-dimensional vectors where each multi-dimensional vector is an encoded representation of a different portion of the input image; concatenating the plurality of multi-dimensional vectors into a combined latent space vector; and decoding the combined latent space vector to generate the output image in accordance with a desired facial identity but with the facial expression and pose of the face in the input image

Abstract: A system includes a machine learning (ML) model-based video downsampler configured to receive an input video sequence having a first display resolution, and to map the input video sequence to a lower resolution video sequence having a second display resolution lower than the first display resolution. The system also includes a neural network-based (NN-based) proxy video codec configured to transform the lower resolution video sequence into a decoded proxy bitstream. In addition, the system includes an upsampler configured to produce an output video sequence using the decoded proxy bitstream.

Abstract: Techniques are disclosed for enhancing videos using a machine learning model that is a temporally-consistent transformer model. The machine learning model processes blocks of frames of a video in which the temporally first input video frame of each block of frames is a temporally second to last output video frame of a previous block of frames. After the machine learning model is trained, blocks of video frames, or features extracted from the video frames, can be warped using an optical flow technique and transformed using a wavelet transform technique. The transformed video frames are concatenated along a channel dimension and input into the machine learning model that generates corresponding processed video frames.

Abstract: A system includes a computing platform having a hardware processor and a memory storing a software code and a neural network (NN) having multiple layers including a last activation layer and a loss layer. The hardware processor executes the software code to identify different combinations of layers for testing the NN, each combination including candidate function(s) for the last activation layer and candidate function(s) for the loss layer. For each different combination, the software code configures the NN based on the combination, inputs, into the configured NN, a training dataset including multiple data objects, receives, from the configured NN, a classification of the data objects, and generates a performance assessment for the combination based on the classification. The software code determines a preferred combination of layers for the NN including selected candidate functions for the last activation layer and the loss layer, based on a comparison of the performance assessments.

Abstract: One embodiment of the present invention sets forth a technique for performing remastering of video content. The technique includes determining a first input frame corresponding to a first frame included in a first video and a first target frame corresponding to a second frame included in a second video based on one or more alignments between the first frame and the second frame. The technique also includes executing a machine learning model to convert the first input frame into a first output frame. The technique further includes training the machine learning model based on one or more losses associated with the first output frame and the first target frame.

Abstract: A system includes a computing platform having a hardware processor and a memory storing a software code and a neural network (NN) having multiple layers including a last activation layer and a loss layer. The hardware processor executes the software code to identify different combinations of layers for testing the NN, each combination including candidate function(s) for the last activation layer and candidate function(s) for the loss layer. For each different combination, the software code configures the NN based on the combination, inputs, into the configured NN, a training dataset including multiple data objects, receives, from the configured NN, a classification of the data objects, and generates a performance assessment for the combination based on the classification. The software code determines a preferred combination of layers for the NN including selected candidate functions for the last activation layer and the loss layer, based on a comparison of the performance assessments.

Abstract: Restoration methods and systems are disclosed for video remastering. Techniques disclosed include receiving a video sequence. For each frame of the video sequence, techniques disclosed include encoding, by a degradation encoder, a video content associated with the frame into a latent vector. The latent vector is a representation of the degradation present in the video content; the degradation present in the video content includes one or more degradation types. Based on the latent vector and the video content, techniques disclosed further include generating, by a backbone network, one or more feature maps, and, then, restoring the frame based on the one or more feature maps.

Abstract: Various embodiments set forth systems and techniques for training a landmark model. The techniques include determining, using the landmark model, a first landmark in a set of first landmarks associated with a first image; performing, on the first image, a first perturbation to obtain a second image; determining, using the landmark model, a second landmark in a set of second landmarks associated with the second image; determining, based on a first distance between the first landmark and the second landmark, a first loss function; and updating, based on the first loss function, a first parameter of the landmark model.

Abstract: A system includes a machine learning (ML) model-based video downsampler configured to receive an input video sequence having a first display resolution, and to map the input video sequence to a lower resolution video sequence having a second display resolution lower than the first display resolution. The system also includes a neural network-based (NN-based) proxy video codec configured to transform the lower resolution video sequence into a decoded proxy bitstream. In addition, the system includes an upsampler configured to produce an output video sequence using the decoded proxy bitstream.

Abstract: A system includes a computing platform having a hardware processor, and a system memory storing a software code and a content labeling predictive model. The hardware processor is configured to execute the software code to scan a database to identify content assets stored in the database, parse metadata stored in the database to identify labels associated with the content assets, and generate a graph by creating multiple first links linking each of the content assets to its corresponding label or labels. The hardware processor is configured to further execute the software code to train, using the graph, the content labeling predictive model, to identify, using the trained content labeling predictive model, multiple second links among the content assets and the labels, and to annotate the content assets based on the second links.

Abstract: According to one implementation, a system for performing re-noising and neural network (NN) based image enhancement includes a computing platform having a processing hardware and a system memory storing a software code, a noise synthesizer, and an image restoration NN. The processing hardware is configured to execute the software code to receive a denoised image component and a noise component extracted from a degraded image, to generate, using the noise synthesizer and the noise component, synthesized noise corresponding to the noise component, and to interpolate, using the noise component and the synthesized noise, an output image noise. The processing hardware is further configured to execute the software code to enhance, using the image restoration NN, the denoised image component to provide an output image component, and to re-noise the output image component, using the output image noise, to produce an enhanced output image corresponding to the degraded image.

Abstract: One embodiment of the present invention sets forth a technique for performing deinterlacing. The technique includes separating a first interlaced video frame into a first sequence of fields ordered by time, the first sequence of fields including a first field. The technique also includes generating, by applying a deinterlacing network to a first field in the first sequence, a second field that is missing from the first sequence of fields and is complementary to the first field. The technique further includes constructing a progressive video frame based on the first field and the second field.