Abstract: Methods and apparatuses are provided for processing video data by using generative face video supplemental enhancement information (SEI) messages. An exemplary method for generating a face picture includes: receiving a bitstream; decoding coded information of the bitstream to obtain a base picture and a supplemental enhancement information (SEI) message; determining whether the SEI message applies to a neural network for generating a face picture; in response to the SEI message applies to the neural network for generating the face picture, determining a mode and a corresponding face information parameter used to code the face picture based on the SEI message; and generating the face picture based on the base picture and the face information parameter by the neural network.

Abstract: A method of decoding a bitstream to output one or more pictures for a video stream, includes: receiving a bitstream; and decoding, using coded information of the bitstream, one or more pictures. The decoding includes: determining, based on an identifying number, whether a face video generative compression scheme is used; in response to a determination that the face video generative compression scheme is used, decoding a supplemental enhancement information (SEI) message, the SEI message comprising facial information; and reconstructing a face picture based on the facial information and a base picture associated with the SEI message.

Abstract: A video decoding method includes decoding an image bitstream associated with a video sequence to obtain a reconstructed key frame; extracting features of the reconstructed key frame; decoding a feature bitstream associated with the video sequence to obtain a motion token; reconstructing a dense motion based on the features of the reconstructed key frame and the motion token; and generating video content based on the reconstructed dense motion.

Abstract: A video decoding method includes decoding an image bitstream associated with a video sequence, wherein the decoding of the image bitstream reconstructs a key reference frame; factorizing the reconstructed key reference frame into a key frame latent feature and a first group of compact motion vectors associated with the reconstructed key reference frame; decoding a feature bitstream associated with the video sequence to obtain a second group of compact motion vectors associated with an inter frame; transforming, based on the first group and second group of compact motion vectors, the key frame latent feature into a first fine-grained motion field for the reconstructed key reference frame and a second fine-grained motion field for the inter frame; predicting a dense motion based on the first and second fine-grained motion fields; and generating the inter frame based on the dense motion and the reconstructed key reference frame.

Abstract: A video encoding method includes: receiving a video sequence including a first key frame and one or more inter frames following the first key frame; generating a reconstructed key frame corresponding to the first key frame; transforming the reconstructed key frame and the one or more inter frames to visual tokens with different granularities; and encoding one or more token bitstreams including coded information for one or more of the visual tokens selected based on a data transmission bandwidth.

Abstract: A video decoding method includes: decoding an image bitstream associated with a video sequence to reconstruct a key frame of the video sequence and obtain extracted features of the reconstructed key frame; decoding a feature bitstream associated with the video sequence to obtain extracted features of one or more inter frames of the video sequence; obtaining motion information and occlusion information based on the extracted features of the reconstructed key frame and the extracted features of the one or more inter frames; resampling, by a neural network, the reconstructed key frame based on the motion information and occlusion information by a neural network; and reconstructing, by the neural network, the video sequence based on the resampled reconstructed key frame. A network width and a network depth of the neural network is adjusted in response to an input resolution.

Abstract: A method for decoding a bitstream includes: receiving a bitstream and decoding, using coded information of the bitstream, one or more pictures. The decoding of the one or more pictures includes: determining whether a generative face video supplemental enhancement information (SEI) message matches with a generative network; and in response to the generative face video SEI message matches with the generative network, decoding the SEI message. The decoding of the SEI message includes: determining a face information parameter and a base picture associated with the SEI message; and reconstructing a face picture based on the face information parameter and the base picture.

Abstract: Methods and apparatuses are provided for compressing video data based on keypoint features. An exemplary video compression method includes: receiving a video sequence; encoding one or more pictures of the video sequence; and generating a bitstream; wherein the encoding includes: representing a first picture by a first set of keypoints and a second set of keypoints, the second set comprising less keypoints than the first set; and compressing the video sequence based on the first set and second set of keypoints.

Abstract: Methods and apparatuses are provided for processing video data. An exemplary method includes: decompressing a compressed frame to generate a key frame representing a face; generating, for the key frame, a first set of parameters associated with a 3-dimensional (3D) face representation of the face; reconstructing, for each of one or more inter frames, a second set of parameters associated with a 3D face representation of the face according to compressed inter-predicted residuals of the second set of parameters; and generating a video comprising the face based on the key frame, the first set of parameters, and the second set of parameters.

Abstract: Methods and systems implement a pleno-generation face video compression framework with bandwidth intelligence for generative models and compression. Heterogeneous-granularity facial description regularizes long-term dependencies between video frames and compensates for motion estimation errors caused by compact representations of motion information. A generative decoder reconstructs heterogeneous-granularity visual representations, providing auxiliary visual signals for attention-based recalibration of a GFVC-reconstructed face signal. A coarse-to-fine generation strategy avoids error accumulation.

Abstract: Generative Face Video Compression (“GFVC”) techniques are provided to improve performance of facial video compression. A computing system is configured to perform GFVC upon heterogeneous-resolution sequences based on consistent resampling factors and based on adaptive resampling factors. Adaptive resampling factors are further implemented by: interpolation of heterogeneous-resolution sequences in GFVC to simplify resolution unification; multi-scale architecture of feature extractors in GFVC to capture details across heterogeneous resolutions by integrating multiple processing layers; and adapting dynamic neural networks in real-time to process varying input resolutions of heterogeneous-resolution sequences in GFVC efficiently.

Abstract: Methods are provided for scalable generative video coding. An exemplary video decoding method includes: receiving a bitstream; decoding a supplemental enhancement information (SEI) message that is associated with a picture from the bitstream; and generating the picture based on the SEI message.

Abstract: Methods and systems implement a progressive generative face video compression framework with bandwidth intelligence, hierarchically accommodating variable bitrate video communication and implementing high-fidelity face reconstruction towards overall bandwidth coverage. Heterogeneous-granularity facial description regularizes long-term dependencies between video frames and compensates for motion estimation errors caused by compact representations of motion information, achieving satisfactory human visual perception and bandwidth intelligence in a progressive fashion.

Abstract: Signaling methods for scalable generative video coding are provided. An exemplary video decoding method includes: decoding a first supplemental enhancement information (SEI) message that is associated with a facial image; and enhancing the facial image based on the first SEI message.

Abstract: Generative Face Video Compression (“GFVC”) techniques are provided to improve performance of facial video compression. A computing system is configured to compute a relative difference metric describing differences in features between frames, and determining, based on the relative difference metric, whether a current frame can be synthesized without entropy coding, or should be re-coded. A computing system is configured to perform two-stage training to stabilize Generative Adversarial Networks (“GAN”) training in GFVC.

Abstract: The present disclosure provides a video encoding method, a decoding method, and an apparatus. The video encoding method includes: obtaining an original reference video frame and an original target video frame to be encoded; adjusting a resolution of the original target video frame to obtain an adjusted target video frame with a first preset resolution; and performing feature extraction on the adjusted target video frame to obtain a target feature through a feature extraction network corresponding to the first preset resolution; encoding the original reference video frame and the target features respectively to obtain a video bitstream, and performing video frame reconstruction based on the video bitstream to generate a reconstructed video frame with a same resolution as the original target video frame.

Abstract: A method including deforming the reference sample frame through generator in an initial generative model to generate reconstructed sample frames; inputting each reconstructed sample frame and the corresponding to-be-encoded sample frame into a first discriminator in the initial generative model to obtain a first identification result; splicing the to-be-encoded sample frames in timestamp order to obtain a spliced to-be-encoded sample frame, and splicing the reconstructed sample frames to obtain a spliced reconstructed sample frame; inputting the spliced to-be-encoded sample frame and the spliced reconstructed sample frame into a second discriminator in the initial generative model to obtain a second identification result; obtaining an adversarial loss value based on the first identification result and the second identification result; and training the initial generative model based on the adversarial loss value.

Abstract: A method of decoding a bitstream to output one or more pictures for a video stream. The method includes receiving a bitstream comprising one or more types of facial representation parameters; and decoding, using coded information of the bitstream, one or more pictures. The decoding includes decoding the one or more types of facial representation parameters; converting the one or more types of facial representation parameters into one or more dense motion flows having a common format; and generating a facial picture based on the one or more dense motion flows and a key reference picture of the one or more pictures.

Abstract: Methods and apparatuses are provided for performing generative face video compression by using a face feature translator. An exemplary method includes receiving a bitstream associated with a first type of facial feature data representing a facial picture; and decoding, using coded information of the bitstream, one or more pictures, wherein the decoding includes: transforming the first type of facial feature data into a second type of facial feature data; and reconstructing the facial picture based on the second type of facial feature data.

Abstract: A method of decoding a bitstream to get one or more pictures for a video stream includes: receiving a bitstream; and decoding the bitstream to get the one or more pictures. The decoding includes: decoding a picture unit comprising one or more supplemental enhancement information (SEI) messages; and generating the one or more pictures based on a key picture and the one or more SEI messages, respectively.