Abstract: A method for performing content-based video compression using reinforcement learning (RL) is provided. The method includes obtaining frame information associated with a frame from a video. The frame information comprises quantization parameter (QP) information associated with the frame, and the QP information indicates an initial compression level for encoding aspects of the frame. The frame information and additional information are processed by an RL agent to generate a generated QP map indicating a plurality of updated values associated with a plurality of macro-blocks (MBs) of the frame. A bitstream is generated comprising a plurality of bits for the frame based on the generated QP map. Specifically, the plurality of updated values from the generated QP map indicates an amount of allocated bits from the bitstream to allocate for each of the plurality of MBs. The bitstream is provided to a downstream model.

Abstract: Systems and methods herein are for a video encoder to be associated with a rate optimization distortion (RDO) module and a calibration module, where the RDO module may be to perform RDO for received frames of a media stream and may be to generate at least an RDO output that is based in part on quality measures between the received frames and decoded frames, and where the calibration module may be to provide an evaluation metric that is to scale or transform at least a range of the quality measures, with the scaling or transforming to potentially reduce an effect on a compression performed in the video encoder.

Abstract: A method for performing content-based video compression using reinforcement learning (RL) for video rate control for a downstream task is provided. The method includes processing frame information associated with a raw frame from a video using an RL agent to generate quantization parameter (QP) information that indicates one or more values associated with a compression level of the raw frame and encoding the raw frame into a bitstream based on the QP information. The method further includes reconstructing the raw frame using the bitstream to obtain a reconstructed frame and processing the raw frame as well as the reconstructed frame using a pre-trained downstream model to generate two outputs. The method then includes determining a downstream task reward based on the two outputs and training the RL agent based on the downstream task reward.

Abstract: A method for performing content-based video compression using reinforcement learning (RL) is provided. The method includes obtaining frame information associated with a frame from a video. The frame information comprises quantization parameter (QP) information associated with the frame, and the QP information indicates an initial compression level for encoding aspects of the frame. The frame information and additional information are processed by an RL agent to generate a generated QP map indicating a plurality of updated values associated with a plurality of macro-blocks (MBs) of the frame. A bitstream is generated comprising a plurality of bits for the frame based on the generated QP map. Specifically, the plurality of updated values from the generated QP map indicates an amount of allocated bits from the bitstream to allocate for each of the plurality of MBs. The bitstream is provided to a downstream model.

Abstract: Systems and methods herein are for at least one execution unit that can perform an inference using a machine learning (ML) model and that is coupled to a video encoder, where the ML model can determine a genre associated with received frames of a media stream based in part on using ML model features associated with different genres, where the video encoder can encode the media stream based in part on the determined genre.

Abstract: A system includes a processing device to receive video content, metadata related to the video content, and a target bit rate for encoding the video content. The processing device further detects a content type of the video content based on the metadata and encodes hardware to perform frame encoding on the video content. The system further includes a controller coupled between the processing device and the encoding hardware. The controller is programmed with machine instructions to generate first QP values on a per-frame basis using a frame machine learning model with a first plurality of weights. The first plurality of weights depends at least in part on the content type and the target bit rate. The controller further provides the first QP values to the encoding hardware for rate control of the frame encoding.

Abstract: Systems and methods herein are for a video encoder to be associated with a rate optimization distortion (RDO) module and a calibration module, where the RDO module may be to perform RDO for received frames of a media stream and may be to generate at least an RDO output that is based in part on quality measures between the received frames and decoded frames, and where the calibration module may be to provide an evaluation metric that is to scale or transform at least a range of the quality measures, with the scaling or transforming to potentially reduce an effect on a compression performed in the video encoder.

Abstract: A system includes a processing device to receive a video content, a quality metric, and a target bit rate for encoding the video content. The system includes encoding hardware to perform frame encoding on the video content and a controller coupled between the processing device and the encoding hardware. The controller is programmed with machine instructions to generate first QP values on a per-frame basis using a frame machine learning model with a first plurality of weights. The first plurality of weights depends at least in part on the quality metric and the target bit rate. The controller is further programmed to provide the first QP values to the encoding hardware for rate control of the frame encoding.

Abstract: Systems and methods herein are for a video encoder to be associated with an interface that is to receive, from an application, at least one metric that is associated with a quality preference for video compression to be performed by the video encoder and that is to provide a weight map to enable the video encoder to perform rate-distortion optimization (RDO) for received frames from the application using the weight map to weigh one or more first blocks associated with an individual one of the frames more than one or more second blocks associated with the individual one of the frames.

Abstract: A system includes a processing device to receive video content and output encoded video of the video content for a client video device. The system includes a controller coupled to the processing device, the controller programmed with machine instructions to receive, from a video encoder while encoding the video content, frame statistics based on one or more encoded frames of the video content corresponding to a current frame. The machine instructions further generate a first quantization parameter (QP) value for the current frame using a frame machine learning model, wherein the frame machine learning model includes states that depend on the frame statistics. The machine instructions further provide the first QP value to the video encoder for rate control of the frame encoding of the current frame.

Abstract: A system includes a processing device to receive video content, metadata related to the video content, and a target bit rate for encoding the video content. The processing device further detects a content type of the video content based on the metadata and encodes hardware to perform frame encoding on the video content. The system further includes a controller coupled between the processing device and the encoding hardware. The controller is programmed with machine instructions to generate first QP values on a per-frame basis using a frame machine learning model with a first plurality of weights. The first plurality of weights depends at least in part on the content type and the target bit rate. The controller further provides the first QP values to the encoding hardware for rate control of the frame encoding.

Abstract: A system includes a processing device to receive a video content, a quality metric, and a target bit rate for encoding the video content. The system includes encoding hardware to perform frame encoding on the video content and a controller coupled between the processing device and the encoding hardware. The controller is programmed with machine instructions to generate first QP values on a per-frame basis using a frame machine learning model with a first plurality of weights. The first plurality of weights depends at least in part on the quality metric and the target bit rate. The controller is further programmed to provide the first QP values to the encoding hardware for rate control of the frame encoding.