Abstract: A processing apparatus and video encoding method are provided which include receiving a portion of a video sequence and determining complexities for blocks of pixels of the portion of the video sequence. Quantization parameter values for corresponding blocks of pixels are selected based on complexities of the corresponding blocks and visually perceived coding artifacts of the corresponding blocks produced by the quantization parameter values. The blocks of pixels are encoded, using the selected quantization parameter values. The blocks of pixels are decoded and the portion of the video sequence is provided for display.

Abstract: Mapping machine learning activation data to a representative value palette, including: selecting, from a plurality of activation values of a model execution, a plurality of representative values; identifying, for each activation value of the plurality of activation values, a representative value of the plurality of representative values; calculating, for each activation value of the plurality of activation values, a corresponding residual value as a difference between an activation value and a corresponding representative value; and storing, for each activation value of the plurality of activation values, the corresponding residual value and an index of the corresponding representative value.

Abstract: Methods and devices are provided for encoding video. By using co-sited gradient and variance values to detect text and line in frames of the video. A processor is configured to receive a plurality of frames of video, determine, for a portion of a frame, a variance of the portion of the frame and a gradient of the portion of the frame and encode, using one of a plurality of different encoding qualities, the portion of the frame based on the gradient and the variance of the portion of the frame. Encoding is performed at both the sub-frame level and frame level. The portion of the frame is classified into one of a plurality of categories based on the gradient and variance and encoded based on the category.

Abstract: Methods and apparatus provide cloud-based video encoding that generates encoded video data by one or more encoders in a cloud platform for a plurality of cloud encoding sessions. The methods and apparatus generate operational improvement tradeoff data in response to operational encoding metrics associated with the one or more encoders and change operational characteristics of the one or more encoders for at least one of the cloud encoding sessions based on the operational improvement tradeoff data.

Abstract: Systems, apparatuses, and methods for performing adaptive audio mixing are disclosed. A trained neural network dynamically selects and mixes pre-recorded, human-composed music stems that are composed as mutually compatible sets. Stem and track selection, volume mixing, filtering, dynamic compression, acoustical/reverberant characteristics, segues, tempo, beat-matching and crossfading parameters generated by the neural network are inferred from the game scene characteristics and other dynamically changing factors. The trained neural network selects an artist's pre-recorded stems and mixes the stems in real-time in unique ways to dynamically adjust and modify background music based on factors such as game scenario, the unique storyline of the player, scene elements, the player's profile, interest, and performance, adjustments made to game controls (e.g., music volume), number of viewers, received comments, player's popularity, player's native language, player's presence, and/or other factors.

Abstract: Systems, apparatuses, and methods for generating a model for determining a quantization strength to use when encoding video frames are disclosed. A pre-encoder performs multiple encoding passes using different quantization strengths on a portion or the entirety of one or more pre-processed video frames. The pre-encoder captures the bit-size of the encoded output for each of the multiple encoding passes. Then, based on the multiple encoding passes, the pre-encoder generates a model for mapping bit-size to quantization strength for encoding video frames or portion(s) thereof. When the encoder begins the final encoding pass for one or more given video frames or any portion(s) thereof, the encoder uses the model to map a preferred bit-size to a given quantization strength. The encoder uses the given quantization strength when encoding the given video frame(s) or frame portion(s) to meet a specified bit-rate for the encoded bitstream.

Abstract: Systems, apparatuses, and methods for reducing latency when consuming an encoded video bitstream in real-time are disclosed. A video encoder encodes a video bitstream and writes chunks of the encoded bitstream to a bitstream buffer. Prior to the encoder completing the encoding of an entire frame, or an entire slice of a frame, a consumer module consumes encoded chunks of the bitstream. In one implementation, to enable pipelining of the consumption with the encoding, the encoder updates a buffer write pointer with an indication of the amount of data that has been written to the bitstream buffer. The consumer module retrieves encoded data from the bitstream buffer up to the location indicated by the buffer write pointer. In this way, the consumer module is able to access and consume encoded video data prior to the encoder finishing encoding an entire frame or an entire slice of the frame.

Abstract: An encoding method is provided which includes receiving a plurality of images, obtaining values of elements in a portion of the images, sorting the elements according to different values of the elements, sorting the elements according to a number of occurrences of the different values and encoding the elements using a subset of the different values having corresponding numbers of occurrences that are higher than corresponding numbers of occurrences of other values. Examples also include a processing device and method for use with palette mode encoding in which the elements are a portion of pixels in images and the values are color values of the portion of pixels in the images.

Abstract: An encoder encodes an image portion by recursively partitioning the portion into a partitioning hierarchy of levels. The top level has a single block representing the entire portion and each lower level has four smaller blocks representing a corresponding larger block at a higher level. A palette table is generated for each bottom-level block based on the pixels of the associated block. For each successively higher level, the encoder generates a palette table for each current-level block by selecting palette colors based on the palette colors from the four palette tables for the associated four blocks at the next-lowest level. A color index map is then generated based on a final palette table selected from the palette tables generated for the partitioning hierarchy. A representation of the portion is then encoded using the final palette table and the color index map to generate a corresponding segment of an encoded bitstream.

Abstract: Calculating, for each frame of a plurality of frames, a corresponding quality value; calculating, for each frame of the plurality of frames, based on one or more visual attributes of a frame, a weight for the corresponding quality value of the frame; calculating an aggregate quality value for the plurality of frames based on the weight and the corresponding quality value for each frame of the plurality of frames; and providing an assessment of the plurality of frames based on the aggregate quality value for the plurality of frames.

Abstract: A system and method for dynamically changing encode quality at a block level based on runtime pre-encoding analysis of content in a video stream. A video encoder continuously analyzes the content during runtime, and collects statistics and/or characteristics of the content before encoding it. This classifies the block among pre-defined categories of content, where every category has its own compression parameters.

Abstract: The present disclosure is directed a system and method for exploiting camera and depth information associated with rendered video frames, such as those rendered by a server operating as part of a cloud gaming service, to more efficiently encode the rendered video frames for transmission over a network. The method and system of the present disclosure can be used in a server operating in a cloud gaming service to improve, for example, the amount of latency, downstream bandwidth, and/or computational processing power associated with playing a video game over its service. The method and system of the present disclosure can be further used in other applications where camera and depth information of a rendered or captured video frame is available.

Abstract: Systems, apparatuses, and methods for reducing latency when consuming an encoded video bitstream in real-time are disclosed. A video encoder encodes a video bitstream and writes chunks of the encoded bitstream to a bitstream buffer. Prior to the encoder completing the encoding of an entire frame, or an entire slice of a frame, a consumer module consumes encoded chunks of the bitstream. In one implementation, to enable pipelining of the consumption with the encoding, the encoder updates a buffer write pointer with an indication of the amount of data that has been written to the bitstream buffer. The consumer module retrieves encoded data from the bitstream buffer up to the location indicated by the buffer write pointer. In this way, the consumer module is able to access and consume encoded video data prior to the encoder finishing encoding an entire frame or an entire slice of the frame.

Abstract: Systems, apparatuses, and methods for performing parallel histogram calculation with application to palette table derivation are disclosed. An encoder calculates a first histogram for a first portion of pixel component value bits of a block of pixels. Then, the encoder selects a first number of the highest pixel count bins from the first histogram. Also, the encoder calculates a second histogram for a second portion of pixel component value bits of the block. The encoder selects a second number of the highest pixel count bins from the second histogram. A third histogram is calculated from the concatenation of bits assigned to the first and second number of bins, and the highest pixel count bins are selected from the third histogram. A palette table is derived based on these highest pixel count bins selected from the third histogram, and the block of pixels is encoded using the palette table.

Abstract: Systems, apparatuses, and methods for calculating multi-pass histograms for palette table derivation are disclosed. An encoder calculates a first histogram for a first portion of most significant bits (MSBs) of pixel component values of a block of an image or video frame. Then, the encoder selects a given number of the highest pixel count bins from the first histogram. The encoder then increases the granularity of these selected highest pixel count bins by evaluating one or more additional bits from the pixel component values. A second histogram is calculated for the concatenation of the original first portion MSBs from the highest pixel count bins and the one or more additional bits, and the highest pixel count bins are selected from the second histogram. A palette table is derived based on these highest pixel count bins selected from the second histogram, and the block is encoded using the palette table.

Abstract: A processing device for executing a machine learning neural network operation includes memory and a processor. The processor is configured to receive input data at a layer of the machine learning neural network operation, receive a plurality of sorted filters to be applied to the input data, apply the plurality of sorted filters to the input data to produce a plurality of different feature maps, compress the plurality of different feature maps according to a sparsity of the feature maps and store the plurality of different feature maps in the memory.

Abstract: A processing device for executing a machine learning neural network operation includes memory and a processor. The processor is configured to receive input data at a layer of the machine learning neural network operation, receive a plurality of sorted filters to be applied to the input data, apply the plurality of sorted filters to the input data to produce a plurality of different feature maps, compress the plurality of different feature maps according to a similarity of the feature maps relative to each other and store the plurality of different feature maps in the memory.

Abstract: Systems, apparatuses, and methods for performing parallel histogram calculation with application to palette table derivation are disclosed. An encoder calculates a first histogram for a first portion of pixel component value bits of a block of pixels. Then, the encoder selects a first number of the highest pixel count bins from the first histogram. Also, the encoder calculates a second histogram for a second portion of pixel component value bits of the block. The encoder selects a second number of the highest pixel count bins from the second histogram. A third histogram is calculated from the concatenation of bits assigned to the first and second number of bins, and the highest pixel count bins are selected from the third histogram. A palette table is derived based on these highest pixel count bins selected from the third histogram, and the block of pixels is encoded using the palette table.

Abstract: An engagement analytics engine of a computing device analyzes one or more of scene representations, player inputs, or player meta information and generates corresponding engagement data indicative of a level of engagement corresponding to the represented scene. The engagement analytics engine generates encoding parameters based on the engagement data to cause scenes or regions within scenes to be encoded with a level of quality based on the indicated level of engagement. In some examples, the engagement analytics engine generates rendering parameters based on the engagement data to cause scenes to be rendered with a frame rate or quality parameters based on the indicated level of engagement. In some examples, the engagement analytics engine causes a load balancer to shift workloads associated with one or more scenes to higher or lower performance servers based on the engagement data.

Abstract: A server offloads graphics effects processing to a client device with graphics processing resources by determining a modification to a graphics effects operation, generating a portion of a rendered video stream using the modification to the graphics effects operation, and providing an encoded representation of the portion of the rendered video stream to the client device, along with metadata representing the modification implemented. The client device decodes the encoded representation to recover the portion of the rendered video stream and selectively performs a graphics effects operation on the recovered portion to at least partially revert the resulting graphics effects for the portion to the intended effects without the modification implemented by the server.