Abstract: Systems, apparatuses, and methods for implementing spatial block-level pixel activity extraction optimization leveraging motion vectors are disclosed. Control logic coupled to an encoder generates block-level pixel activity metrics for a new frame based on the previously calculated block-level pixel activity data from a reference frame. A cost is calculated for each block of a new frame with respect to a corresponding block of the reference frame. If the cost is less than a first threshold, then the control logic generates an estimate of a pixel activity metric for the block which is equal to a previously calculated pixel activity metric for a corresponding block of the reference frame. If the cost is greater than the first threshold but less than a second threshold, an estimate of the pixel activity metric is generated by extrapolating from the previously calculated pixel activity metric.

Abstract: Systems, apparatuses, and methods for detecting personal-space violations in artificial intelligence (AI) based non-player characters (NPCs) are disclosed. An AI engine creates a NPC that accompanies and/or interacts with a player controlled by a user playing a video game. During gameplay, measures of context-dependent personal space around the player and/or one or more NPCs are generated. A control circuit monitors the movements of the NPC during gameplay and determines whether the NPC is adhering to or violating the measures of context-dependent personal space. The control circuit can monitor the movements of multiple NPCs simultaneously during gameplay, keeping a separate score for each NPC. After some amount of time has elapsed, the scores of the NPCs are recorded, and then the scores are provided to a machine learning engine to retrain the AI engines controlling the NPCs.

Abstract: A processing system estimates motion between blocks of images based on differences in pixel activities between blocks. Blocks having similar pixel activities are more likely to be matches for motion estimation than are blocks having dissimilar pixel activities. Accordingly, the processing system compares pixel activity between current block of a current image and candidate blocks within a search area in the reference image, and estimates motion based on a subset of candidate blocks having a difference in pixel activity from that of the current block within a threshold.

Abstract: A technique for performing video operations is provided. The technique includes decoding underlying content to obtain a decoded block; and applying a shade pattern to the decoded block to obtain a final block.

Abstract: An apparatus and method for efficiently creating less computationally intensive nodes for a neural network. In various implementations, a computing system includes a memory that stores multiple input data values for training a neural network, and a processor. Rather than determine a bit width P of an integer accumulator of a node of the neural network based on bit widths of the input data values and corresponding weight values, the processor selects the bit width P during training. The processor adjusts the magnitudes of the weight values during iterative stages of training the node such that an L1 norm value of the weight values of the node does not exceed a corresponding weight magnitude limit.

Abstract: A multimedia system allocates, during encoding of a multimedia stream, bits to portions of frames based on quality metrics and bit usages for different quantization parameters (QPs). An encoder of the multimedia system encodes a frame in a first pass with a first QP and in a second pass with a second QP. A comparator of the multimedia system measures and compares quality metrics, such as mean squared error, for each portion of the frame encoded with the first QP and the second QP. The comparator compares the difference between the quality metrics for each portion encoded with each QP to a threshold. If the difference in quality metrics for a portion exceeds the threshold, the comparator selects the portion for inclusion in a subset of portions to be encoded with the second QP.

Abstract: Methods and devices are provided for processing data using a neural network. Activations from a previous layer of the neural network are received by a layer of the neural network. Weighted values, to be applied to values of elements of the activations, are determined based on a spatial correlation of the elements and a task error output by the layer. The weighted values are applied to the values of the elements and a combined error is determined based on the task error and the spatial correlation.

Abstract: An encoder calculates a first local encoding parameter for a first block of video content based on one or more local metrics. The encoder modifies the first local encoding parameter based on one or more second local encoding parameters for one or more second blocks of video content that are adjacent to the first block of video content. The encoder then encodes the first block using the modified first local encoding parameter. In some cases, the local encoding parameters are quantization parameters used to quantize values of pixels or compression parameters used to compress values of the pixels. The local metric can include one or more of a target bit rate, a texture complexity, a contrast, an indicator of motion in the first block, and an importance map.

Abstract: Systems, apparatuses, and methods for performing adaptive audio mixing are disclosed. A trained neural network dynamically selects and mixes pre-recorded, human-composed music stems that are composed as mutually compatible sets. Stem and track selection, volume mixing, filtering, dynamic compression, acoustical/reverberant characteristics, segues, tempo, beat-matching and crossfading parameters generated by the neural network are inferred from the game scene characteristics and other dynamically changing factors. The trained neural network selects an artist's pre-recorded stems and mixes the stems in real-time in unique ways to dynamically adjust and modify background music based on factors such as game scenario, the unique storyline of the player, scene elements, the player's profile, interest, and performance, adjustments made to game controls (e.g., music volume), number of viewers, received comments, player's popularity, player's native language, player's presence, and/or other factors.

Abstract: Systems, apparatuses, and methods for bit budgeting in video encode pre-analysis based on context and features are disclosed. A pre-encoder receives a video frame and evaluates each block of the frame for the presence of several contextual indicators. The contextual indicators can include memory colors, text, depth of field, and other specific objects. For each contextual indicator detected, a coefficient is generated and added with other coefficients to generate a final importance value for the block. The coefficients can be adjusted so that only a defined fraction of the picture is deemed important. The final importance value of the block is used to determine the bit budget for the block. The block bit budgets are provided to the encoder and used to influence the quantization parameters used for encoding the blocks.

Abstract: Systems, methods, and techniques utilize reinforcement learning to efficiently schedule a sequence of jobs for execution by one or more processing threads. A first sequence of execution jobs associated with rendering a target frame of a sequence of frames is received. One or more reward metrics related to rendering the target frame are selected. A modified sequence of execution jobs for rendering the target frame is generated, such as by reordering the first sequence of execution jobs. The modified sequence is evaluated with respect to the selected reward metric(s); and rendering the target frame is initiated based at least in part on the evaluating of the modified sequence with respect to the one or more selected reward metric(s).

Abstract: Systems, apparatuses, and methods for detecting personal-space violations in artificial intelligence (AI) based non-player characters (NPCs) are disclosed. An AI engine creates a NPC that accompanies and/or interacts with a player controlled by a user playing a video game. During gameplay, measures of context-dependent personal space around the player and/or one or more NPCs are generated. A control circuit monitors the movements of the NPC during gameplay and determines whether the NPC is adhering to or violating the measures of context-dependent personal space. The control circuit can monitor the movements of multiple NPCs simultaneously during gameplay, keeping a separate score for each NPC. After some amount of time has elapsed, the scores of the NPCs are recorded, and then the scores are provided to a machine learning engine to retrain the AI engines controlling the NPCs.

Abstract: An apparatus includes a processor configured to determine a first distribution associated with an artificial agent based on behavior associated with the artificial agent and a second distribution based on behavior of a user. The processor is further configured to generate a human-likeness similarity measurement by comparing the first distribution to the second distribution and modify the behavior of the artificial agent in response to the similarity measurement failing to satisfy a similarity threshold.

Abstract: A server offloads graphics effects processing to a client device with graphics processing resources by determining a modification to a graphics effects operation, generating a portion of a rendered video stream using the modification to the graphics effects operation, and providing an encoded representation of the portion of the rendered video stream to the client device, along with metadata representing the modification implemented. The client device decodes the encoded representation to recover the portion of the rendered video stream and selectively performs a graphics effects operation on the recovered portion to at least partially revert the resulting graphics effects for the portion to the intended effects without the modification implemented by the server.

Abstract: Systems, apparatuses, and methods for updating and optimizing task scheduling policies are disclosed. A new policy is obtained and updated at runtime by a client based on a server analyzing a wide spectrum of telemetry data on a relatively long time scale. Instead of only looking at the telemetry data from the client's execution of tasks for the previous frame, the server analyzes the execution times of tasks for multiple previous frames so as to determine a more optimal policy for subsequent frames. This mechanism enables making a more informed task scheduling policy decision as well as customizing the policy per application, game, and user without requiring a driver update. Also, this mechanism facilitates improved load balancing across the various processing engines, each of which has their own task queues. The improved load balancing is achieved by analyzing the telemetry data including resource utilization statistics for the different processing engines.

Abstract: Systems, methods, and techniques utilize reinforcement learning to efficiently schedule a sequence of jobs for execution by one or more processing threads. A first sequence of execution jobs associated with rendering a target frame of a sequence of frames is received. One or more reward metrics related to rendering the target frame are selected. A modified sequence of execution jobs for rendering the target frame is generated, such as by reordering the first sequence of execution jobs. The modified sequence is evaluated with respect to the selected reward metric(s); and rendering the target frame is initiated based at least in part on the evaluating of the modified sequence with respect to the one or more selected reward metric(s).

Abstract: A processing device for executing a machine learning neural network operation includes memory and a processor. The processor is configured to receive input data at a layer of the machine learning neural network operation, receive a plurality of sorted filters to be applied to the input data, apply the plurality of sorted filters to the input data to produce a plurality of different feature maps, compress the plurality of different feature maps according to a similarity of the feature maps relative to each other and store the plurality of different feature maps in the memory.

Abstract: A processing device for executing a machine learning neural network operation includes memory and a processor. The processor is configured to receive input data at a layer of the machine learning neural network operation, receive a plurality of sorted filters to be applied to the input data, apply the plurality of sorted filters to the input data to produce a plurality of different feature maps, compress the plurality of different feature maps according to a sparsity of the feature maps and store the plurality of different feature maps in the memory.

Abstract: Methods and devices are provided for encoding video. By using co-sited gradient and variance values to detect text and line in frames of the video. A processor is configured to receive a plurality of frames of video, determine, for a portion of a frame, a variance of the portion of the frame and a gradient of the portion of the frame and encode, using one of a plurality of different encoding qualities, the portion of the frame based on the gradient and the variance of the portion of the frame. Encoding is performed at both the sub-frame level and frame level. The portion of the frame is classified into one of a plurality of categories based on the gradient and variance and encoded based on the category.

Abstract: An apparatus includes a processor and a collision detection unit operatively coupled to the processor. The collision detection unit is configured to process, using a machine learning model, one or more parameters associated with a ray cast in virtual environment comprising an object. The machine learning model is configured to approximate a mesh representing the object. The collision detection unit is further configured to determine if the ray collides with the object based on processing the one or more parameters. In response to determining if the ray collides with the object, the collision detection unit is configured to generate collision data associated with the ray and the object.