Abstract: A method for generating a sparsified convolutional neural network (CNN) is provided that includes training the CNN to generate coefficient values of filters of convolution layers, and performing sparsified fine tuning on the convolution layers to generate the sparsified CNN, wherein the sparsified fine tuning causes selected nonzero coefficient values of the filters to be set to zero.

Abstract: Several methods and systems for masking multimedia data are disclosed. In an embodiment, a method for masking includes performing a prediction for at least one multimedia data block based on a prediction mode of a plurality of prediction modes. The at least one multimedia data block is associated with a region of interest (ROI). A residual multimedia data associated with the at least one multimedia data block is generated based on the prediction. A quantization of the residual multimedia data is performed based on a quantization parameter (QP) value. The QP value is variable such that varying the QP value controls a degree of masking of the ROI.

Abstract: Technology is disclosed herein to execute an inference model by a processor which includes a reshape layer. In an implementation, the reshape layer of the inference model receives an output produced by a previous layer of the inference model and inserts padding into the output, then supplies the padded output as an input to a next layer of the inference model. In an implementation, the inference model includes a stitching layer at the beginning of the inference model and an un-stitch layer at the end of the model. The stitching layer of the inference model stitches together multiple input images into an image batch and supplies the image batch as an input to a subsequent layer. The un-stitch layer receives output from a penultimate layer of the inference model and unstitches the output to produce multiple output images corresponding to the multiple input images.

Abstract: In described examples, an integrated circuit includes a memory storing weights and biases, an N-bit fixed point matrix operations accelerator, and a processor. Starting with a first convolution layer, a convolution layer modeled using the processor receives input feature values. A feature scale and weight scale are reduced if an accumulator scale is greater than a maximum bias scale. The input feature values are rescaled using the feature scale, the weights are quantized using the weight scale, and the biases are quantized using the feature scale and weight scale. The rescaled input feature values and quantized weights and biases are convolved using the N-bit fixed point matrix operations accelerator to generate output feature values. The process repeats from the receive action using the output feature values as the input feature values of the next convolution layer. The process then repeats for all layers, feeding back an output feature range.

Abstract: Techniques for executing machine learning (ML) models including receiving an indication to run an ML model on a processing core; receiving a static memory allocation for running the ML model on the processing core; determining that a layer of the ML model uses more memory than the static memory allocated; transmitting, to a shared memory, a memory request for blocks of the shared memory; receiving an allocation of the requested blocks; running the layer of the ML model using the static memory and the range of memory addresses; and outputting results of running the layer of the ML model.

Abstract: Disclosed herein are systems and methods for performing post training quantization. A processor obtains fixed-point output values from a layer of an artificial neural network (ANN) wherein the layer includes fixed-point weights determined based on floating-point weights and a weight scaling factor determined based on an output scaling factor. Next, the processor converts the fixed-point output values to floating-point output values based on the output scaling factor. Then, the processor expands a range of floating-point values. Next, the processor calculates a new output scaling factor based on the expanded range of floating-point output values. Finally, the processor stores the new output scaling factor in an associated memory.

Abstract: In an example, a method includes executing, using one or more processors, a power-of-2 parametric activation (PACT2) function to quantize a set of data. The executing of the PACT2 function includes determining a distribution for the set of data; discarding a portion of the data corresponding to a tail of the distribution to form a remaining set of data; estimating a maximum value of the remaining set of data; determining a new maximum value of the remaining set of data using a moving average and at least one historical value of at least one prior remaining set of data; determining a clipping value by expanding the new maximum value to a nearest power of two value; and quantizing the set of data using the clipping value to form a quantized set of data.

Abstract: Several methods and systems for masking multimedia data are disclosed. In an embodiment, a method for masking includes performing a prediction for at least one multimedia data block based on a prediction mode of a plurality of prediction modes. The at least one multimedia data block is associated with a region of interest (ROI). A residual multimedia data associated with the at least one multimedia data block is generated based on the prediction. A quantization of the residual multimedia data is performed based on a quantization parameter (QP) value. The QP value is variable such that varying the QP value controls a degree of masking of the ROI.

Abstract: Techniques for executing machine learning (ML) models including receiving an indication to run an ML model on a processing core; receiving a static memory allocation for running the ML model on the processing core; determining that a layer of the ML model uses more memory than the static memory allocated; transmitting, to a shared memory, a memory request for blocks of the shared memory; receiving an allocation of the requested blocks; running the layer of the ML model using the static memory and the range of memory addresses; and outputting results of running the layer of the ML model.

Abstract: In described examples of a method for quantizing data for a convolutional neural network (CNN) is provided. A set of data is received and quantized the using a power-of-2 parametric activation (PACT2) function. The PACT2 function arranges the set of data as a histogram and discards a portion of the data corresponding to a tail of the histogram to form a remaining set of data. A clipping value is determined by expanding the remaining set of data to a nearest power of two value. The set of data is then quantized using the clipping value. With PACT2, a model can be quantized either using post training quantization or using quantization aware training. PACT2 helps a quantized model to achieve close accuracy compared to the corresponding floating-point model.

Abstract: Several methods and systems for masking multimedia data are disclosed. In an embodiment, a method for masking includes performing a prediction for at least one multimedia data block based on a prediction mode of a plurality of prediction modes. The at least one multimedia data block is associated with a region of interest (ROI). A residual multimedia data associated with the at least one multimedia data block is generated based on the prediction. A quantization of the residual multimedia data is performed based on a quantization parameter (QP) value. The QP value is variable such that varying the QP value controls a degree of masking of the ROI.

Abstract: Techniques for executing machine learning (ML) models including receiving an indication to run an ML model on a processing core; receiving a static memory allocation for running the ML model on the processing core; determining that a layer of the ML model uses more memory than the static memory allocated; transmitting, to a shared memory, a memory request for blocks of the shared memory; receiving an allocation of the requested blocks; running the layer of the ML model using the static memory and the range of memory addresses; and outputting results of running the layer of the ML model.

Abstract: A method for convolution in a convolutional neural network (CNN) is provided that includes accessing a coefficient value of a filter corresponding to an input feature map of a convolution layer of the CNN, and performing a block multiply accumulation operation on a block of data elements of the input feature map, the block of data elements corresponding to the coefficient value, wherein, for each data element of the block of data elements, a value of the data element is multiplied by the coefficient value and a result of the multiply is added to a corresponding data element in a corresponding output block of data elements comprised in an output feature map.

Abstract: Various examples disclosed herein relate to neural network quantization techniques, and more particularly, to selecting inference precisions for the layers of the neural network. In an example embodiment, a method is provided herein that includes determining an accuracy improvement of a layer of a neural network implemented using a first bit precision relative to using a second bit precision and determining a latency degradation of the layer of the neural network implemented using the first bit precision relative to using the second bit precision. The method further includes selecting, based on the accuracy improvement and the latency degradation, the first bit precision or the second bit precision for use in implementing the layer of the neural network.

Abstract: Technology is disclosed herein to execute an inference model by a processor which includes a reshape layer. In an implementation, the reshape layer of the inference model receives an output produced by a previous layer of the inference model and inserts padding into the output, then supplies the padded output as an input to a next layer of the inference model. In an implementation, the inference model includes a stitching layer at the beginning of the inference model and an un-stitch layer at the end of the model. The stitching layer of the inference model stitches together multiple input images into an image batch and supplies the image batch as an input to a subsequent layer. The un-stitch layer receives output from a penultimate layer of the inference model and unstitches the output to produce multiple output images corresponding to the multiple input images.

Abstract: Disclosed herein are systems and methods for determining the scaling factors for a neural network that satisfy the activation functions employed by the nodes of the network. A processor identifies a saturation point of an activation function. Next, the processor determines a scaling factor for an output feature map based on the saturation point of the activation function. Then, the processor determines a scaling factor for an accumulator based on the scaling for the output feature map and further based on a shift value related to a quantization. Finally, the processor determines a scaling factor for a weight map based on the scaling factor for the accumulator.

Abstract: An example image feature extraction system comprises an encoder neural network having a first set of layers and a decoder neural network having a second set of layers and a third set of layers. The encoder neural network receives an input image, processes the input image through the first set of layers, and computes an encoded feature map based on the input image. The decoder neural network receives the encoded feature map, processes the encoded feature map through the second set of layers to compute a keypoint score map, and processes the encoded feature map through at least a portion of the third set of layers to compute a feature description map.

Abstract: Techniques including receiving a first set of values for processing by a machine learning (ML) network, storing a first portion of the first set of values in an on-chip memory, processing the first portion of the first set of values in a first layer of the ML network to generate a second portion of a second set of values, overwriting the stored first portion with the generated second portion, processing the second portion in a second layer of the ML network to generate a third portion of a third set of values, storing the third portion, repeating the steps of storing the first portion, processing the first portion, overwriting the stored first portion, processing the second portion, and storing the third portion for a fourth portion of the first set of values until all portions of the first set of values are processed to generate the third set of values.

Abstract: Disclosed herein are systems and methods for inference model scheduling of a multi priority inference model system. A processor determines an interrupt flag has been set indicative of a request to interrupt execution of a first inference model in favor of a second inference model. In response to determining that the interrupt flag has been set, the processor determines a state of the execution of the first inference model based on one or more factors. In response to determining the state of the execution is at a preemptable boundary, the processor deactivates the first inference model and activates the second inference model.

Abstract: An example image feature extraction system comprises an encoder neural network having a first set of layers and a decoder neural network having a second set of layers and a third set of layers. The encoder neural network receives an input image, processes the input image through the first set of layers, and computes an encoded feature map based on the input image. The decoder neural network receives the encoded feature map, processes the encoded feature map through the second set of layers to compute a keypoint score map, and processes the encoded feature map through at least a portion of the third set of layers to compute a feature description map.