Abstract: Embodiments of the disclosure provide systems and methods for processing video content. The method can include: receiving raw video data of a video; determining a texture complexity for the video based on the raw video data; determining an encoding mode for the raw video data based on the texture complexity; and encoding the raw video data using the determined encoding mode.

Abstract: The present disclosure relates to a method for compensating an image. The method comprises estimating transform coefficients of a frequency component for a first image based on the first image, performing a dot multiplication operation between the estimated transform coefficients and a basis function associated with the frequency component to generate a compensation image, and combining the first image and the compensation image to generate a combined image.

Abstract: Systems and methods are disclosed for storing neural networks and weights for neural networks. In some implementations, a method is provided. The method includes storing a plurality of weights of a neural network comprising a plurality of nodes and a plurality of connections between the plurality of nodes. Each weight of the plurality of weights is associated with a connection of the plurality of connections. The neural network comprises a binarized neural network. The method also includes receiving input data to be processed by the neural network. The method further includes determining whether a set of weights of the plurality of weights comprises one or more errors. The method further includes refraining from using the set of weights to process the input data using the neural network in response to determining that the set of weights comprises the one or more errors.

Abstract: The system described herein can include neural networks with noise-injection layers. The noise-injection layers can enable the neural networks to be trained such that the neural networks are able to maintain their classification and prediction performance in the presence of noisy data signals. Once trained, the parameters from the neural networks with noise-injection layers can be used in the neural networks of systems that include resistive random-access memory (ReRAM), memristors, or phase change memory (PCM), which use analog signals that can introduce noise into the system. The use of ReRAM, memristors, or PCM can enable large-scale parallelism that improves the speed and computational efficiency of neural network training and classification. Using the parameters from the neural networks trained with noise-injection layers, enables the neural networks to make robust predictions and calculations in the presence of noisy data.

Abstract: A system for processing encoded image components for artificial intelligence tasks. The system can include one or more compute units, one or more controllers and memory. The one or more controllers can include one or more micro-op schedulers and one or more channel switches. The one or more compute units can be configured to process components of the transformed domain image data according to one or more micro-operations for an artificial intelligence task. The one or more channel switches can be configured to selectively control the transfer of the components of transformed domain image data to the one or more compute units based on one or more gating flags. The one or more channel switches can also be configured to selectively control generation of the one or more micro-operations by the one or more micro-op schedulers based on the one or more gating flags.

Abstract: Methods and systems are provided for implementing static channel filtering operations upon image datasets transformed to frequency domain representations, including decoding images of an image dataset to generate a frequency domain representation of the image dataset; discarding coefficient values of one or more particular frequency channels of each image of the image dataset in a frequency domain representation; and transporting the image dataset in a frequency domain representation to one or more special-purpose processor(s). Methods and systems of the present disclosure may enable a filtered image dataset to be input to a second layer of a learning model, bypassing a first layer, or may enable a learning model to be designed with a reduced-size first layer. This may achieve benefits such as reducing computational overhead and time of machine learning training and inference computations, reducing volume of image data input into the learning model, and reducing convergence time.

Abstract: A method for of encoding an application screen comprises partitioning graphic data into a plurality of graphic layers and classifying each of the plurality of graphic layers as either a screen content (SC) or a non-screen content (non-SC) layer. The method further comprises classifying each of the plurality of graphic layers as either a screen content (SC) or a non-screen content (non-SC) layer. Further, the method comprises rendering and encoding the one or more SC layers using a first codec and the one or more non-SC layers using a second codec.

Abstract: A video processing unit can include a non-object-based region-of-interest detection neural network, a threshold selection module and a region-of-interest map generator. The non-object-based region-of-interest detection neural network can be configured to receive a video frame and generate a plurality of candidate non-object-based region-of-interest blocks. The threshold selection module can be configured to receive the plurality of candidate non-object-based region-of-interest blocks and identify a plurality of selected region-of-interest blocks based on a predetermined threshold. The region-of-interest map generator can be configured to receive the selected non-object-based region-of-interest blocks and generate a region-of-interest map.

Abstract: This application describes methods, systems, and apparatus, including computer programs encoded on computer storage media, for microarchitecture-aware program sampling. An exemplary method includes receiving one or more traces collected from one or more microarchitectures executing a computer program for evaluating hardware configurations; training a machine learning (ML) model with multi-task learning based on the one or more traces as one or more training tasks; generating a plurality of embedded vectors representing the computer program; and updating, based on the trained ML model, the plurality of embedded vectors.

Abstract: The present disclosure relates to a method for compensating an image. The method comprises estimating transform coefficients of a frequency component for a first image based on the first image, performing a dot multiplication operation between the estimated transform coefficients and a basis function associated with the frequency component to generate a compensation image, and combining the first image and the compensation image to generate a combined image.

Abstract: Image data is accessed. The image data includes frequency domain components. A subset of the frequency domain components is selected based on the relative importance of the frequency domain components. Only the subset of the frequency domain components is provided to an accelerator that executes a neural network to perform an artificial intelligence task using the subset of frequency domain components.

Abstract: Exemplary methods and apparatus are disclosed that implement super-sparse image/video compression by storing image dictionary elements within a cross-bar resistive random access memory (ReRAM) array (or other suitable cross-bar NVM array). In illustrative examples, each column of the cross-bar ReRAM array stores the values for one dictionary element (such as one 4×4 dictionary element). Methods and apparatus are described for training (configuring) the cross-bar ReRAM array to generate and store the dictionary elements by sequentially applying patches from training images to the array using an unstructured Hebbian training procedure. Additionally, methods and apparatus are described for compressing an input image by applying patches from the input image to the ReRAM array to read out cross-bar column indices identifying the columns storing the various dictionary elements that best fit the image. This may be done in parallel using a set of ReRAM arrays.

Abstract: A method and apparatus for characteristic-based video processing include: in response to receiving a region of a picture of a video sequence, determining a characteristic in the region, the region being independent of other regions of the picture for video coding; determining a class associated with the region based on the characteristic, the class being selected from a plurality of classes; and encoding the region using a parameter set associated with the class, the parameter set being selected from a plurality of parameter sets for video coding at different quality levels.

Abstract: Use of a NAND array architecture to realize a binary neural network (BNN) allows for matrix multiplication and accumulation to be performed within the memory array. A unit synapse for storing a weight of a BNN is stored in a pair of series connected memory cells. A binary input is applied as a pattern of voltage values on a pair of word lines connected to the unit synapse to perform the multiplication of the input with the weight by determining whether or not the unit synapse conducts. The results of such multiplications are determined by a sense amplifier, with the results accumulated by a counter.

Abstract: The present disclosure relates to a method for compensating an image. The method comprises estimating transform coefficients of a frequency component for a first image based on the first image, performing a dot multiplication operation between the estimated transform coefficients and a basis function associated with the frequency component to generate a compensation image, and combining the first image and the compensation image to generate a combined image.

Abstract: The present disclosure relates to a method for scheduling computation resources for generating feature maps for video. The method comprises determining runtime for generating feature maps of a reference picture and a predicted picture, determining available computation resources for generating the feature maps, and allocating, based on the runtime, one or more computation resources among the available computation resources for generating the feature maps such that the feature maps are generated at regular time intervals.

Abstract: The systems and methods are configured to efficiently and effectively determine or find an estimated optimal encoding parameter set. In one embodiment, a video encoding parameter set estimation method comprises: performing an offline encoding parameter set characteristic prediction process that determines an estimate of a candidate encoding parameter set characteristic; and performing an encoding parameter set search process that identifies a predicted or estimated optimal video encoding parameter set. The encoding parameter set search process can include applying a constraint to the candidate encoding parameter set characteristic; and determining if candidate encoding parameter set meets an objective, wherein the determining is performed if the constraint is satisfied. The candidate encoding parameter set characteristic can be an estimated encoding time of the candidate encoding parameter set. The objective can be the best video quality out of a plurality of candidate encoding parameter sets.

Abstract: Exemplary methods and apparatus are disclosed that implement super-sparse image/video compression by storing image dictionary elements within a cross-bar resistive random access memory (ReRAM) array (or other suitable cross-bar NVM array). In illustrative examples, each column of the cross-bar ReRAM array stores the values for one dictionary element (such as one 4×4 dictionary element). Methods and apparatus are described for training (configuring) the cross-bar ReRAM array to generate and store the dictionary elements by sequentially applying patches from training images to the array using an unstructured Hebbian training procedure. Additionally, methods and apparatus are described for compressing an input image by applying patches from the input image to the ReRAM array to read out cross-bar column indices identifying the columns storing the various dictionary elements that best fit the image. This may be done in parallel using a set of ReRAM arrays.

Abstract: Video coding techniques including differential bit rate or quality coding of one or more regions of interest and one or more non-regions of interest based on information including one or more of coordinates of the one or more regions of interest, a target complexity, residual encoder bit data, a requested quality, a difference between the current video data frame and a reconstructed video data frame, a target quality, a requested bit rate, frame target bit allocation and an as encoded bit rate.

Abstract: Systems and methods for providing a neural network with multiple sparsity levels include sparsifying a matrix associated with the neural network to form a first sparse matrix; training the neural network using the first sparse matrix to form a second sparse matrix by fixing values and locations of non-zero elements of the first sparse matrix and updating a zero-value element of the first sparse matrix to be a non-zero value, wherein non-zero elements of the second sparse matrix includes the non-zero elements of the first sparse matrix; and outputting the second sparse matrix for executing the neural network.