Abstract: A class-specific neural network for video compressed sensing and methods for training and testing the class-specific neural network are provided. The class-specific neural network includes a Gaussian-mixture model (GMM) and a plurality of encoders, where the GMM classifies video frame blocks with a plurality of clusters and assigns the video frame blocks to the plurality of clusters. Further, the plurality of encoders receive the video frame blocks and generate a plurality of compressed-sensed frame block vectors, where the plurality of encoders correspond to the plurality of clusters.

Abstract: A neural network system, a method and an apparatus for image compression are provided. The neural network may include a generator including an encoder, an entropy estimator, and a decoder, where the encoder receives an input image and generates an encoder output, a plurality of quantized feature entries are obtained based on the encoder output outputted at a last encoder block, the entropy estimator receives the plurality of quantized feature entries and calculates an entropy loss based on the plurality of quantized feature entries, and the decoder receives the plurality of quantized feature entries and generates a reconstructed image. Furthermore, the neural network may include a discriminator that determines whether the reconstructed image different from the input image based on a discriminator loss. Moreover, the generator may determine whether content of the reconstructed image matches content of the input image based on a generator loss including the entropy loss.

Abstract: A method for processing data in a multi-mode single-engine system, an apparatus, and a non-transitory computer-readable storage medium are provided. In the method, a graphic processing engine receives a first input query. Further, the graphic processing engine obtains a first set of model parameters by switching between multiple sets of model parameters based on the first input query. Moreover, the graphic processing engine infers a first output for the first input query based on the first set of parameters.

Abstract: Methods and apparatuses are provided for serving multiple models with a single engine. The method includes: providing a single engine for a plurality of models on a server, where the server includes at least one graphics processing unit (GPU) and memory coupled with the at least one GPU; loading the plurality of models onto the memory of the server at once; and serving, by the single engine, the plurality of models, where the single engine accommodates all structures and weights of the plurality of models with a shared input and output of the memory.

Abstract: A method and an apparatus for training a transferable vision transformer (TVT) for unsupervised domain adaption (UDA) in heterogeneous devices are provided. The method includes that a heterogeneous device including one or more graphic processing units (GPUs) loads multiple patches into the TVT which includes a transferability adaption module (TAM). Furthermore, a patch-level domain discriminator in the TAM assigns weights to the multiple patches and determines one or more transferable patches based on the weights. Moreover, the heterogeneous device generates a transferable attention output for an attention module in the TAM based on the one or more transferable patches.

Abstract: A method and an apparatus for video processing are provided. The method includes that a decoding terminal receives a plurality of coded video frames coded using one or more generative adversarial networks (GANs), receives network parameters related to the one or more GANs, and decodes the plurality of coded video frames using GANs based on the network parameters. Further, the one or more GANs respectively implement one or more video coding functions including reference-frame coding, motion-compensated frame prediction, and residue-frame coding.

Abstract: A neural network system, a method and an apparatus for image compression are provided. The neural network may include a generator including an encoder, an entropy estimator, and a decoder, where the encoder receives an input image and generates an encoder output, a plurality of quantized feature entries are obtained based on the encoder output outputted at a last encoder block, the entropy estimator receives the plurality of quantized feature entries and calculates an entropy loss based on the plurality of quantized feature entries, and the decoder receives the plurality of quantized feature entries and generates a reconstructed image. Furthermore, the neural network may include a discriminator that determines whether the reconstructed image different from the input image based on a discriminator loss. Moreover, the generator may determine whether content of the reconstructed image matches content of the input image based on a generator loss including the entropy loss.

Abstract: A method, an apparatus, and a non-transitory computer-readable storage medium for video compression using a generative adversarial network (GAN) are provided. The method includes obtaining, by a generator of the GAN, a reconstructed target frame based on a reference frame and a raw target frame to be reconstructed; concatenating, by a transformer-based discriminator of the GAN, the reference frame, the raw target frame and the reconstructed target frame to obtain a paired data; determining, by the transformer-based discriminator of the GAN, whether the paired data is real or fake to guide reconstruction of the raw target frame; and determining a generator loss and a transformer-based discriminator loss, and performing gradient back propagation and updating network parameters of the GAN based on the generator loss and the transformer-based discriminator loss.

Abstract: A method, apparatus, and a non-transitory computer-readable storage medium for image denoising. The method may include obtaining a raw image captured by a camera. The method may also include obtaining a color modeled image based on the raw image. The method may further include obtaining a subsampled raw image based on the raw image. The method may also include obtaining a denoised image based on a neural network processing the color modeled image and the subsampled raw image.

Abstract: A method, apparatus, and a non-transitory computer-readable storage medium for generating heterogenous platform code. The method may obtain a neural network model. The neural network model may be programed to run on at least one platform. The method may also obtain an initial intermediate representation (IR) code by encoding the neural network model, and obtain a target IR code by adding decorations to the initial IR code based on a target platform. The method may also output an executable code optimized to run on the target platform by decoding the target IR code.

Abstract: Systems and methods are provided for automatic speech recognition. In the method, the system obtains a padded sequence by processing a plurality of acoustic signals. The system compresses the padded sequence by reducing the size of the padded sequence to obtain a compressed sequence. The system inputs the compressed sequence into a pre-trained encoder neural network to obtain an encoded sequence and then decompresses the encoded sequence by recovering the encoded sequence to an original sequential ordering. The system inputs the encoded sequence to a decoding module to obtain recognition texts.

Abstract: An automatic speech recognition system and a method thereof are provided. The system includes an encoder and a decoder. The encoder comprises a plurality of encoder layers. At least one encoder layer includes a plurality of encoder sublayers fused into one or more encoder kernels. The system further comprises a first pair of ping-pong buffers communicating with the one or more encoder kernels. The decoder comprises a plurality of decoder layers. At least one decoder layer includes a plurality of decoder sublayers fused into one or more decoder kernels. The decoder receives a decoder output related to the encoder output and generates a decoder output. The encoder sends the decoder output to a beam search kernel.

Abstract: Methods and apparatuses are provided for movie and television series video data analysis. The method includes: gathering and reading, by a processor, a plurality of input movies; removing a video border of each input movie; splitting the input movie into short clips, based on accuracy and efficiency requirements of different analyzing models; assessing attributes of each input movie by analyzing, with the different analyzing models, the input movie, the short clips cut from the input movie, and the frame images extracted from the input movie; and summarizing the plurality of input movies based on matching and integrating the attributes assessed for each input movie.

Abstract: A method for training a neural network system for generating paired low resolution (LR) and high resolution (HR) images, the neural network system, an apparatus, and a non-transitory computer-readable storage medium thereof are provided. The method includes that a first generator in the neural network system generates a LR image based on a random vector; a second generator in the neural network system generates a HR image based on the random vector, where the HR image is paired with the LR image; obtaining a plurality of losses based on the LR image and the HR image; and updating the first generator based on the plurality of losses.

Abstract: Methods and apparatuses are provided for movie and television series video data analysis. The method includes: gathering and reading, by a processor, a plurality of input movies; removing a video border of each input movie; splitting the input movie into short clips, based on accuracy and efficiency requirements of different analyzing models; assessing attributes of each input movie by analyzing, with the different analyzing models, the input movie, the short clips cut from the input movie, and the frame images extracted from the input movie; and summarizing the plurality of input movies based on matching and integrating the attributes assessed for each input movie.

Abstract: A neural network system for restoring images, a method and a non-transitory computer-readable storage medium thereof are provided. The neural network system includes an encoder and a generative adversarial network (GAN) prior network. The encoder includes a plurality of encoder blocks, where each encoder block includes at least one transformer block and one convolution layer, where the encoder receives an input image and generates a plurality of encoder features and a plurality of latent vectors. Additionally, the GAN prior network includes a plurality of pre-trained generative prior layers, where the GAN prior network receives the plurality of encoder features and the plurality of latent vectors from the encoder and generates an output image with super-resolution.

Abstract: A method and an apparatus for length-aware local tiling in a sparse attention module in a transformer in heterogeneous devices are provided. The method includes that a heterogeneous device including one or more GPUs: divides a transformed sparsity mask into a plurality of first tiles and obtaining one or more effective first tiles from the plurality of first tiles, where each effective first tile includes at least one non-zero element; loads the one or more effective first tiles into a shared memory in the one or more GPUs and loads a plurality of elements in a first matrix corresponding to the one or more effective first tiles into the shared memory; and performs multiplication by a first sampled dense-dense matrix multiplication (SDDMM) kernel in the sparse attention module in the transformer by fetching the one or more effective first tiles and the plurality of elements from the shared memory.

Abstract: A method, apparatus, and a non-transitory computer-readable storage medium for generating heterogenous platform code. The method may obtain a neural network model. The neural network model may be programed to run on at least one platform. The method may also obtain an initial intermediate representation (IR) code by encoding the neural network model, and obtain a target IR code by adding decorations to the initial IR code based on a target platform. The method may also output an executable code optimized to run on the target platform by decoding the target IR code.

Abstract: A method and an apparatus for accelerating a transformer with a sparse attention pattern are provided. The method includes that a heterogeneous device including one or more GPUs loads a first matrix, a second matrix, and a transformed sparsity mask into a first sampled dense-dense matrix multiplication (SDDMM) kernel in a sparse attention module in the transformer and generates a first output based on the first matrix, the second matrix, and the transformed sparsity mask by the first SDDMM kernel, generates a second output by a softmax kernel in the sparse attention module based on the first output, loads the second output, a third matrix, and the transformed sparsity mask into a matrix multiplication kernel in the sparse attention module, and generates an output of the sparse attention module.

Abstract: A method, apparatus, and a non-transitory computer-readable storage medium for image denoising. The method may include obtaining a raw image captured by a camera. The method may also include obtaining a color modeled image based on the raw image. The method may further include obtaining a subsampled raw image based on the raw image. The method may also include obtaining a denoised image based on a neural network processing the color modeled image and the subsampled raw image.