Abstract: This application describes a hardware accelerator and a device for accelerating neural network computations. An example accelerator may include multiple cores and a central processing unit (CPU) respectively associated with DDRs, a data exchange interface connecting a host device to the accelerator, and a three-layer NoC architecture. The three-layer NoC architecture includes an outer-layer NoC configured to transfer data between the host device and the DDRs, a middle-layer NoC configured to transfer data among the plurality of cores; and an inner-layer NoC within each core and including a cross-bar network for broadcasting weights and activations of neural networks from a global buffer of the core to a plurality of processing entity (PE) clusters within the core.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for balanced-weight sparse convolution processing. An exemplary method comprises: obtaining an input tensor and a plurality of filters at a layer within a neural network; segmenting the input tensor into a plurality of sub-tensors; dividing a channel dimension of each of the plurality of filters into a plurality of channel groups; pruning each of the plurality of filters so that each of the plurality of channel groups of each filter comprises a same number of non-zero weights; segmenting each of the plurality of filters into a plurality of the sub-filters according to the plurality of channel groups; and assigning the plurality of sub-tensors and the plurality of sub-filters to a plurality of processors for parallel convolution processing.

Abstract: This application describes hybrid hardware accelerators, systems, and apparatus for performing various computations in neural network applications using the same set of hardware resources. An example accelerator may include weight selectors, activation input interfaces, and a plurality of Multiplier-Accumulation (MAC) circuits organized as a plurality of MAC lanes Each of the plurality of MAC lanes may be configured to: receive a control signal indicating whether to perform convolution or vector operations; receive one or more weights according to the control signal; receive one or more activations according to the control signal; and generate output data based on the one or more weights and the one or more input activations according to the control signal and feed the output data into an output buffer. Each of the plurality of MAC lanes includes a plurality of multiplier circuits and a plurality of adder-subtractor circuits.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for few-shot learning-based generator training are disclosed. An exemplary method may start with obtaining a teacher model and a plurality of training samples, as well as a generator for generating more training samples. After generating a plurality of additional training samples using the method may continue with feeding the plurality of generated additional training samples into the teacher model to obtain a plurality of first statistics; and feeding the plurality of training samples into the teacher model to obtain a plurality second statistics. Then the method further includes training the generator to minimize a distance between the plurality of first statistics and the plurality of second statistics.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for few-shot learning-based generator training based on raw data collected from a specific domain or class. In cases where the raw data is collected from multiple domains but is not easily divisible into classes, the invention describes training multiple generators based on a pivot-sample-based training process. Pivot samples are randomly selected from the raw data for clustering, and each cluster of raw data may be used to train a generator using the few-shot learning-based training process.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for optimizing neural network training, are described. The method may include: during a forward propagation at a current layer of a neural network, generating, based on a sparse input tensor and a sparse weight tensor of the current layer, a dense output tensor, and sparsifying the dense output tensor to obtain a sparse output tensor; during a backward propagation at the current layer of the neural network: determining a first sparse derivative tensor based on the sparse output tensor, obtaining a dense derivative tensor based on the first sparse derivative tensor and the sparse weight tensor of the current layer, and sparsifying the dense derivative tensor to obtain a second sparse derivative tensor; and training weight tensors of the neural network based on the first sparse derivative tensor and the second sparse derivative tensor.