Abstract: Methods, systems, and apparatus, including computer-readable storage media for hardware-accelerated fine-grained sparse computation. The accelerator provides for improved performance for structured fine-grained sparse AI workloads, for example by accelerating sparse matrix multiplication required to execute or train AI models. Sparse data is compressed to remove zero-valued elements before being streamed into a matrix multiplication unit (MXU) of the accelerator. The accelerator stores a gains matrix, which can be the matrix for multiplying with the received input matrix. The accelerator uses an index array mapping locations of elements in the compressed matrix with locations of elements in the matrix's pre-compressed form, to generate a multiplier matrix from the gains matrix. Aspects of the disclosure also provide for accelerated gains matrix loading in a hardware accelerator or other type of processor.

Abstract: A processor remaps stored data and the corresponding memory addresses of the data for different processing units of a heterogeneous processor. The processor includes a data remap engine that changes the format of the data (that is, how the data is physically arranged in segments of memory) in response to a transfer of the data from system memory to a local memory hierarchy of an accelerated processing module (APM) of the processor. The APM's local memory hierarchy includes an address remap engine that remaps the memory addresses of the data at the local memory hierarchy so that the data can be accessed by routines at the APM that are unaware of the data remapping. By remapping the data, and the corresponding memory addresses, the APM can perform operations on the data more efficiently.