Abstract: Vector-scaled hierarchical codebook quantization reduces precision (bitwidth) vectors of parameters and may enable energy-efficient acceleration of deep neural networks. A vector (block array) comprises one or more parameters within a single dimension of a multi-dimensional tensor (or kernel). For example, block array comprises 4 sub-vectors (blocks) and each sub-vector comprises 8 parameters. The parameters may be represented in integer, floating-point, or any other suitable format. A vector cluster quantization technique is used to quantize blocks of parameters in real-time. Hardware circuitry within a datapath identifies an optimal codebook of a plurality of codebooks for quantizing each block of parameters and the block is encoded using the identified codebook. During processing, the identified codebook is used to obtain the quantized parameter and perform computations at the reduced precision.

Abstract: Vector clustered quantization reduces precision (bitwidth) vectors of parameters and may enable energy-efficient acceleration of deep neural networks. A vector comprises one or more parameters within a single dimension of a multi-dimensional tensor (matrix or kernel). A set of quantizers is initialized for a first step (vector-clustering). After initialization, vectors are mapped into clusters based on quantization errors, where each one of the clusters is associated with a different one of the quantizers. During the second step (per-cluster quantization) each quantizer is optimized to quantize the vectors in the cluster that is associated with the quantizer. In an embodiment, the quantizers are optimized using the Lloyd-Max algorithm, which effectively minimizes the per-cluster quantization noise. The first and second steps may be repeated before the vectors are quantized for processing by a neural network model.

Abstract: Mechanisms to exploit the inherent resiliency of deep learning inference workloads to improve the energy efficiency of computer processors such as graphics processing units with these workloads. The mechanisms provide energy-accuracy tradeoffs in the computation of deep learning inference calculations via energy-efficient floating point data path micro-architectures with integer accumulation, and enhanced mechanisms for per-vector scaled quantization (VS-Quant) of floating-point arguments.