Abstract: An apparatus and method for local quantization for convolutional neural networks. For example, one embodiment of an apparatus comprises: a convolutional neural network module comprising a neuron network structure to perform pattern recognition within an input image using a set of input image values; and a quantization module to quantize input image values to reduce processing requirements within one or more stages of the neuron network structure; the quantization module to perform quantization of each of a plurality of patches of the input image using a first quantization policy to generate a first matrix of quantized input data and to perform quantization of each of a plurality of kernel data using a second quantization policy to generate a second matrix of quantized kernel data.