Abstract: A method to interpret a deep neural network that includes receiving a set of images, analyzing the set of images via a deep neural network, selecting an internal layer of the deep neural network, extracting neuron activations at the internal layer, factorizing the neuron activations via a matrix factorization algorithm to select prototypes and generate weights for each of the selected prototypes, replacing the neuron activations of the internal layer with selected prototypes and weights for each of the selected prototypes, receiving a second set of images, and classifying the second set of images via the deep neural network using the weighted prototypes without the internal layer.