Abstract: A model visualizer visualizes a neural network model at a neuron level. The model visualizer receives a plurality of instances comprising a plurality of features, and receives a neural network model comprising a plurality of layers, each layer comprising a plurality of neurons. For each neuron of each layer, the model visualizer selects a group of instances that highly activate that neuron to generate a summary feature vector, calculates an average feature value of the summary feature vector, compares each feature value of the summary feature vector with the average feature value, and assigns an attribute to each element of the summary feature vector based on the comparison. The model visualizer visualizes the received neural network model based on the assignment to each neuron.