Abstract: The layers of a neural network model are traversed in sequence one or more times while generating a plurality of relevance scores each time based on neuron weights and neuron biases of the neuron network model. Each relevance score of the plurality of relevance scores quantifies a relevance of a neuron in a lower layer of the sequence of layers to a higher layer of the sequence of layers. One or more relevance vectors can be populated from the plurality of relevance scores generated at the one or more times. Each of the relevance scores in each relevance vector quantifies a relevance of one of the input features to a task for which the neural network model is trained to perform. An explanation of a behavior of the neural network as a whole is generated based on the one or more relevance vectors.

Abstract: A network output is generated by feeding an input vector to an input layer of a neural network model having a plurality of neurons arranged in a sequence of layers, a plurality of neuron weights, and a plurality of neuron biases. The network output is used to determine an output relevance score. Relevance scores at a last layer of the sequence of layers are generated. Relevance scores are obtained at a first layer of the sequence of layers by reverse propagating the relevance scores generated at the last layer through the sequence of layers other than the last layer using the neuron weights and neuron biases. A feature relevance vector is formed based on the input vector and the relevance scores obtained at the first layer and included in a local explainability dataset, which is then used to generate a local explanation of a prediction of the neural network model.