Abstract: Systems and methods in accordance with embodiments of the invention enable identifying informative features within input data using a neural network data structure. One embodiment includes a data structure describing a neural network that comprises a plurality of neurons; wherein the processor is configured by the feature application to: determine contributions of individual neurons to activation of a target neuron by comparing activations of a set of neurons to their reference values, where the contributions are computed by dynamically backpropagating an importance signal through the data structure describing the neural network; extracting aggregated features detected by the target neuron by: segmenting the determined contributions; clustering into clusters of similar segments; aggregating data to identify aggregated features of input data that contribute to the activation of the target neuron; and displaying aggregated features of input data to highlight important features.