Abstract: Embodiments of the present disclosure disclose a method and a system for video anomaly detection. The system is configured to collect a sequence of input video frames of an input video of a scene. In addition, the system is configured to partition each input video frame of the sequence of input video frames into a plurality of input video patches. Further, the system is configured to process each of the plurality of input video patches with one or more classifiers. Each of the one or more classifiers corresponds to a deep neural network trained to estimate one or more attributes of the plurality of input video patches from an output of a penultimate layer of the deep neural network. Furthermore, the system is configured to compare the output of the penultimate layer. The system is further configured to detect an anomaly based on the output of the penultimate layer.