Abstract: The present disclosure provides a system and method for detecting motion vectors in an image frame using a recursive hierarchical process with a non-rasterized vector-scanning motion to reduce erroneous motion vectors in an image frame of a digital video sequence. In general, a resolution hierarchy is generated for an image frame, wherein the resolution hierarchy comprises the original image frame and one or more copy image frames each having a different, lower resolution than the original image frame. Each image frame in the hierarchy is partitioned into image patches disposed in columns and rows, and the image patches are scanned in a non-rasterized motion to detect motion vectors in each image patch. The disclosed system and method provides faster convergence and improved accuracy by converging motion vectors in multiple directions and minimizing erroneous motion vectors in the image sequence.