Abstract: A method and system for detecting and classifying defects in a paved surface is disclosed. A sequence of images of the paved surface is obtained from at least one imaging device that can be mounted on a vehicle. The images are used to form a three-dimensional reconstruction. A machine learning process is used to train the system to recognize different kinds of defects and defect-free surfaces. Performing a pixel-by-pixel comparison of the images obtained for a particular paved surface with a database of images of surfaces with known defects provides a determination of the locations of defects in that paved surface. The system and method disclosed herein do not require the use of artificial lighting and are unaffected by transient changes in ambient light.