Abstract: The video summarization system and method supports a moving sensor or multiple sensors by mapping imagery back to a common ortho-rectified geometry. The video summarization system includes at least one video sensor to acquire video data, of at least one area of interest (AOI), including video frames having a plurality of different perspectives. The video sensor may be a moving sensor or a plurality of sensors to acquire video data, of the at least one AOI, from respective different perspectives. A memory stores the video data, and a processor is configured to cooperate with the memory to register video frames from the AOI, ortho-rectify registered video frames based upon a common geometry, identify events within the ortho-rectified registered video frames, and generate a video summary of selected events shifted in time within a selected AOI based upon identified events within the ortho-rectified registered video frames.