Abstract: A system and method for tracking movement of objects in a scene from a stream of video frames using first and second correspondence graphs. A first correspondence graph, called an object correspondence graph, is formed comprising a plurality of nodes representing region clusterss in the scene which are hypotheses of objects to be tracked, and a plurality of tracks. Each track comprises an ordered sequence of nodes in consecutive video frames that represents a track segment of an object through the scene. A second correspondence graph, called a track correspondence graph, is created, comprising a plurality of nodes, each node corresponding to at least one track in the first correspondence graph. A track comprising an ordered sequence of nodes in the second correspondence graph represents the path of an object through the scene. Tracking information for objects, such as persons, in the scene, is accumulated based on the first correspondence graph and second correspondence graph.