Abstract: A rule-based method which detects and tracks the moving objects in video frames in an automated way is invented. The method can be used in object based image sequence compression applications. Motion of pixels between two consecutive frames are estimated using a block based motion estimation method, and the resultant dense motion vector field is segmented. Using the same method, a second segmentation is achieved based on partitioning of the image according to color values of the pixels. The image, which is formed by the translation of the final segmentation results of the previous frame using the estimated motion vectors, is used as the third segmentation. By the help of a sequence of rules, which uses the three different segmentation masks as its inputs, detection and tracking of moving objects in the scene is achieved within accurate boundaries; the problems caused by occlusion or uncovered background are insignificant.