Abstract: A system and method for determining the efficient number of reference frames which are to be evaluated for an image using frame level adaptation techniques. The system and method determines and updates the number of reference frames which are to be searched based on the reference frame selection statistics of the previously coded frame. Preferably, the invention increases the number of reference frames to be referred when a first desired ratio of blocks refer to the furthest reference frame. The invention decreases the number of reference frames if a second desired ratio of blocks refer to the furthest reference frame. The motion value is considered in increasing the number of reference frames. The invention evaluates whether portions of the image frames are static, thereby terminating motion vector searches on that portion.