Abstract: A system and method is disclosed for representing each frame in a sequence of video frames in terms of a correlation between successive video frames. The system and method of the present invention segment the current video frame data and a previous frame into blocks for which motion vectors and/or affine mapping coefficients may be determined. The motion vectors may then be used to represent the current frame data without requiring periodic transmission of a base, self-referential frames of video data. Preferably, the compressor which generates the codewords to represent the current frame data subdivides the blocks of the current frame and previous frame buffers to further evaluate motion factors which may be used to represent current frame data. This evaluation is performed in a recursive manner and preferably includes evaluation of a bit rate/distortion factor to optimally select codewords for representing the current frame data.