Abstract: A system (and a method) are disclosed for intelligently fetch one or multiple reference blocks from memory for each block to be motion compensated or motion estimated within a video processing system. The system includes a reference block configuration evaluation unit and a motion compensation memory fetching unit. The reference block configuration evaluation unit analyzes the reference block configuration of the block being motion compensated with a plurality of reference block configurations of its neighboring blocks. In response to a reference block configuration evaluation result, the reference block configuration evaluation unit decides the configuration of reference blocks to be fetched from a memory. The motion vector memory fetching unit fetches the number of reference blocks for motion compensation accordingly.