Abstract: A method, system and program application is provided for automatically testing the operation of a media player with media files (e.g., video files) that are embodied in various formats. In one illustrative example visually encoded metrics are embedded in a media file that is to serve as a test file. These metrics can be detected and decoded when the test file is rendered by the media player. A testing program automatically executes various playback commands to simulate the way a user would operate the media player when viewing a media file. The testing program captures the media player's display buffer after the execution of each command. The display buffer includes the frame or frames that are played by the media player as a result of the commands. The metric or metrics embedded in the captured frames are detected and decoded and compared to a database that includes the metric or metrics that would be expected if the media player is correctly playing the test file.