Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for real-time background replacement. Visual characteristics of a visual background, comprised of an image or series of images that make up a motion video, are identified. Frames of a real-time video are captured with a video camera. Foreground areas are distinguished from background areas in the frames of the real-time video by using the visual background. The visual background displays an area that overlaps with an area in the one or more captured frames. The modifying occurs in real-time with capturing the frames by the video camera. Video data is provided for display that superimposes images from the foreground areas over the visual background without superimposing the background areas. The identified visual characteristics of the visual background are used to modify visual characteristics of particular pixels in the obtained one or more frames of the real-time video.