Abstract: A visual tracking and mapping system builds panoramic images in a handheld device equipped with optical sensor, orientation sensors, and visual display. The system includes an image acquirer for obtaining image data from the optical sensor of the device, an orientation detector for interpreting the data captured by the orientation sensors of the device, an orientation tracker for tracking the orientation of the device, and a display arranged to display image data generated by said tracker to a user.

Abstract: A computerized system receives a source video stream presenting a panning view of a panoramic environment at a plurality of different viewing angles. The system stores angular metadata and sequential timestamp of each frame of the video stream, and extracts overlapping frames from the video stream which may be optically aligned using keypoint tracking to compile a static panoramic background image. During playback, a display device can align and present each frame of the video stream at a respective viewing angle relative to the panoramic environment, in sequence according to the sequential timestamp of each frame.