Abstract: The present invention is directed to a system and method for image to world registration for medical reality applications, using a world spatial map. This invention is a system and method to link any point in a fluoroscopic image to its corresponding position in the visual world using spatial mapping with a head mounted display (HMD) (world tracking). On a projectional fluoroscopic 2D image, any point on the image can be thought of as representing a line that is perpendicular to the plane of the image that intersects that point. The point itself could lie at any position in space along this line, located between the X-Ray source and the detector. With the aid of the HMD, a virtual line is displayed in the visual field of the user.