Abstract: A system for generating an augmented reality image comprises a video camera for obtaining video data and a sensor for obtaining sensed data. An augmented reality processor is coupled to the video camera and to the sensor. The augmented reality processor is configured to receive the video data from the video camera and to receive the sensed data from the sensor and to receive other types of computed data, such as any imaging modality. A display device is coupled to the augmented reality processor. The augmented reality processor is further configured to generate for display on the display device a video image from the video data received from the video camera and to generate a corresponding data image from the sensed data received from the sensor or other computed data. The augmented reality processor is further configured to merge the video image and the corresponding data image generated from computed data or sensed data so as to generate the augmented reality image.