Abstract: The subject invention provides mixed simulator systems that combine the advantages of both physical objects/simulations and virtual representations. In-context integration of virtual representations with physical simulations or objects can facilitate education and training. Two modes of mixed simulation are provided. In the first mode, a virtual representation is combined with a physical simulation or object by using a tracked display capable of displaying an appropriate dynamic virtual representation as a user moves around the physical simulation or object. In the second mode, a virtual representation is combined with a physical simulation or object by projecting the virtual representation directly onto the physical object or simulation. In further embodiments, user action and interaction can be tracked and incorporated within the mixed simulator system to provide a mixed reality after-action review.

Abstract: An interactive training system includes computer vision provided by at least one video camera for obtaining trainee image data, and pattern recognition and image understanding algorithms to recognize features present in the trainee image data to detect gestures of the trainee. Graphics coupled to a display device is provide for rendering images of at least one virtual individual. The display device is viewable by the trainee. A computer receives the trainee image data or gestures of the trainee, and optionally the voice of the trainee, and implements an interaction algorithm. An output of the interaction algorithm provides data to the graphics and moves the virtual character to provide dynamically alterable images of the virtual character, as well as well as an optional virtual voice. The virtual individual can be a medical patient, where the trainee practices diagnosis on the patient.