Abstract: The present disclosure explains a system and method for determining a spatial orientation of a weapon in an augmented reality training environment. Fiducial markers are mounted on the weapon and two cameras are mounted on a user's head to capture spatial coordinates of the plurality of fiducial markers. The spatial coordinates of fiducial markers are processed to determine the spatial orientation of the weapon and a simulated discharging of the weapon. Augmented reality imagery is generated based on the spatial orientation and the detected movement. The augmented reality imagery corresponding to the spatial orientation and discharging of the weapon is rendered for the user with the head-mounted display.

Abstract: A simulation system is disclosed in which images are captured of one or more subjects within a simulation volume, the images are analyzed to determine 2D keypoints for each of subject (where the 2D keypoints for a subject collectively represent the body position and/or posture of the respective subject), and corresponding 3D keypoints (and thus a complete 3D skeletal pose of each subject) of each subject are created from the 2D keypoints. This can be done for multiple frames, and tracking can be performed by matching 3D keypoints of a subject from a first frame with corresponding 3D keypoints from a successive frame.