Abstract: Methods and systems for providing a mixed reality (MR) interaction are provided. In one example, the method comprises: capturing a first image of a user's appendage and a physical peripheral input device, wherein the user's appendage covers at least a part of the physical peripheral input device; performing a segmentation of the first image into a first region of pixels corresponding to the user's appendage, wherein a second region of pixels corresponds to the physical peripheral input device; generating a composite image from the first image based on rendering, in place of the second region of pixels, pixels representing a virtual peripheral input device, such that the virtual peripheral input device appears in the composite image as at a same spatial location as the physical peripheral input device and covered at least partially by the user's appendage; and displaying the composite image to the user.

Abstract: Methods and systems for providing a mixed reality (MR) interaction are provided. In one example, a method comprises: capturing, at a first time and using a camera of a head-mounted display (HMD) of a user, a first image of a physical interaction of the user with a physical object; measuring a movement of the HMD with respect to the physical object between the first time and a second time; processing the first image based on the measurement of the movement of the HMD to generate a second image; generating, based on the second image, a composite image of a virtual interaction involving the user; and displaying, via the HMD and based on the composite image, the virtual interaction in place of the physical interaction to the user at the second time.

Abstract: Methods and systems for providing a mixed reality (MR) interaction are provided. In one example, a method comprises: capturing, at a first time and using a camera of a head-mounted display (HMD) of a user, a first image of a physical interaction of the user with a physical object; measuring a movement of the HMD with respect to the physical object between the first time and a second time; processing the first image based on the measurement of the movement of the HMD to generate a second image; generating, based on the second image, a composite image of a virtual interaction involving the user; and displaying, via the HMD and based on the composite image, the virtual interaction in place of the physical interaction to the user at the second time.

Abstract: Methods and systems for providing a mixed reality (MR) interaction are provided. In one example, the method comprises: capturing a first image of a user's appendage and a physical peripheral input device, wherein the user's appendage covers at least a part of the physical peripheral input device; performing a segmentation of the first image into a first region of pixels corresponding to the user's appendage, wherein a second region of pixels corresponds to the physical peripheral input device; generating a composite image from the first image based on rendering, in place of the second region of pixels, pixels representing a virtual peripheral input device, such that the virtual peripheral input device appears in the composite image as at a same spatial location as the physical peripheral input device and covered at least partially by the user's appendage; and displaying the composite image to the user.