Abstract: A computer-implemented method, comprising accessing an image comprising a handheld device, wherein the image is captured by one or more cameras associated with the computing device, generating a cropped image that comprises a hand of a user or the handheld device from the image by processing the image, generating a vision-based six degrees of freedom (6DoF) pose estimation for the handheld device by processing the cropped image, metadata associated with the image, and first sensor data from one or more sensors associated with the handheld device, generating a map-based 6DoF pose estimation using the handheld device, and generating a final 6DoF pose estimation for the handheld device based on the vision-based 6DoF pose estimation and the map-based 6DoF pose estimation generated using the handheld device.

Abstract: In one embodiment, a method includes capturing images of a first user wearing a VR display device in a real-world environment. The method includes receiving a VR rendering of a VR environment. The VR rendering is from the perspective of the mobile computing device with respect to the VR display device. The method includes generating a first MR rendering of the first user in the VR environment. The first MR rendering of the first user is based on a compositing of the images of the first user and the VR rendering. The method includes receiving an indication of a user interaction with one or more elements of the VR environment in the first MR rendering. The method includes generating, in real-time responsive to the indication of the user interaction with the one or more elements, a second MR rendering of the first user in the VR environment. The one or more elements are modified according to the interaction.

Abstract: A method includes generating a local map of a real environment, the local map being defined by first spatial relationships between first feature descriptors corresponding to a visible feature in the real environment captured by a device. The device receives a downloaded map defined by second spatial relationships between an anchor point and second feature descriptors corresponding to visible features captured by another device, wherein the anchor point corresponds to a location of a virtual object. The local map is updated by merging the downloaded map with the local map based on a comparison between the first feature descriptors and the second feature descriptors and a pose of the device is determined relative to a particular feature descriptor in the updated local map. Virtual content is rendered based on the pose and one or more spatial relationships linking the particular feature descriptor and the anchor in the updated local map.

Abstract: In one embodiment, a method includes using one or more cameras of a mobile computing device to capture one or more images of a first user wearing a VR display device in a real-world environment. The mobile computing device transmits a pose of the mobile computing device with respect to the VR display device to a VR system. The mobile computing device receives from the VR system a VR rendering of a VR environment. The VR rendering is from the perspective of the mobile computing device with respect to the VR display device. The method includes segmenting the first user from the one or more images and generating, in real-time responsive to capturing the one or more images, a MR rendering of the first user in the VR environment. The MR rendering of the first user is based on a compositing of the segmented one or more images of the first user and the VR rendering.

Abstract: In one embodiment, a method includes using one or more cameras of a mobile computing device to capture one or more images of a first user wearing a VR display device in a real-world environment. The mobile computing device transmits a pose of the mobile computing device with respect to the VR display device to a VR system. The mobile computing device receives from the VR system a VR rendering of a VR environment. The VR rendering is from the perspective of the mobile computing device with respect to the VR display device. The method includes segmenting the first user from the one or more images and generating, in real-time responsive to capturing the one or more images, a MR rendering of the first user in the VR environment. The MR rendering of the first user is based on a compositing of the segmented one or more images of the first user and the VR rendering.