Abstract: Techniques for improving the visualization of content with a mixed reality (MR) device are described. One techniques includes monitoring a gaze of a user of a MR device through one or more displays of the MR device. A convergence distance of the gaze of the user is determined based on the monitoring. A selection between a first projection mode and a second projection to display content on the one or more displays is made based on whether the convergence distance exceeds a threshold distance from the user. Content is displayed on the one or more displays, based on the selection.

Abstract: An environment map, such as a cube map, can be dynamically generated for a scene using image data captured by a device executing an augmented reality (AR) application. The current orientation of the device, along with field of view information for the camera, can be used to determine a portion of the environment map that corresponds to the captured image data. The image data can be used to fill in that portion, and subsequent image data captured for other orientations used to fill the remaining portions. The generated environment map can then be used to render AR content to be displayed with a live view of the scene. This can include determining the illumination and/or reflections effects for the AR content. The rendered AR content can be displayed as an overlay with respect to a live view of the scene.

Abstract: An environment map, such as a cube map, can be obtained for a scene that is appropriate for the current lighting state. A grayscale image representation is generated that represents physical objects visible in the scene. The grayscale representation is provided to a device for rendering AR content. A color lookup table (LUT) is generated for coloring the grayscale image representation. The color LUT can be appropriate for the current lighting conditions of the scene. As the lighting state changes, such as over the course of a day, different color LUTs can be sent to the device for purposes of updating the environment map. The grayscale image representation, once colored, can serve as an environment map for purposes of creating reflection effects on AR content to be rendered with respect to a live view of the scene.

Abstract: Realistic reflections can be generated for augmented reality content displayed with respect to a view of a physical scene. A device executing an AR application obtains a three dimensional representation of the scene. The representation is used to determine surface orientation information for various physical objects in the physical scene, indicating how incident light would be reflected from those surfaces. The device can also use an infrared detector or transceiver to obtain an infrared view of the scene. The infrared data can be analyzed to determine the reflectivity of the various surfaces in the scene. Utilizing the surface orientation and reflectivity data, reflections from various AR content rendered for the scene can be determined. The corresponding pixels of the live view of the scene can then be color adjusted to account for the reflection, or an overlay can be generated that simulates the color adjustment for the reflection.