Abstract: A head-mounted display is presented. The head-mounted display includes an inertial measurement unit (IMU), one or more displays and a controller. The controller is configured to establish a reprojection plane for displaying imagery on the one or more displays. Based on output from the IMU, the position and orientation of the HMD relative to the reprojection plane is determined. Image data is then reprojected to the displays. A depth treatment is applied to each of a plurality of locations on the displays based on the determined position and orientation of the HMD relative to the reprojection plane.

Abstract: Enhanced passthrough images are generated and displayed. A current visibility condition of an environment is determined. Based on the current visibility condition, a first camera or a second camera, which detect light spanning different ranges of illuminance, is selected to generate a passthrough image of the environment. The selected camera is then caused to generate the passthrough image. Additionally, a third camera, which is structured to detect long wave infrared radiation, is caused to generate a thermal image of the environment. Parallax correction is performed by aligning coordinates of the thermal image with corresponding coordinates identified within the passthrough image. Subsequently, the parallax-corrected thermal image is overlaid onto the passthrough image to generate a composite passthrough image, which is then displayed.

Abstract: Improved techniques for generating a depth map are disclosed herein. Initially, a stereo pair of images comprising a first and second image are obtained. Both an overlap region and a non-overlap region are identified as between these two images. A depth map is generated based on the stereo pair of images. Generating this depth map is performed by determining, for the overlap region, depths for a portion of an environment represented by the overlap region via stereo matching. The generation process is also performed by determining, for the non-overlap region, depths for a portion of the environment represented by the non-overlap region by acquiring depth information from a source different from the stereo pair of images.

Abstract: Systems and methods are provided for controlling and/or modifying operation of a red, green, blue (RGB) laser assembly for creating images in mixed-reality environments. Initially, the lasers in the RGB laser assembly operate in a low power or non-emitting state. Then, the lasers emit laser light to illuminate a pixel or a group of pixels. This illumination occurs for a period of time spanning less than 15 nanoseconds. By causing the lasers to emit laser light only during this short period of time, the resulting laser light is structured with a reduced spatial coherence level. Once the time period elapses, then the lasers again return to the non-emitting state. This process repeats for each pixel such that the lasers generate pulsed emissions. By operating the lasers with a reduced spatial coherence, undesired visual artifacts can be reduced or eliminated within the target display area.

Abstract: Systems are configured for performing GPS-based and sensor-based relocalization. During the relocalization, the systems are configured to obtain radio-based positioning data indicating an estimated position of the system within a mapped environment. The systems are also configured to identify, based on the estimated position, a subset of keyframes of a map of the mapped environment, wherein the map of the mapped environment includes a plurality of keyframes captured from a plurality of locations within the mapped environment, and the plurality of keyframes are associated with anchor points identified within the mapped environment. The systems are further configured to perform relocalization within the mapped environment based on the subset of keyframes.

Abstract: Systems are provided for estimating 6DOF positioning of a computing device while in a pedestrian dead reckoning mode. The systems obtain a set of inertial tracking data from the set of one or more inertial tracking components while the system is in a pedestrian dead reckoning mode. Then, the systems obtain an estimated 3DOF velocity of the system based inertial tracking data, using a predictive model trained on a set of observed exteroceptive sensor data and observed inertial tracking data. The systems also obtain estimated 6DOF positioning of the systems based on the estimated 3DOF velocity.

Abstract: A computing system is provided. The computing system includes a visible light camera, a thermal camera, and a processor with associated storage. The processor is configured to execute instructions stored in the storage to receive, from the visible light camera, a visible light image for a frame of a scene and receive, from the thermal camera, a thermal image for the frame of the scene. The processor is configured to detect image discrepancies between the visible light image and the thermal image and, based on the detected image discrepancies, determine a presence of a transparent object in the scene. The processor is configured to, based on the detected image discrepancies, output an identification of at least one location in the scene that is associated with the transparent object.

Abstract: Techniques are provided to dynamically generate and render an object bounding fence in a mixed-reality scene. Initially, a sparse spatial mapping is accessed. The sparse spatial mapping beneficially includes perimeter edge data describing an object's edge perimeters. A gravity vector is also generated. Based on the perimeter edge data and the gravity vector, two-dimensional (2D) boundaries of the object are determined and a bounding fence mesh of the environment is generated. A virtual object is then rendered, where the virtual object is representative of at least a portion of the bounding fence mesh and visually illustrates a bounding fence around the object.

Abstract: Optimizations are provided for a high dynamic range (HDR) sensor. This sensor is a spatially multiplexed image sensor that includes at least two sets of red, green, and blue (RGB) pixels. Each red pixel in the second set of RGB pixels is positioned proximately and sometimes, adjacently, to at least one red pixel in the first set of RGB pixels. Each green pixel in the second set of RGB pixels is positioned proximately to at least one green pixel in the first set of RGB pixels. Each blue pixel in the second set of RGB pixels is positioned proximately to at least one blue pixel in the first set of RGB pixel. This spatially multiplexed image sensor is able to generate a digital image with reduced motion blurring artifacts.

Abstract: A multi-section laser device is configured with a gain section and an integrated photodetector section. The photodetector section, rather than being a separate component, is integrated directly into the body of the laser. The integrated photodetector section absorbs photons generated by the gain section and creates a photocurrent that is proportional to the output of the multi-section laser device. The measured photocurrent is usable to calculate power output of the multi-section laser device and to identify any adjustments that may be needed to be made to the laser in order to achieve desired laser light output.

Abstract: A head-mounted display is presented. The head-mounted display includes an inertial measurement unit (IMU), one or more displays and a controller. The controller is configured to establish a reprojection plane for displaying imagery on the one or more displays. Based on output from the IMU, the position and orientation of the HMD relative to the reprojection plane is determined. Image data is then reprojected to the displays. A depth treatment is applied to each of a plurality of locations on the displays based on the determined position and orientation of the HMD relative to the reprojection plane.

Abstract: Techniques for aligning images generated by an integrated camera physically mounted to an HMD with images generated by a detached camera physically unmounted from the HMD are disclosed. A 3D feature map is generated and shared with the detached camera. Both the integrated camera and the detached camera use the 3D feature map to relocalize themselves and to determine their respective 6 DOF poses. The HMD receives the detached camera's image of the environment and the 6 DOF pose of the detached camera. A depth map of the environment is accessed. An overlaid image is generated by reprojecting a perspective of the detached camera's image to align with a perspective of the integrated camera and by overlaying the reprojected detached camera's image onto the integrated camera's image.

Abstract: Improved techniques for providing passthrough images in the form of a stylized image embodying a novel perspective. A raw texture image of an environment is generated. A depth map is acquired for the environment. A stylized image is generated by applying a stylization filter to the raw texture image. Subsequent to acquiring the depth map and subsequent to generating the stylized image, a stylized parallax-corrected image is generated by reprojecting the stylized image to a new perspective using depth data.

Abstract: Techniques for de-aliasing depth ambiguities included within infrared phase depth images are described herein. An illuminator emits reference light towards a target object. Some of this light is reflected back and detected. A phase image is generated based on phase differences between the reference light and the reflected light. The phase differences represent changes in depth within overlapping sinusoidal periods of the reference and reflected light. The phase image also includes ambiguities because multiple different depths within the phase image share the same phase difference value, even though these depths actually correspond to different real-world depths. The phase image is fed as input to a machine learning (“ML”) component, which is configured to de-alias the ambiguities by determining, for each pixel in the phase image, a corresponding de-aliasing interval. A depth map is generated based on the phase image and any de-aliasing intervals generated by the ML component.

Abstract: Systems and methods for providing a mixed-reality pass-through experience include implement acts of obtaining a texture map of a real-world environment, obtaining a depth map of the real-world environment, obtaining an updated texture map of the real-world environment subsequent to the obtaining of the depth map and the texture map, and rendering a virtual representation of the real-world environment utilizing both the depth map and the updated texture map that was obtained subsequent to the depth map. The texture map and the depth map may be based on a same image pair obtained from a pair of stereo cameras, the depth map being obtained by performing stereo matching on the same image pair. Additionally, the acts may further include detecting a predicted pose of a user and reprojecting a portion of the depth map to conform to a user perspective associated with the predicted pose.

Abstract: Improved techniques for generating depth maps are disclosed. A stereo pair of images of an environment is accessed. This stereo pair of images includes first and second texture images. A signal to noise ratio (SNR) is identified within one or both of those images. Based on the SNR, which may be based on the texture image quality or the quality of the stereo match, there is a process of selectively computing and imposing a smoothness penalty against a smoothness term of a cost function used by a stereo depth matching algorithm. A depth map is generated by using the stereo depth matching algorithm to perform stereo depth matching on the stereo pair of images. The stereo depth matching algorithm performs the stereo depth matching using the smoothness penalty.

Abstract: Modifications are performed to cause a style of an image to match a different style. A first image is accessed, where the first image has the first style. A second image is also accessed, where the second image has a second style. Subsequent to a deep neural network (DNN) learning these styles, a copy of the first image is fed as input to the DNN. The DNN modifies the first image copy by transitioning the first image copy from being of the first style to subsequently being of the second style. As a consequence, a modified style of the transitioned first image copy bilaterally matches the second style.

Abstract: Enhanced passthrough images are generated and displayed. A current visibility condition of an environment is determined. Based on the current visibility condition, a first camera or a second camera, which detect light spanning different ranges of illuminance, is selected to generate a passthrough image of the environment. The selected camera is then caused to generate the passthrough image. Additionally, a third camera, which is structured to detect long wave infrared radiation, is caused to generate a thermal image of the environment. Parallax correction is performed by aligning coordinates of the thermal image with corresponding coordinates identified within the passthrough image. Subsequently, the parallax-corrected thermal image is overlaid onto the passthrough image to generate a composite passthrough image, which is then displayed.

Abstract: Enhanced passthrough images are generated and displayed. A current visibility condition of an environment is determined. Based on the current visibility condition, a first camera or a second camera, which detect light spanning different ranges of illuminance, is selected to generate a passthrough image of the environment. The selected camera is then caused to generate the passthrough image. Additionally, a third camera, which is structured to detect long wave infrared radiation, is caused to generate a thermal image of the environment. Parallax correction is performed by aligning coordinates of the thermal image with corresponding coordinates identified within the passthrough image. Subsequently, the parallax-corrected thermal image is overlaid onto the passthrough image to generate a composite passthrough image, which is then displayed.

Abstract: Mapping common features between images that commonly represent an environment using different light spectrum data is performed. A first image having first light spectrum data is accessed, and a second image having second light spectrum data is accessed. These images are fed as input to a DNN, which then identifies feature points that are common between the two images. A generated mapping lists the feature points and lists coordinates of the feature points from both of the images. Differences between the coordinates of the feature points in the two images are determined. Based on these differences, the second image is warped to cause the coordinates of the feature points in the second image to correspond to the coordinates of the feature points in the first image.