Abstract: Systems having rolling shutter sensors with a plurality of sensor rows are configured for compensating for rolling shutter artifacts that result from different sensor rows in the plurality of sensor rows outputting sensor data at different times. The systems compensate for the rolling shutter artifacts by identifying readout timepoints for the plurality of sensor rows of the rolling shutter sensor while the rolling shutter sensor captures an image of an environment and identifying readout poses each readout timepoint, as well as obtaining a depth map based on the image. The depth map includes a plurality of different rows of depth data that correspond to the different sensor rows. The system further compensates for the rolling shutter artifacts by generating a 3D representation of the environment while unprojecting the rows of depth data into 3D space using the readout poses.

Abstract: A head-mounted device (HMD) is configured to perform depth detection with a stereo camera pair comprising a first camera and a second camera, both of which are configured to detect/capture visible light and IR light. The fields of view for both of the cameras overlap to form an overlapping field of view. The HMD also includes an IR dot-pattern illuminator that is mounted on the HMD with the cameras and that is configured to emit an IR dot-pattern illumination. The IR dot-pattern illuminator emits a dot-pattern illumination that spans at least a part of the overlapping field of view. The IR dot-pattern illumination adds texture to objects in the environment and enables the HMD to determine depth for those objects, even if they have textureless/smooth surfaces.

Abstract: Techniques are provided to dynamically generate and render an object bounding fence in a mixed-reality scene. Initially, a sparse spatial mapping is accessed. The sparse spatial mapping beneficially includes perimeter edge data describing an object's edge perimeters. A gravity vector is also generated. Based on the perimeter edge data and the gravity vector, two-dimensional (2D) boundaries of the object are determined and a bounding fence mesh of the environment is generated. A virtual object is then rendered, where the virtual object is representative of at least a portion of the bounding fence mesh and visually illustrates a bounding fence around the object.

Abstract: In some instances, spatial mapping data from one spatial mapping is used to augment the spatial mapping data in another spatial mapping. First and second spatial mapping data is accessed, where both the first and second spatial mapping data overlap in that they both, at least partially, describe the same portion of an environment three-dimensionally. A determination is made as to whether the second spatial mapping data is to augment the first spatial mapping data. If so, then the second spatial mapping data is used to augment the first spatial mapping data. Otherwise, the second spatial mapping data is not used to augment the first spatial mapping data. These determinations may be based, at least in part, on determined quality levels of the first and second spatial mapping data.

Abstract: In some instances, undesired content is selectively omitted from a mixed-reality scene via use of tags. An environment's spatial mapping is initially accessed. Based on an analysis of this spatial mapping, any number of segmented objects are identified from within the spatial mapping. These segmented objects correspond to actual physical objects located within the environment and/or to virtual objects that are selected for potential projection into the mixed-reality scene. For at least some of these segmented objects, a corresponding tag is then accessed. A subset of virtual content is then generated based on certain attributes associated with those tags. The content that is included in the subset is specially chosen for actual projection. Thereafter, the selected content is either projected into the mixed-reality scene or scheduled for projection.

Abstract: A structure for facilitating virtual experiences comprises: a structural support having a first side facing toward an interior of the structure, a second side opposite the first side, and an intra-support hollow disposed between the first and second sides. An IR reflective surface is adjacent to at least a portion of the second side of the structural support. An IR emitter within the hollow between the first and second sides is configured to emit IR light toward the IR reflective surface, such that the IR light is reflected toward the interior of the structure.

Abstract: Techniques are provided for increasing a laser light's spectral linewidth while simultaneously improving how a laser is controlled by causing the laser to operate at higher power levels. An illumination energy value for a pixel and an illumination time period for the pixel are both determined. A number of laser pulses that are to be emitted by the laser assembly to illuminate the pixel during the illumination time period is also determined. This number is based on the illumination energy value for the pixel. Then, within the illumination time period and in accordance with the determined number of laser pulses, the pixel is illuminated by causing the laser assembly to emit one or more laser pulses that cause the pixel to be illuminated at the illumination energy value.

Abstract: Optimizations are provided for generating passthrough visualizations for Head Mounted Displays. The interpupil distance of a user wearing a head-mounted device is determined and a stereo camera pair with a left and right camera is used to capture raw images. The center-line perspectives of the images captured by the left camera have non-parallel alignments with respect to center-line perspectives of any images captured by the right camera. After the raw images are captured, various camera distortion corrections are applied to the images to create corrected images. Epipolar transforms are then applied to the corrected images to create transformed images having parallel center-line perspectives. Thereafter, a depth map is generated of the transformed images. Finally, left and right passthrough visualizations are generated and rendered by reprojecting the transformed left and right images.

Abstract: A head-mounted device (HMD) is configured to perform depth detection with a stereo camera pair comprising a first camera and a second camera, both of which are configured to detect/capture visible light and IR light. The fields of view for both of the cameras overlap to form an overlapping field of view. The HMD also includes an IR dot-pattern illuminator that is mounted on the HMD with the cameras and that is configured to emit an IR dot-pattern illumination. The IR dot-pattern illuminator emits a dot-pattern illumination that spans at least a part of the overlapping field of view. The IR dot-pattern illumination adds texture to objects in the environment and enables the HMD to determine depth for those objects, even if they have textureless/smooth surfaces.

Abstract: Multi-section laser systems are configured with a gain/modulation section and a pre-bias section. Both sections are electrically connected to a diode laser resonator and both sections are independently controllable via laser driver circuitry. The multi-section laser can be used to provide pulsing optimizations that include reducing the turn-on delay of the laser while also ensuring that the resulting laser light's spectral linewidth satisfies a threshold linewidth requirement. During use, a pre-bias current is applied to the pre-bias section. This current causes some photons to be spontaneously emitted. During this time, a gain current is refrained from being applied to the gain section until the resonator is seeded with a spectrum of photons from the pre-bias section. Once the resonator is sufficiently seeded, the gain current is applied to the gain section, thereby producing a seeded pulse of laser light having a desired spectral linewidth.

Abstract: Disclosed herein are optimized techniques for controlling the exposure time or illumination intensity of a depth sensor. Invalid-depth pixels are identified within a first depth map of an environment. For each invalid-depth pixel, a corresponding image pixel is identified in a depth image that was used to generate the first depth map. Multiple brightness intensities are identified from the depth image. Each brightness intensity is categorized as corresponding to either an overexposed or underexposed image pixel. An increased exposure time or illumination intensity or, alternatively, a decreased exposure time or illumination intensity is then used to capture another depth image of the environment. After a second depth map is generated based on the new depth image, portion(s) of the second depth map are selectively merged with the first depth map by replacing the invalid-depth pixels of the first depth map with corresponding valid-depth pixels of the second depth map.

Abstract: Techniques for de-aliasing depth ambiguities included within infrared phase depth images are described herein. An illuminator emits reference light towards a target object. Some of this light is reflected back and detected. A phase image is generated based on phase differences between the reference light and the reflected light. The phase differences represent changes in depth within overlapping sinusoidal periods of the reference and reflected light. The phase image also includes ambiguities because multiple different depths within the phase image share the same phase difference value, even though these depths actually correspond to different real-world depths. The phase image is fed as input to a machine learning (“ML”) component, which is configured to de-alias the ambiguities by determining, for each pixel in the phase image, a corresponding de-aliasing interval. A depth map is generated based on the phase image and any de-aliasing intervals generated by the ML component.

Abstract: Systems are provided for performing surface reconstruction with reduced power consumption. A surface mesh of an environment is generated, where the surface mesh is generated from multiple depth maps that are obtained of the environment. After the surface mesh is generated, a change detection image of that environment is captured while refraining from obtaining a new depth map of the environment. The change detection image is compared to the surface mesh. If a difference between the change detection image and the surface mesh is detected and if that difference satisfies a pre-determined difference threshold, then a new depth map of the environment is obtained. The surface mesh is then updated using the new depth map.

Abstract: An illumination module and a depth camera on a near-eye-display (NED) device used for depth tracking may be subject to strict power consumption budgets. To reduce power consumption of depth tracking, the illumination power of the illumination module is controllably varied. Such variation entails using a previous frame, or previously recorded data, to inform the illumination power used to generate a current frame. Once the NED determines the next minimum illumination power, the illumination module activates at that power level. The illumination module emits electromagnetic (EM) radiation (e.g. IR light), the EM radiation reflects off surfaces in the scene, and the reflected light is captured by the depth camera. The method repeats for subsequent frames, using contextual information from each of the previous frames to dynamically control the illumination power. Thus, the method reduces the overall power consumption of the depth camera assembly of the NED to a minimum level.

Abstract: A method to process a contributing digital image of a subject in an image-processing computer. The contributing digital image is received in a depth-resolving machine configured to furnish a depth image based at least in part on the contributing digital image. The contributing digital image is also received in a classification machine previously trained to classify a pixel of the contributing digital image as liable to corrupt a depth value of a corresponding pixel of the depth image. A repair value is computed for the depth value of the corresponding pixel of the depth image, which is then corrected based on the repair value and returned to the calling process.

Abstract: An optical isolation system is disclosed for use in a display to reduce light that is transmitted from one or more light sources to a camera. The system can include a gasket arranged next to the camera, where the gasket includes an aperture that substantially surrounds a region that is adjacent to a lens of the camera. In some cases, the gasket can reduce optical crosstalk associated with visible light as well as infrared light. The gasket can include a material that is optical opaque to the wavelengths of the light being transmitted. In addition, some layers of the display can include optical disrupting regions formed in a thickness of the layer.

Abstract: A computing system is provided. The computing system includes a visible light camera, a thermal camera, and a processor with associated storage. The processor is configured to execute instructions stored in the storage to receive, from the visible light camera, a visible light image for a frame of a scene and receive, from the thermal camera, a thermal image for the frame of the scene. The processor is configured to detect image discrepancies between the visible light image and the thermal image and, based on the detected image discrepancies, determine a presence of a transparent object in the scene. The processor is configured to, based on the detected image discrepancies, output an identification of at least one location in the scene that is associated with the transparent object.

Abstract: Improved techniques for re-localizing Internet-of-Things (IOT) devices are disclosed herein. Sensor data digitally representing one or more condition(s) monitored by an IOT device is received. In response, a sensor readings map is accessed, where this map is associated with the IOT device. The map also digitally represents the IOT device's environment and includes data representative of a location of the IOT device within the environment. The map also includes data representative of the conditions monitored by the IOT device. Additionally, the map is updated by attaching the sensor data to the map. In some cases, a coverage map can also be computed. Both the sensors readings map and the coverage map can be automatically updated in response to the TOT device being re-localized.

Abstract: Techniques for improving how surface reconstruction data is prepared and passed between multiple devices are disclosed. For example, an environment is scanned to generate 3D scanning data. This 3D scanning data is then transmitted to a central processing service. The 3D scanning data is structured or otherwise configured to enable the central processing service to generate a digital 3D representation of the environment using the 3D scanning data. Reduced resolution representation data is received from the central processing service. This reduced resolution representation data was generated based on 3D scanning data generated by one or more other computer systems that were also scanning the same environment. A first visualization corresponding to the original 3D scanning data is then displayed simultaneously with one or more secondary visualization(s) corresponding to the reduced resolution representation data.

Abstract: Techniques are provided to re-arrange the placement of a photodiode within an illumination system to achieve improved characteristics and reduced form factor. An illumination system includes a laser assembly, a MEMS mirror system, a beam combiner, and a photodiode. The laser assembly includes RGB lasers, and the MEMS mirror system redirects laser light produced by the RGB lasers to illuminate pixels in an image frame. The beam combiner combines the laser light. The photodiode is provided to determine a power output of the laser assembly by receiving and measuring some of the laser light. The photodiode may be beneficially positioned before or after collimating optics and/or the beam combiner.