Abstract: A system for dense depth computation aided by sparse feature matching generates a first image using a first camera, a second image using a second camera, and a third image using a third camera. The system generates a sparse disparity map using the first image and the third image by (1) identifying a set of feature points within the first image and a set of corresponding feature points within the third image, and (2) identifying feature disparity values based on the set of feature points and the set of corresponding feature points. The system also applies the first image, the second image, and the sparse disparity map as inputs for generating a dense disparity map.

Abstract: Systems and methods are provided performing for low compute depth map generation by implementing acts of obtaining a stereo pair of images of a scene, downsampling the stereo pair of images, generating a depth map by stereo matching the downsampled stereo pair of images, and generating an upsampled depth map based on the depth map using an edge-preserving filter for obtaining at least some data of at least one image of the stereo pair of images.

Abstract: Optimizations are provided for a high dynamic range (HDR) sensor. This sensor is a spatially multiplexed image sensor that includes at least two sets of red, green, and blue (RGB) pixels. Each red pixel in the second set of RGB pixels is positioned proximately and sometimes, adjacently, to at least one red pixel in the first set of RGB pixels. Each green pixel in the second set of RGB pixels is positioned proximately to at least one green pixel in the first set of RGB pixels. Each blue pixel in the second set of RGB pixels is positioned proximately to at least one blue pixel in the first set of RGB pixel. This spatially multiplexed image sensor is able to generate a digital image with reduced motion blurring artifacts.

Abstract: A system for image acquisition with reduced noise using SPADs is configured to perform a plurality of sequential exposure and readout operations. Each exposure and readout operation includes (i) applying a set of shutter operations to configure each SPAD pixel of the SPAD array to enable photon detection, and (ii) for each SPAD pixel of the SPAD array, reading out a number of photons detected during the set of shutter operations. The system is also configured to generate an image based on the number of photons detected for each SPAD pixel during each of the plurality of sequential exposure and readout operations.

Abstract: Techniques for aligning images generated by an integrated camera physically mounted to an HMD with images generated by a detached camera physically unmounted from the HMD are disclosed. A 3D feature map is generated and shared with the detached camera. Both the integrated camera and the detached camera use the 3D feature map to relocalize themselves and to determine their respective 6 DOF poses. The HMD receives the detached camera's image of the environment and the 6 DOF pose of the detached camera. A depth map of the environment is accessed. An overlaid image is generated by reprojecting a perspective of the detached camera's image to align with a perspective of the integrated camera and by overlaying the reprojected detached camera's image onto the integrated camera's image.

Abstract: Techniques for updating a position of overlaid image content using IMU data to reflect subsequent changes in camera positions to minimize latency effects are disclosed. A “system camera” refers to an integrated camera that is a part of an HMD. An “external camera” is a camera that is separated from the HMD. The system camera and the external camera generate images. These images are overlaid on one another and aligned to form an overlaid image. Content from the external camera image is surrounded by a bounding element in the overlaid image. IMU data associated with both the system camera and the external camera is obtained. Based on that IMU data, an amount of movement that the system camera and/or the external camera have moved since the images were originally generated is determined. Based on that movement, the bounding element is shifted to a new position in the overlaid image.

Abstract: A near-eye display device comprises a pupil-expansion optic, a laser, a drive circuit coupled operatively to the first and second lasers, a spatial light modulator (SLM), and a computer. The SLM has a matrix of electronically controllable pixel elements and is configured to receive emission from the laser and to direct the emission in spatially modulated form to the pupil-expansion optic. Coupled operatively to the drive circuit and to the SLM, the computer is configured to parse a digital image, trigger the emission from the laser by causing the drive circuit to drive a periodic current through a gain structure of the laser, and control the matrix of pixel elements such that the spatially modulated form of the emission projects an optical image corresponding to the digital image, wherein the periodic current includes plural cycles of modulation driven through the gain structure while the optical image is projected.

Abstract: A near-eye display device comprises a pupil-expansion optic, first and second lasers, a drive circuit coupled operatively to the first and second lasers, a beam combiner, a spatial light modulator (SLM), and a computer. The first and second lasers are configured to emit in respective first and second wavelength bands. The beam combiner is configured to geometrically combine emission from the first and second lasers into a collimated beam. The SLM is configured to receive the collimated beam and to direct the emission in spatially modulated form to the pupil-expansion optic. The computer is configured to parse a digital image, trigger the emission from the first and second lasers by causing the drive circuit to drive current through the first and second lasers, and control the SLM such that the spatially modulated form of the emission projects an optical image corresponding to the digital image.

Abstract: Improved techniques for generating images are disclosed herein. A first image is generated by an integrated camera. The pose of the computer system is determined based on the image, and a timestamp is determined. A detached camera generates a second image. The second image is aligned with the first image. An overlaid image is generated by overlaying the second image onto the first image based on the alignment. A pose difference is then identified between a current pose of the computer and the initial pose. Consequently, late stage reprojection (LSR) is performed on the overlaid image to account for the pose difference. The LSR-corrected overlaid image is then displayed.

Abstract: A head-mounted device (HMD) is configured to perform depth detection with a stereo camera pair comprising a first camera and a second camera, both of which are configured to detect/capture visible light and IR light. The fields of view for both of the cameras overlap to form an overlapping field of view. The HMD also includes an IR dot-pattern illuminator that is mounted on the HMD with the cameras and that is configured to emit an IR dot-pattern illumination. The IR dot-pattern illuminator emits a dot-pattern illumination that spans at least a part of the overlapping field of view. The IR dot-pattern illumination adds texture to objects in the environment and enables the HMD to determine depth for those objects, even if they have textureless/smooth surfaces.

Abstract: A system for power efficient image acquisition is configurable to capture, using an image sensor, a plurality of partial image frames including at least a first partial image frame and a second partial image frame. The first partial image frame is captured at a first timepoint using a first subset of image sensing pixels of the plurality of image sensing pixels of the image sensor. The second partial image frame is captured at a second timepoint using a second subset of image sensing pixels of the plurality of image sensing pixels of the image sensor. The second subset of image sensing pixels includes different image sensing pixels than the first subset of image sensing pixels, and the second timepoint is temporally subsequent to the first timepoint. The system is configurable to generate a composite image frame based on the plurality of partial image frames.

Abstract: A system for dark current compensation in SPAD imagery is configurable to capture an image frame with the SPAD array and generate a temporally filtered image by performing a temporal filtering operation using the image frame and at least one preceding image frame. The at least one preceding image frame is captured by the SPAD array at a timepoint that temporally precedes a timepoint associated with the image frame. The system is also configurable to obtain a dark current image frame. The dark current image frame includes data indicating one or more SPAD pixels of the plurality of SPAD pixels that detect an avalanche event without detecting a corresponding photon. The system is also configurable to generate a dark current compensated image by performing a subtraction operation on the temporally filtered image or the image frame based on the dark current image frame.

Abstract: A system for obtaining color imagery using SPADs includes a SPAD array that has a plurality of SPAD pixels. Each of the plurality of SPAD pixels includes a respective color filter positioned thereover. The system is configurable to capture an image frame using the SPAD array and generate a filtered image by performing a temporal filtering operation using the image frame and at least one preceding image frame. The at least one preceding image frame is captured by the SPAD array at a timepoint that temporally precedes a timepoint associated with the image frame. The system is also configurable to, after performing the temporal filtering operation, generate a color image by demosaicing the filtered image.

Abstract: Systems are configured for generating temporally corrected pass-through images. In some instances, the systems obtain depth maps of an environment at a first timepoint, generate a 3D representation of the environment by unprojecting the depth information represented in the depth map, and obtain one or more first images of the environment captured at a second timepoint. The systems may also be configured to perform a first intermediate projection to identify first texture information from the one or more first images, identify a display pose associated with the system, generate a display projection of the 3D representation, and creating a composite image based on the display projection and the first texture information.

Abstract: Examples are disclosed that relate to blending different types of images captured by different types of cameras employing different sensing modalities based on a dynamic weighting. The dynamic weighting is calculated based on a dynamic quality proxy that serves as an approximation of image quality that may change from image to image. In one example, a first image of a scene is received from a first camera. A dynamic quality proxy is received. A second image of the scene is received from a second camera with a different sensing modality than the first camera. A composite image blended from the first and second images in proportion to a dynamic weighting that is based on the dynamic quality proxy is output.

Abstract: A head-mounted device (HMD) is configured to perform depth detection with a stereo camera pair comprising a first camera and a second camera, both of which are configured to detect/capture visible light and IR light. The fields of view for both of the cameras overlap to form an overlapping field of view. The HMD also includes an IR dot-pattern illuminator that is mounted on the HMD with the cameras and that is configured to emit an IR dot-pattern illumination. The IR dot-pattern illuminator emits a dot-pattern illumination that spans at least a part of the overlapping field of view. The IR dot-pattern illumination adds texture to objects in the environment and enables the HMD to determine depth for those objects, even if they have textureless/smooth surfaces.

Abstract: A system for structured light depth computation using single photon avalanche diodes (SPADs) is configurable to, over a frame capture time period, selectively activate the illuminator to perform interleaved structured light illumination operations. The interleaved structured light illumination operations comprise alternately emitting at least a first structured light pattern from the illuminator and emitting at least a second structured light pattern from the illuminator. The system is also configurable to, over the frame capture time period, perform a plurality of sequential shutter operations to configure each SPAD pixel of the SPAD array to enable photon detection. The plurality of sequential shutter operations generates, for each SPAD pixel of the SPAD array, a plurality of binary counts indicating whether a photon was detected during each of the plurality of sequential shutter operations.

Abstract: Disclosed herein are techniques for providing an illumination system that emits illumination into an environment while also enabling that system to be undetectable to certain types of external light detection systems. The system includes a single photon avalanche diode (SPAD) low light (LL) detection device and a light emitting device. The light emitting device provides illumination having a wavelength of at least 950 nanometers (nm). An intensity of the illumination is set to a level that causes the illumination to be undetectable from a determined distance away based on the roll off rate of the light. While the light emitting device is providing the illumination, the SPAD LL detection device generates an image of an environment in which the illumination is being provided.

Abstract: A system for single photon avalanche diode image capture is configurable to, over a frame capture time period, selectively activate an illuminator to alternately emit light from the illuminator and refrain from emitting light from the illuminator. The system is configurable to, over the frame capture time period, perform a plurality of sequential shutter operations to configure each SPAD pixel of the SPAD array to enable photon detection. The plurality of sequential shutter operations generates, for each SPAD pixel of the SPAD array, a plurality of binary counts indicating whether a photon was detected during each of the plurality of sequential shutter operations. The system is configurable to, based on a first set of binary counts of the plurality of binary counts, generate an ambient light image. The system is configurable to, based on a second set of binary counts of the plurality of binary counts, generate an illuminated image.

Abstract: A system for efficiently generating SPAD imagery with persistence is configurable to capture an image frame, capture pose data associated with the capturing of the image frame, and access a persistence frame. The persistence frame includes a preceding composite image frame generated based on at least two preceding image frames. The at least two preceding image frames are associated with timepoints that precede a capture timepoint associated with the image frame. The system is configurable to generate a persistence term based on (i) the pose data, (ii) a similarity comparison based on the image frame and the persistence frame, or (iii) a signal strength associated with the image frame. The system is configurable to generate a composite image based on the image frame, the persistence frame, and the persistence term. The persistence term defines a contribution of the image frame and the persistence frame to the composite image.