Abstract: A method for enhancing digital imagery. The method comprises receiving a linear, intensity-based image of an environment. A histogram of intensity values is generated for a plurality of pixels within the linear, intensity-based image. Based on the histogram of intensity values, local contrast enhancement is applied to the linear, intensity-based image to generate a contrast enhanced version of the linear, intensity-based image, and artificial colorization is applied to the linear, intensity-based image to generate an artificially colorized version of the linear, intensity-based image. A composite image of the environment is then generated based on at least a portion of the contrast enhanced version of the linear, intensity-based image and at least a portion of the artificially colorized version of the linear, intensity-based image.

Abstract: A system for generating depth information from low-resolution images is configured to access a plurality of image frames capturing an environment, identify a first group of image frames from the plurality of image frames, and generate a first image comprising a first composite image of the environment using the first group of image frames as input. The first composite image has an image resolution that is higher than an image resolution of the image frames of the first group of image frames. The system is also configured to obtain a second image of the environment, where parallax exists between a capture perspective associated with the first image and a capture perspective associated with the second image. The system is also configured to generate depth information for the environment based on the first image and the second image.

Abstract: An image sensor configured to capture imagery with mitigated noise includes a plurality of image sensing pixels arranged to form a sensor array. Each image sensing pixel of the plurality of image sensing pixels comprises an active area configured to receive photons to facilitate image capture. Each active area comprises a length and a width. For at least one image sensing pixel of the plurality of image sensing pixels, the length or the width of the active area is smaller than about 80% of a pixel pitch measurement between the at least one image sensing pixel and an adjacent image sensing pixel. A size of the active area relative to the pixel pitch measurement contributes to mitigating sensor noise for the image sensor.

Abstract: Techniques for updating a position of overlaid image content using IMU data to reflect subsequent changes in camera positions to minimize latency effects are disclosed. A “system camera” refers to an integrated camera that is a part of an HMD. An “external camera” is a camera that is separated from the HMD. The system camera and the external camera generate images. These images are overlaid on one another and aligned to form an overlaid image. Content from the external camera image is surrounded by a bounding element in the overlaid image. IMU data associated with both the system camera and the external camera is obtained. Based on that IMU data, an amount of movement that the system camera and/or the external camera have moved since the images were originally generated is determined. Based on that movement, the bounding element is shifted to a new position in the overlaid image.

Abstract: A system for generating high-resolution video from low-resolution images is configured to access a first video stream and a second video stream capturing an environment. The first video stream is captured by a first video capture device. The second video stream is captured by a second video capture device. Image frames of the first video stream are temporally synchronized with corresponding image frames of the second video stream. The system is also configured to generate a composite video stream with a higher resolution than the first or second video streams. Each composite image frame of the composite video stream is generated using a respective image frame of the first video stream and a temporally synchronized corresponding image frame of the second video stream as input.

Abstract: Techniques for generating a fused enhanced image. A first image is generated using a first camera of a first modality, and a second image is generated using a second camera of a second modality. Pixels that are common between the two images are identified. Textures for the common pixels are determined. A camera characteristic, which is linked to noise, is identified. A scaling factor is applied to the textures in the first image. A first saliency is determined using the scaled textures. A second saliency is determined using the textures from the second image. An alpha map is generated and reflects edge detection weights that have been computed for each one of the common pixels based on the two saliencies. Based on the alpha map, textures are merged from the common pixels to generate the fused enhanced image.

Abstract: A system for facilitating intensity image capture and time of flight capture. The system includes a single photon avalanche diode (SPAD) array comprising a plurality of SPAD pixels, one or more processors, and one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the system to facilitate intensity image capture and time of flight capture by configuring the system to perform interleaved intensity image capture and time of flight capture operations using the SPAD array.

Abstract: Examples are described that relate to correcting line bias in images. One example provides a method comprising receiving, from an imaging device, a plurality of images each comprising a plurality of lines of pixels. The method further comprises, for each image of the plurality of images, for each line of pixels of the plurality of lines of pixels, based at least on one or more pixel values of one or more pixels in the line of pixels, determining a line bias correction for the line, and applying the line bias correction to each pixel in the line, the line bias correction comprising an offset applied to each pixel value in the line of pixels.

Abstract: A system for obtaining dark current images includes one or more processors and one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the system to perform various acts. The acts include obtaining a first image frame, generating a first low-pass filtered image by applying a low-pass filter to the first image frame, and generating a first estimated dark current image by subtracting the first low-pass filtered image from the first image frame.

Abstract: Techniques to temporally filter images via a filtering weight computation are disclosed. A first image having a first timestamp and a second image having a second timestamp are acquired. These images are generated by a camera, and the first timestamp is before the second timestamp. A motion compensation (MC) operation is performed on the first image to produce an MC image. A difference image is generated using the MC image and the second image. The difference image reflects differences in intensities that exist between the two images. A local weight map is generated based on those differences. A global weight map is generated based on certain IMU data. A final weight map is generated based on a combination of the local weight map and the global weight map. The final weight map is used to generate a filtered image.

Abstract: A method for estimating human body temperature includes receiving, via a thermal camera, a thermal image captured of a real-world environment, the thermal image including thermal intensity values for each of a plurality of pixels of the thermal image. A position of a first human face is identified within the thermal image, the first human face corresponding to a first human subject. A skin tone of the first human face is identified. Based on the identified skin tone, a skin tone correction factor is applied to one or more thermal intensity values of one or more pixels corresponding to the first human face to give one or more tone-corrected thermal intensity values. Based on the one or more tone-corrected thermal intensity values an indication of a body temperature of the first human subject is reported.

Abstract: A system for obtaining color imagery using SPADs includes a SPAD array that has a plurality of SPAD pixels. Each of the plurality of SPAD pixels includes a respective color filter positioned thereover. The system is configurable to capture an image frame using the SPAD array and generate a filtered image by performing a temporal filtering operation using the image frame and at least one preceding image frame. The at least one preceding image frame is captured by the SPAD array at a timepoint that temporally precedes a timepoint associated with the image frame. The system is also configurable to, after performing the temporal filtering operation, generate a color image by demosaicing the filtered image.

Abstract: A system for single photon avalanche diode image capture is configurable to, over a frame capture time period, selectively activate an illuminator to alternately emit light from the illuminator and refrain from emitting light from the illuminator. The system is configurable to, over the frame capture time period, perform a plurality of sequential shutter operations to configure each SPAD pixel of the SPAD array to enable photon detection. The plurality of sequential shutter operations generates, for each SPAD pixel of the SPAD array, a plurality of binary counts indicating whether a photon was detected during each of the plurality of sequential shutter operations. The system is configurable to, based on a first set of binary counts of the plurality of binary counts, generate an ambient light image. The system is configurable to, based on a second set of binary counts of the plurality of binary counts, generate an illuminated image.

Abstract: A system for dark current compensation in SPAD imagery is configurable to capture an image frame with the SPAD array and generate a temporally filtered image by performing a temporal filtering operation using the image frame and at least one preceding image frame. The at least one preceding image frame is captured by the SPAD array at a timepoint that temporally precedes a timepoint associated with the image frame. The system is also configurable to obtain a dark current image frame. The dark current image frame includes data indicating one or more SPAD pixels of the plurality of SPAD pixels that detect an avalanche event without detecting a corresponding photon. The system is also configurable to generate a dark current compensated image by performing a subtraction operation on the temporally filtered image or the image frame based on the dark current image frame.

Abstract: A system for HDR image capture is configurable to perform a split long exposure operation by applying a first set of long exposure shutter operations to configure each sensor pixel of the image sensor array to enable photon detection and applying a second set of long exposure shutter operations to configure each sensor pixel to enable photon detection. A time period intervenes between the first and second sets of long exposure shutter. The system is configurable to perform a short exposure operation by applying a set of short exposure shutter operations to configure each sensor pixel to enable photon detection. The short exposure operation occurs during the time period that intervenes between the first and second sets of long exposure shutter operations. The system is also configurable to generate an image based on the split long exposure operation and the short exposure operation.

Abstract: Systems are configured for performing GPS-based and sensor-based relocalization. During the relocalization, the systems are configured to obtain radio-based positioning data indicating an estimated position of the system within a mapped environment. The systems are also configured to identify, based on the estimated position, a subset of keyframes of a map of the mapped environment, wherein the map of the mapped environment includes a plurality of keyframes captured from a plurality of locations within the mapped environment, and the plurality of keyframes are associated with anchor points identified within the mapped environment. The systems are further configured to perform relocalization within the mapped environment based on the subset of keyframes.

Abstract: Techniques for using motion data to generate a high resolution output color image from multiple images having sparse color information are disclosed. A camera generates multiple images. The camera's sensor is configured to have a sparse Bayer pattern. While the camera is generating the images, IMU data for each image is acquired. The IMU data indicates a corresponding pose the camera was in while the camera generated each image. The images and the IMU data are fed as input into a motion model. The motion model performs temporal filtering on the images and uses the IMU data to generate a red-only image, a green-only image, and a blue-only image. A high resolution output color image is generated by combining the red-only image, the green-only image, and the blue-only image.

Abstract: A system for structured light depth computation using single photon avalanche diodes (SPADs) is configurable to, over a frame capture time period, selectively activate the illuminator to perform interleaved structured light illumination operations. The interleaved structured light illumination operations comprise alternately emitting at least a first structured light pattern from the illuminator and emitting at least a second structured light pattern from the illuminator. The system is also configurable to, over the frame capture time period, perform a plurality of sequential shutter operations to configure each SPAD pixel of the SPAD array to enable photon detection. The plurality of sequential shutter operations generates, for each SPAD pixel of the SPAD array, a plurality of binary counts indicating whether a photon was detected during each of the plurality of sequential shutter operations.

Abstract: A near-eye display device comprises a pupil-expansion optic, first and second lasers, a drive circuit coupled operatively to the first and second lasers, a beam combiner, a spatial light modulator (SLM), and a computer. The first and second lasers are configured to emit in respective first and second wavelength bands. The beam combiner is configured to geometrically combine emission from the first and second lasers into a collimated beam. The SLM is configured to receive the collimated beam and to direct the emission in spatially modulated form to the pupil-expansion optic. The computer is configured to parse a digital image, trigger the emission from the first and second lasers by causing the drive circuit to drive current through the first and second lasers, and control the SLM such that the spatially modulated form of the emission projects an optical image corresponding to the digital image.

Abstract: A system for updating continuous image alignment of separate cameras identifies a previous alignment matrix associated with a previous frame pair captured at one or more previous timepoints by a reference camera and a match camera. The previous alignment matrix is based on visual correspondences in the previous frame pair. The system also identifies a current matrix associated with a current frame pair captured at one or more current timepoints by the reference camera and the match camera. The current matrix is based on visual correspondences in the current frame pair. The system also identifies a difference value associated with the reference camera or the match camera relative to the one or more previous timepoints and the one or more current timepoints. The system also generates an updated alignment matrix by using the previous alignment matrix, the current matrix, and the difference value as inputs.