Abstract: A method for image colorization includes receiving, from a camera, an input image including a plurality of input image pixels. One or more input interest pixels of the plurality of input image pixels are classified as corresponding to an object of interest. A display image is generated having a plurality of display image pixels each having pixel values based on relative temperature values of objects in a real-world environment, the display image pixels including display interest pixels corresponding to the input interest pixels. The display interest pixels are colorized with a color selected based on a recognized class of the object of interest to give a colorized display image, the selected color being independent of the relative temperature values of the object of interest. The colorized display image is displayed with the display interest pixels colorized with the selected color.

Abstract: A system for HDR image capture is configurable to perform a split long exposure operation by applying a first set of long exposure shutter operations to configure each SPAD pixel of the SPAD array to enable photon detection and applying a second set of long exposure shutter operations to configure each SPAD pixel of the SPAD array to enable photon detection. A time period intervenes between the first and second sets of long exposure shutter. The system is configurable to perform a short exposure operation by applying a set of short exposure shutter operations to configure each SPAD pixel of the SPAD array to enable photon detection. The short exposure operation occurs during the time period that intervenes between the first and second sets of long exposure shutter operations. The system is also configurable to generate an image based on the split long exposure operation and the short exposure operation.

Abstract: A system for image acquisition with reduced noise using SPADs is configured to perform a plurality of sequential exposure and readout operations. Each exposure and readout operation includes (i) applying a set of shutter operations to configure each SPAD pixel of the SPAD array to enable photon detection, and (ii) for each SPAD pixel of the SPAD array, reading out a number of photons detected during the set of shutter operations. The system is also configured to generate an image based on the number of photons detected for each SPAD pixel during each of the plurality of sequential exposure and readout operations.

Abstract: Techniques for generating an enhanced image. A first image is generated using a first camera of a first modality, and a second image is generated using a second camera of a second modality. Pixels that are common between the two images are identified. Textures for the common pixels are determined. Saliencies of the two images are determined, where the saliencies reflect amounts of texture variation present in those images. An alpha map is generated and reflects edge detection weights that have been computed for each one of the common pixels based on the two saliencies. A determination is made as to how much texture from the first and/or second images to use to generate an enhanced image. This determining process is based on the edge detection weights included within the alpha map. Based on the edge detection weights, textures are merged from the common pixels to generate the enhanced image.

Abstract: A method to process a contributing digital image of a subject in an image-processing computer. The contributing digital image is received in a depth-resolving machine configured to furnish a depth image based at least in part on the contributing digital image. The contributing digital image is also received in a classification machine previously trained to classify a pixel of the contributing digital image as liable to corrupt a depth value of a corresponding pixel of the depth image. A repair value is computed for the depth value of the corresponding pixel of the depth image, which is then corrected based on the repair value and returned to the calling process.

Abstract: Techniques are provided to re-arrange the placement of a photodiode within an illumination system to achieve improved characteristics and reduced form factor. An illumination system includes a laser assembly, a MEMS mirror system, a beam combiner, and a photodiode. The laser assembly includes RGB lasers, and the MEMS mirror system redirects laser light produced by the RGB lasers to illuminate pixels in an image frame. The beam combiner combines the laser light. The photodiode is provided to determine a power output of the laser assembly by receiving and measuring some of the laser light. The photodiode may be beneficially positioned before or after collimating optics and/or the beam combiner.

Abstract: Examples are disclosed that relate to motion compensation on a single photon avalanche detector (SPAD) array camera. One example provides a method enacted on an imaging device comprising a SPAD array camera and a motion sensor, the SPAD array camera comprising a plurality of pixels. The method comprises acquiring a plurality of subframes of image data. Each subframe of image data comprises a binary value for each pixel. Based upon motion data from the motion sensor, the method further comprises determining a change in pose of the imaging device between adjacent subframes, applying a positional offset to a current subframe based upon the motion data to align a location of a stationary imaged feature in the current subframe with a location of the stationary imaged feature in a prior subframe to create aligned subframes, summing the aligned subframes to form an image, and outputting the image.

Abstract: An example computing system comprises a processor and a storage device holding instructions executable by the processor to receive a thermal image acquired via a thermal imaging system, each pixel of the thermal image comprising an intensity level, and generate a histogram via binning pixels by intensity level. The instructions are further executable to, based at least on the histogram, determine a subset of pixels to colorize, colorize the subset of pixels to produce a selectively colorized image, and output the selectively colorized image.

Abstract: An image capture module configured for improved heat dissipation includes an image sensor, a first heat spreading element positioned to direct heat from the image sensor along a first heat dissipation path toward a first portion of the image capture module, a processing board in data communication with the image sensor, and a second heat spreading element positioned to dissipate heat from the processing board along a second heat dissipation path toward a second portion of the image capture module. Thermal isolation is used to isolate the different heat paths. The first heat dissipation path does not overlap the second heat dissipation path, the first portion of the image capture module is separate from the second portion of the image capture module.

Abstract: A method for enhancing digital imagery. The method comprises receiving a linear, intensity-based image of an environment. A histogram of intensity values is generated for a plurality of pixels within the linear, intensity-based image. Based on the histogram of intensity values, local contrast enhancement is applied to the linear, intensity-based image to generate a contrast enhanced version of the linear, intensity-based image, and artificial colorization is applied to the linear, intensity-based image to generate an artificially colorized version of the linear, intensity-based image. A composite image of the environment is then generated based on at least a portion of the contrast enhanced version of the linear, intensity-based image and at least a portion of the artificially colorized version of the linear, intensity-based image.

Abstract: A method for image colorization includes receiving, from a camera, an input image including a plurality of input image pixels. One or more input interest pixels of the plurality of input image pixels are classified as corresponding to an object of interest. A display image is generated having a plurality of display image pixels each having pixel values based on relative temperature values of objects in a real-world environment, the display image pixels including display interest pixels corresponding to the input interest pixels. The display interest pixels are colorized with a color selected based on a recognized class of the object of interest to give a colorized display image, the selected color being independent of the relative temperature values of the object of interest. The colorized display image is displayed with the display interest pixels colorized with the selected color.

Abstract: Examples are described that relate to correcting line bias in images. One example provides a method comprising receiving, from an imaging device, a plurality of images each comprising a plurality of lines of pixels. The method further comprises, for each image of the plurality of images, for each line of pixels of the plurality of lines of pixels, based at least on one or more pixel values of one or more pixels in the line of pixels, determining a line bias correction for the line, and applying the line bias correction to each pixel in the line, the line bias correction comprising an offset applied to each pixel value in the line of pixels.

Abstract: One aspect of this disclosure includes a method for operating a head-mounted display system that includes an imaging device. The method includes receiving an indication that an ambient light condition in an environment is below a lighting threshold. Responsive to the low lighting condition, an amount of motion of the head-mounted display relative to the environment is determined based on one or more signals received from an inertial measurement unit included in the head-mounted display system. An exposure time, frame rate, and a pixel-binning mode are automatically selected for the imaging device based on the determined amount of motion. Imagery is captured from the environment using the automatically selected exposure time, frame rate, and pixel-binning mode for the imaging device. The captured imagery is then displayed at the head-mounted display system.

Abstract: A system for facilitating the identifying of correspondences between images experiencing motion blur obtains a reference frame captured by a reference camera at a reference camera and obtains a match frame captured by a match camera at a match camera timepoint. The system identifies a motion attribute that includes (1) a reference camera motion attribute associated with the reference camera at the reference camera timepoint, and/or (2) a match camera motion attribute associated with the match camera at the match camera timepoint. The system determines a downsampling resolution using at least as inputs at least one of: the motion attribute, a camera exposure time, a camera field of view, or a camera angular resolution. The system generates a downsampled reference frame and a downsampled match frame based on the downsampling resolution. The system identifies correspondences between the downsampled reference frame and the downsampled match frame.

Abstract: Examples are disclosed that relate to blending different types of images captured by different types of cameras employing different sensing modalities based on a dynamic weighting. The dynamic weighting is calculated based on a dynamic quality proxy that serves as an approximation of image quality that may change from image to image. In one example, a first image of a scene is received from a first camera. A dynamic quality proxy is received. A second image of the scene is received from a second camera with a different sensing modality than the first camera. A composite image blended from the first and second images in proportion to a dynamic weighting that is based on the dynamic quality proxy is output.

Abstract: Techniques for updating a position of overlaid image content using IMU data to reflect subsequent changes in camera positions to minimize latency effects are disclosed. A “system camera” refers to an integrated camera that is a part of an HMD. An “external camera” is a camera that is separated from the HMD. The system camera and the external camera generate images. These images are overlaid on one another and aligned to form an overlaid image. Content from the external camera image is surrounded by a bounding element in the overlaid image. IMU data associated with both the system camera and the external camera is obtained. Based on that IMU data, an amount of movement that the system camera and/or the external camera have moved since the images were originally generated is determined. Based on that movement, the bounding element is shifted to a new position in the overlaid image.

Abstract: An HMD includes a single photon avalanche diode (SPAD) array comprising a plurality of SPAD pixels. The HMD also includes a display positioned to display images for viewing by an eye of a user. The HMD also includes one or more processors and one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the HMD to perform various acts associated with using the SPAD array to capture an image frame of an environment for display to the user.

Abstract: A system for facilitating intensity image capture and time of flight capture. The system includes a single photon avalanche diode (SPAD) array comprising a plurality of SPAD pixels, one or more processors, and one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the system to facilitate intensity image capture and time of flight capture by configuring the system to perform interleaved intensity image capture and time of flight capture operations using the SPAD array.

Abstract: Modifications are performed to cause a style of an image to match a different style. A first image is accessed, where the first image has the first style. A second image is also accessed, where the second image has a second style. Subsequent to a deep neural network (DNN) learning these styles, a copy of the first image is fed as input to the DNN. The DNN modifies the first image copy by transitioning the first image copy from being of the first style to subsequently being of the second style. As a consequence, a modified style of the transitioned first image copy bilaterally matches the second style.

Abstract: Enhanced passthrough images are generated and displayed. A current visibility condition of an environment is determined. Based on the current visibility condition, a first camera or a second camera, which detect light spanning different ranges of illuminance, is selected to generate a passthrough image of the environment. The selected camera is then caused to generate the passthrough image. Additionally, a third camera, which is structured to detect long wave infrared radiation, is caused to generate a thermal image of the environment. Parallax correction is performed by aligning coordinates of the thermal image with corresponding coordinates identified within the passthrough image. Subsequently, the parallax-corrected thermal image is overlaid onto the passthrough image to generate a composite passthrough image, which is then displayed.