Abstract: Systems and methods are provided performing for low compute depth map generation by implementing acts of obtaining a stereo pair of images of a scene, downsampling the stereo pair of images, generating a depth map by stereo matching the downsampled stereo pair of images, and generating an upsampled depth map based on the depth map using an edge-preserving filter for obtaining at least some data of at least one image of the stereo pair of images.

Abstract: A system for low compute high-resolution depth map generation using low-resolution cameras is configured to obtain a stereo pair of images and generate a depth map by performing stereo matching on the stereo pair of images. The system is also configured to obtain a first image comprising first texture information for the environment that has a first image resolution that is higher than an image resolution of images of the stereo pair of images. The system is further configured to generate a reprojected first image by reprojecting the first image to correspond to an image capture perspective associated with the depth map. The reprojection of the first image is based on depth information from the depth map and includes reprojected first texture information for the environment. The system is also configured to generate an upsampled depth map based on the depth map.

Abstract: A system for generating high-resolution video from low-resolution images is configured to access a first video stream and a second video stream capturing an environment. The first video stream is captured by a first video capture device. The second video stream is captured by a second video capture device. Image frames of the first video stream are temporally synchronized with corresponding image frames of the second video stream. The system is also configured to generate a composite video stream with a higher resolution than the first or second video streams. Each composite image frame of the composite video stream is generated using a respective image frame of the first video stream and a temporally synchronized corresponding image frame of the second video stream as input.

Abstract: Techniques for generating an enhanced image. A first image is generated using a camera of a first modality, and a second image is generated using a camera of a second modality. Pixels that are common between the two images are identified. An alpha map is generated. The alpha map reflects edge detection weights that are computed for the common pixels based on saliency values. A determination is made as to how much texture from the images to use to generate an enhanced image. This determination is based on the edge detection weights included within the alpha map. Based on the edge detection weights, textures are merged from the common pixels to generate the enhanced image. Color is also added to the enhanced image, where the color reflects an additional property (e.g., the texture source for the pixel) that is associated with one or both of the images.

Abstract: A system for generating depth information from low-resolution images is configured to access a plurality of image frames capturing an environment, identify a first group of image frames from the plurality of image frames, and generate a first image comprising a first composite image of the environment using the first group of image frames as input. The first composite image has an image resolution that is higher than an image resolution of the image frames of the first group of image frames. The system is also configured to obtain a second image of the environment, where parallax exists between a capture perspective associated with the first image and a capture perspective associated with the second image. The system is also configured to generate depth information for the environment based on the first image and the second image.

Abstract: A head-mounted device (HMD) is structured to include at least one computer vision camera that omits an IR light filter. Consequently, this computer vision's sensor is able to detect IR light, including IR laser light, in the environment. The HMD is configured to generate an image of the environment using the computer vision camera. This image is then fed as input into a machine learning (ML) algorithm that identifies IR laser light, which is detected by the sensor and which is recorded in the image. The HMD then visually displays a notification comprising information corresponding to the detected IR laser light.

Abstract: Techniques for aligning images generated by an integrated camera physically mounted to an HMD with images generated by a detached camera physically unmounted from the HMD are disclosed. A 3D feature map is generated and shared with the detached camera. Both the integrated camera and the detached camera use the 3D feature map to relocalize themselves and to determine their respective 6 DOF poses. The HMD receives the detached camera's image of the environment and the 6 DOF pose of the detached camera. A depth map of the environment is accessed. An overlaid image is generated by reprojecting a perspective of the detached camera's image to align with a perspective of the integrated camera and by overlaying the reprojected detached camera's image onto the integrated camera's image.

Abstract: A system for HDR image capture is configurable to perform a split long exposure operation by applying a first set of long exposure shutter operations to configure each sensor pixel of the image sensor array to enable photon detection and applying a second set of long exposure shutter operations to configure each sensor pixel to enable photon detection. A time period intervenes between the first and second sets of long exposure shutter. The system is configurable to perform a short exposure operation by applying a set of short exposure shutter operations to configure each sensor pixel to enable photon detection. The short exposure operation occurs during the time period that intervenes between the first and second sets of long exposure shutter operations. The system is also configurable to generate an image based on the split long exposure operation and the short exposure operation.

Abstract: A system for continuous image alignment of separate cameras identifies a reference camera transformation matrix between a base reference camera pose and an updated reference camera pose. The system also identifies a match camera transformation matrix between a base match camera pose and an updated match camera pose and an alignment matrix based on visual correspondences between one or more reference frames captured by the reference camera and one or more match frames captured by the match camera. The system also generates a motion model configured to facilitate mapping of a set of pixels of a reference frame captured by the reference camera to a corresponding set of pixels of a match frame captured by the match camera based on the reference camera transformation matrix, the match camera transformation matrix, and the alignment matrix.

Abstract: Systems having rolling shutter sensors with a plurality of sensor rows are configured for compensating for rolling shutter artifacts that result from different sensor rows in the plurality of sensor rows outputting sensor data at different times. The systems compensate for the rolling shutter artifacts by identifying readout timepoints for the plurality of sensor rows of the rolling shutter sensor while the rolling shutter sensor captures an image of an environment and identifying readout poses each readout timepoint, as well as obtaining a depth map based on the image. The depth map includes a plurality of different rows of depth data that correspond to the different sensor rows. The system further compensates for the rolling shutter artifacts by generating a 3D representation of the environment while unprojecting the rows of depth data into 3D space using the readout poses.

Abstract: A system for reducing a search area for identifying correspondences identifies an overlap region within a first match frame captured by a match camera. The overlap region includes one or more points of the first match frame that are associated with one or more same portions of an environment as one or more corresponding points of a first reference frame captured by a reference camera. The system obtains a second reference frame captured by the reference camera and a second match frame captured by the match camera. The system identifies a reference camera transformation matrix, and/or a match camera transformation matrix. The system defines a search area within the second match frame based on the overlap region and the reference camera transformation matrix and/or the match camera transformation matrix.

Abstract: Systems and methods are provided for upsampling low temporal resolution depth maps. This upsampling is performed by obtaining a stereo pair of images of a scene captured at a first timepoint, generating a first depth map of the scene for the first timepoint by performing stereo matching on the stereo pair of images, obtaining a subsequent stereo pair of images captured at a subsequent timepoint to the first timepoint, and generating a subsequent depth map that corresponds to the subsequent timepoint by applying an edge-preserving filter using the first depth map without performing stereo matching on the subsequent stereo pair of images.

Abstract: A method for image colorization includes receiving, from a camera, an input image including a plurality of input image pixels. One or more input interest pixels of the plurality of input image pixels are classified as corresponding to an object of interest. A display image is generated having a plurality of display image pixels each having pixel values based on relative temperature values of objects in a real-world environment, the display image pixels including display interest pixels corresponding to the input interest pixels. The display interest pixels are colorized with a color selected based on a recognized class of the object of interest to give a colorized display image, the selected color being independent of the relative temperature values of the object of interest. The colorized display image is displayed with the display interest pixels colorized with the selected color.

Abstract: A system for HDR image capture is configurable to perform a split long exposure operation by applying a first set of long exposure shutter operations to configure each SPAD pixel of the SPAD array to enable photon detection and applying a second set of long exposure shutter operations to configure each SPAD pixel of the SPAD array to enable photon detection. A time period intervenes between the first and second sets of long exposure shutter. The system is configurable to perform a short exposure operation by applying a set of short exposure shutter operations to configure each SPAD pixel of the SPAD array to enable photon detection. The short exposure operation occurs during the time period that intervenes between the first and second sets of long exposure shutter operations. The system is also configurable to generate an image based on the split long exposure operation and the short exposure operation.

Abstract: A system for image acquisition with reduced noise using SPADs is configured to perform a plurality of sequential exposure and readout operations. Each exposure and readout operation includes (i) applying a set of shutter operations to configure each SPAD pixel of the SPAD array to enable photon detection, and (ii) for each SPAD pixel of the SPAD array, reading out a number of photons detected during the set of shutter operations. The system is also configured to generate an image based on the number of photons detected for each SPAD pixel during each of the plurality of sequential exposure and readout operations.

Abstract: Techniques for generating an enhanced image. A first image is generated using a first camera of a first modality, and a second image is generated using a second camera of a second modality. Pixels that are common between the two images are identified. Textures for the common pixels are determined. Saliencies of the two images are determined, where the saliencies reflect amounts of texture variation present in those images. An alpha map is generated and reflects edge detection weights that have been computed for each one of the common pixels based on the two saliencies. A determination is made as to how much texture from the first and/or second images to use to generate an enhanced image. This determining process is based on the edge detection weights included within the alpha map. Based on the edge detection weights, textures are merged from the common pixels to generate the enhanced image.

Abstract: Examples are disclosed that relate to motion compensation on a single photon avalanche detector (SPAD) array camera. One example provides a method enacted on an imaging device comprising a SPAD array camera and a motion sensor, the SPAD array camera comprising a plurality of pixels. The method comprises acquiring a plurality of subframes of image data. Each subframe of image data comprises a binary value for each pixel. Based upon motion data from the motion sensor, the method further comprises determining a change in pose of the imaging device between adjacent subframes, applying a positional offset to a current subframe based upon the motion data to align a location of a stationary imaged feature in the current subframe with a location of the stationary imaged feature in a prior subframe to create aligned subframes, summing the aligned subframes to form an image, and outputting the image.

Abstract: An image capture module configured for improved heat dissipation includes an image sensor, a first heat spreading element positioned to direct heat from the image sensor along a first heat dissipation path toward a first portion of the image capture module, a processing board in data communication with the image sensor, and a second heat spreading element positioned to dissipate heat from the processing board along a second heat dissipation path toward a second portion of the image capture module. Thermal isolation is used to isolate the different heat paths. The first heat dissipation path does not overlap the second heat dissipation path, the first portion of the image capture module is separate from the second portion of the image capture module.

Abstract: A method for enhancing digital imagery. The method comprises receiving a linear, intensity-based image of an environment. A histogram of intensity values is generated for a plurality of pixels within the linear, intensity-based image. Based on the histogram of intensity values, local contrast enhancement is applied to the linear, intensity-based image to generate a contrast enhanced version of the linear, intensity-based image, and artificial colorization is applied to the linear, intensity-based image to generate an artificially colorized version of the linear, intensity-based image. A composite image of the environment is then generated based on at least a portion of the contrast enhanced version of the linear, intensity-based image and at least a portion of the artificially colorized version of the linear, intensity-based image.

Abstract: A method for image colorization includes receiving, from a camera, an input image including a plurality of input image pixels. One or more input interest pixels of the plurality of input image pixels are classified as corresponding to an object of interest. A display image is generated having a plurality of display image pixels each having pixel values based on relative temperature values of objects in a real-world environment, the display image pixels including display interest pixels corresponding to the input interest pixels. The display interest pixels are colorized with a color selected based on a recognized class of the object of interest to give a colorized display image, the selected color being independent of the relative temperature values of the object of interest. The colorized display image is displayed with the display interest pixels colorized with the selected color.