Abstract: A system for single photon avalanche diode image capture is configurable to, over a frame capture time period, selectively activate an illuminator to alternately emit light from the illuminator and refrain from emitting light from the illuminator. The system is configurable to, over the frame capture time period, perform a plurality of sequential shutter operations to configure each SPAD pixel of the SPAD array to enable photon detection. The plurality of sequential shutter operations generates, for each SPAD pixel of the SPAD array, a plurality of binary counts indicating whether a photon was detected during each of the plurality of sequential shutter operations. The system is configurable to, based on a first set of binary counts of the plurality of binary counts, generate an ambient light image. The system is configurable to, based on a second set of binary counts of the plurality of binary counts, generate an illuminated image.

Abstract: A system for efficiently generating SPAD imagery with persistence is configurable to capture an image frame, capture pose data associated with the capturing of the image frame, and access a persistence frame. The persistence frame includes a preceding composite image frame generated based on at least two preceding image frames. The at least two preceding image frames are associated with timepoints that precede a capture timepoint associated with the image frame. The system is configurable to generate a persistence term based on (i) the pose data, (ii) a similarity comparison based on the image frame and the persistence frame, or (iii) a signal strength associated with the image frame. The system is configurable to generate a composite image based on the image frame, the persistence frame, and the persistence term. The persistence term defines a contribution of the image frame and the persistence frame to the composite image.

Abstract: Techniques for generating a fused enhanced image. A first image is generated using a first camera of a first modality, and a second image is generated using a second camera of a second modality. Pixels that are common between the two images are identified. Textures for the common pixels are determined. A camera characteristic, which is linked to noise, is identified. A scaling factor is applied to the textures in the first image. A first saliency is determined using the scaled textures. A second saliency is determined using the textures from the second image. An alpha map is generated and reflects edge detection weights that have been computed for each one of the common pixels based on the two saliencies. Based on the alpha map, textures are merged from the common pixels to generate the fused enhanced image.

Abstract: A system for adding persistence to SPAD imagery is configurable to capture, using a SPAD array, a plurality of image frames. The system is configurable to capture, using an IMU, pose data associated with the plurality of image frames. The pose data includes at least respective pose data associated with each of the plurality of image frames. The system is configurable to determine a persistence term based on the pose data. The system is also configurable to generate a composite image based on the plurality of image frames, the respective pose data associated with each of the plurality of image frames, and the persistence term. The persistence term defines a contribution of each of the plurality of image frames to the composite image.

Abstract: Systems and methods are provided for performing temporally consistent depth map generation by implementing acts of obtaining a first stereo pair of images of a scene associated with a first timepoint and a first pose, generating a first depth map of the scene based on the first stereo pair of images, obtaining a second stereo pair of images of the scene associated with at a second timepoint and a second pose, generating a reprojected first depth map by reprojecting the first depth map to align the first depth map with the second stereo pair of images, and generating a second depth map that corresponds to the second stereo pair of images using the reprojected first depth map.

Abstract: In one method, device data including an orientation of a targeting device is received in a computing system. Target coordinates of the targeting device as projected onto a field-of-view of a display device are then located based on the device data. Pursuant to locating the target coordinates within a predefined margin, a target graphic indicating the target coordinates is superposed onto the field-of-view. Pursuant to locating the target coordinates outside of the predefined margin, an off-target graphic is superposed onto the field-of-view and aligned to a display perimeter of the display device.

Abstract: Systems and methods are provided performing for low compute depth map generation by implementing acts of obtaining a stereo pair of images of a scene, downsampling the stereo pair of images, generating a depth map by stereo matching the downsampled stereo pair of images, and generating an upsampled depth map based on the depth map using an edge-preserving filter for obtaining at least some data of at least one image of the stereo pair of images.

Abstract: A system for low compute high-resolution depth map generation using low-resolution cameras is configured to obtain a stereo pair of images and generate a depth map by performing stereo matching on the stereo pair of images. The system is also configured to obtain a first image comprising first texture information for the environment that has a first image resolution that is higher than an image resolution of images of the stereo pair of images. The system is further configured to generate a reprojected first image by reprojecting the first image to correspond to an image capture perspective associated with the depth map. The reprojection of the first image is based on depth information from the depth map and includes reprojected first texture information for the environment. The system is also configured to generate an upsampled depth map based on the depth map.

Abstract: A system for generating high-resolution video from low-resolution images is configured to access a first video stream and a second video stream capturing an environment. The first video stream is captured by a first video capture device. The second video stream is captured by a second video capture device. Image frames of the first video stream are temporally synchronized with corresponding image frames of the second video stream. The system is also configured to generate a composite video stream with a higher resolution than the first or second video streams. Each composite image frame of the composite video stream is generated using a respective image frame of the first video stream and a temporally synchronized corresponding image frame of the second video stream as input.

Abstract: Techniques for generating an enhanced image. A first image is generated using a camera of a first modality, and a second image is generated using a camera of a second modality. Pixels that are common between the two images are identified. An alpha map is generated. The alpha map reflects edge detection weights that are computed for the common pixels based on saliency values. A determination is made as to how much texture from the images to use to generate an enhanced image. This determination is based on the edge detection weights included within the alpha map. Based on the edge detection weights, textures are merged from the common pixels to generate the enhanced image. Color is also added to the enhanced image, where the color reflects an additional property (e.g., the texture source for the pixel) that is associated with one or both of the images.

Abstract: A system for generating depth information from low-resolution images is configured to access a plurality of image frames capturing an environment, identify a first group of image frames from the plurality of image frames, and generate a first image comprising a first composite image of the environment using the first group of image frames as input. The first composite image has an image resolution that is higher than an image resolution of the image frames of the first group of image frames. The system is also configured to obtain a second image of the environment, where parallax exists between a capture perspective associated with the first image and a capture perspective associated with the second image. The system is also configured to generate depth information for the environment based on the first image and the second image.

Abstract: A head-mounted device (HMD) is structured to include at least one computer vision camera that omits an IR light filter. Consequently, this computer vision's sensor is able to detect IR light, including IR laser light, in the environment. The HMD is configured to generate an image of the environment using the computer vision camera. This image is then fed as input into a machine learning (ML) algorithm that identifies IR laser light, which is detected by the sensor and which is recorded in the image. The HMD then visually displays a notification comprising information corresponding to the detected IR laser light.

Abstract: Techniques for aligning images generated by an integrated camera physically mounted to an HMD with images generated by a detached camera physically unmounted from the HMD are disclosed. A 3D feature map is generated and shared with the detached camera. Both the integrated camera and the detached camera use the 3D feature map to relocalize themselves and to determine their respective 6 DOF poses. The HMD receives the detached camera's image of the environment and the 6 DOF pose of the detached camera. A depth map of the environment is accessed. An overlaid image is generated by reprojecting a perspective of the detached camera's image to align with a perspective of the integrated camera and by overlaying the reprojected detached camera's image onto the integrated camera's image.

Abstract: A system for HDR image capture is configurable to perform a split long exposure operation by applying a first set of long exposure shutter operations to configure each sensor pixel of the image sensor array to enable photon detection and applying a second set of long exposure shutter operations to configure each sensor pixel to enable photon detection. A time period intervenes between the first and second sets of long exposure shutter. The system is configurable to perform a short exposure operation by applying a set of short exposure shutter operations to configure each sensor pixel to enable photon detection. The short exposure operation occurs during the time period that intervenes between the first and second sets of long exposure shutter operations. The system is also configurable to generate an image based on the split long exposure operation and the short exposure operation.

Abstract: A structure for facilitating virtual experiences comprises: a structural support having a first side facing toward an interior of the structure, a second side opposite the first side, and an intra-support hollow disposed between the first and second sides. An IR reflective surface is adjacent to at least a portion of the second side of the structural support. An IR emitter within the hollow between the first and second sides is configured to emit IR light toward the IR reflective surface, such that the IR light is reflected toward the interior of the structure.

Abstract: A system for continuous image alignment of separate cameras identifies a reference camera transformation matrix between a base reference camera pose and an updated reference camera pose. The system also identifies a match camera transformation matrix between a base match camera pose and an updated match camera pose and an alignment matrix based on visual correspondences between one or more reference frames captured by the reference camera and one or more match frames captured by the match camera. The system also generates a motion model configured to facilitate mapping of a set of pixels of a reference frame captured by the reference camera to a corresponding set of pixels of a match frame captured by the match camera based on the reference camera transformation matrix, the match camera transformation matrix, and the alignment matrix.

Abstract: Systems having rolling shutter sensors with a plurality of sensor rows are configured for compensating for rolling shutter artifacts that result from different sensor rows in the plurality of sensor rows outputting sensor data at different times. The systems compensate for the rolling shutter artifacts by identifying readout timepoints for the plurality of sensor rows of the rolling shutter sensor while the rolling shutter sensor captures an image of an environment and identifying readout poses each readout timepoint, as well as obtaining a depth map based on the image. The depth map includes a plurality of different rows of depth data that correspond to the different sensor rows. The system further compensates for the rolling shutter artifacts by generating a 3D representation of the environment while unprojecting the rows of depth data into 3D space using the readout poses.

Abstract: A system for reducing a search area for identifying correspondences identifies an overlap region within a first match frame captured by a match camera. The overlap region includes one or more points of the first match frame that are associated with one or more same portions of an environment as one or more corresponding points of a first reference frame captured by a reference camera. The system obtains a second reference frame captured by the reference camera and a second match frame captured by the match camera. The system identifies a reference camera transformation matrix, and/or a match camera transformation matrix. The system defines a search area within the second match frame based on the overlap region and the reference camera transformation matrix and/or the match camera transformation matrix.

Abstract: Systems are provided for estimating 6DOF positioning of a computing device while in a pedestrian dead reckoning mode. The systems obtain a set of inertial tracking data from the set of one or more inertial tracking components while the system is in a pedestrian dead reckoning mode. Then, the systems obtain an estimated 3DOF velocity of the system based inertial tracking data, using a predictive model trained on a set of observed exteroceptive sensor data and observed inertial tracking data. The systems also obtain estimated 6DOF positioning of the systems based on the estimated 3DOF velocity.

Abstract: Systems and methods are provided for upsampling low temporal resolution depth maps. This upsampling is performed by obtaining a stereo pair of images of a scene captured at a first timepoint, generating a first depth map of the scene for the first timepoint by performing stereo matching on the stereo pair of images, obtaining a subsequent stereo pair of images captured at a subsequent timepoint to the first timepoint, and generating a subsequent depth map that corresponds to the subsequent timepoint by applying an edge-preserving filter using the first depth map without performing stereo matching on the subsequent stereo pair of images.