Abstract: An image sensor configured to capture imagery with mitigated noise includes a plurality of image sensing pixels arranged to form a sensor array. Each image sensing pixel of the plurality of image sensing pixels comprises an active area configured to receive photons to facilitate image capture. Each active area comprises a length and a width. For at least one image sensing pixel of the plurality of image sensing pixels, the length or the width of the active area is smaller than about 80% of a pixel pitch measurement between the at least one image sensing pixel and an adjacent image sensing pixel. A size of the active area relative to the pixel pitch measurement contributes to mitigating sensor noise for the image sensor.

Abstract: A system for generating depth information from low-resolution images is configured to access a plurality of image frames capturing an environment, identify a first group of image frames from the plurality of image frames, and generate a first image comprising a first composite image of the environment using the first group of image frames as input. The first composite image has an image resolution that is higher than an image resolution of the image frames of the first group of image frames. The system is also configured to obtain a second image of the environment, where parallax exists between a capture perspective associated with the first image and a capture perspective associated with the second image. The system is also configured to generate depth information for the environment based on the first image and the second image.

Abstract: A system for facilitating intensity image capture and time of flight capture. The system includes an image sensor array comprising a plurality of image sensor pixels, one or more processors, and one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the system to facilitate intensity image capture and time of flight capture by configuring the system to perform interleaved intensity image capture and time of flight capture operations using the image sensor array.

Abstract: Techniques for updating a position of overlaid image content using IMU data to reflect subsequent changes in camera positions to minimize latency effects are disclosed. A “system camera” refers to an integrated camera that is a part of an HMD. An “external camera” is a camera that is separated from the HMD. The system camera and the external camera generate images. These images are overlaid on one another and aligned to form an overlaid image. Content from the external camera image is surrounded by a bounding element in the overlaid image. IMU data associated with both the system camera and the external camera is obtained. Based on that IMU data, an amount of movement that the system camera and/or the external camera have moved since the images were originally generated is determined. Based on that movement, the bounding element is shifted to a new position in the overlaid image.

Abstract: A system for generating high-resolution video from low-resolution images is configured to access a first video stream and a second video stream capturing an environment. The first video stream is captured by a first video capture device. The second video stream is captured by a second video capture device. Image frames of the first video stream are temporally synchronized with corresponding image frames of the second video stream. The system is also configured to generate a composite video stream with a higher resolution than the first or second video streams. Each composite image frame of the composite video stream is generated using a respective image frame of the first video stream and a temporally synchronized corresponding image frame of the second video stream as input.

Abstract: A digital image processing method performed by a computer is disclosed. A digital image captured by a real camera having intrinsic and extrinsic parameters is received. The intrinsic parameters include a native principal point defined relative to an origin of a coordinate system of the digital image. The digital image is sub-divided into a plurality of sub-images. For each sub-image of the plurality of sub-images, the sub-image is associated with a synthesized recapture camera having synthesized intrinsic and extrinsic parameters mapped from the real camera. The synthesized intrinsic parameters include the native principal point defined relative to an origin of a coordinate system of the sub-image.

Abstract: Techniques for generating a fused enhanced image. A first image is generated using a first camera of a first modality, and a second image is generated using a second camera of a second modality. Pixels that are common between the two images are identified. Textures for the common pixels are determined. A camera characteristic, which is linked to noise, is identified. A scaling factor is applied to the textures in the first image. A first saliency is determined using the scaled textures. A second saliency is determined using the textures from the second image. An alpha map is generated and reflects edge detection weights that have been computed for each one of the common pixels based on the two saliencies. Based on the alpha map, textures are merged from the common pixels to generate the fused enhanced image.

Abstract: Techniques to temporally filter images via a filtering weight computation are disclosed. A first image having a first timestamp and a second image having a second timestamp are acquired. These images are generated by a camera, and the first timestamp is before the second timestamp. A motion compensation (MC) operation is performed on the first image to produce an MC image. A difference image is generated using the MC image and the second image. The difference image reflects differences in intensities that exist between the two images. A local weight map is generated based on those differences. A global weight map is generated based on certain IMU data. A final weight map is generated based on a combination of the local weight map and the global weight map. The final weight map is used to generate a filtered image.

Abstract: A system for facilitating intensity image capture and time of flight capture. The system includes a single photon avalanche diode (SPAD) array comprising a plurality of SPAD pixels, one or more processors, and one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the system to facilitate intensity image capture and time of flight capture by configuring the system to perform interleaved intensity image capture and time of flight capture operations using the SPAD array.

Abstract: Examples are described that relate to correcting line bias in images. One example provides a method comprising receiving, from an imaging device, a plurality of images each comprising a plurality of lines of pixels. The method further comprises, for each image of the plurality of images, for each line of pixels of the plurality of lines of pixels, based at least on one or more pixel values of one or more pixels in the line of pixels, determining a line bias correction for the line, and applying the line bias correction to each pixel in the line, the line bias correction comprising an offset applied to each pixel value in the line of pixels.

Abstract: A computing system includes an illumination light source configured to emit illumination light into an external environment and an orientation sensor configured to estimate an orientation of the illumination light source relative to the external environment. The computing system includes a logic subsystem and a storage subsystem holding instructions executable by the logic subsystem to define a light restriction zone within the external environment. Based at least in part on the orientation of the illumination light source, the illumination light source is dynamically controlled to direct the illumination light toward at least a portion of the external environment outside the light restriction zone, while mitigating emission of the illumination light into the light restriction zone.

Abstract: Examples are disclosed that relate to providing image data to a user in a defined space of a surrounding environment from a perspective of the user. One example provides a computing system, comprising a logic subsystem, and a storage subsystem comprising instructions executable by the logic subsystem to obtain information regarding a pose of a user within a defined space, based upon the pose of the user, determine a portion of an environment surrounding the defined space toward which the user is looking, obtain image data representing the portion of the environment from a perspective of the user, and provide the image data for display via a display device within the defined space.

Abstract: A system for obtaining color imagery using SPADs includes a SPAD array that has a plurality of SPAD pixels. Each of the plurality of SPAD pixels includes a respective color filter positioned thereover. The system is configurable to capture an image frame using the SPAD array and generate a filtered image by performing a temporal filtering operation using the image frame and at least one preceding image frame. The at least one preceding image frame is captured by the SPAD array at a timepoint that temporally precedes a timepoint associated with the image frame. The system is also configurable to, after performing the temporal filtering operation, generate a color image by demosaicing the filtered image.

Abstract: A system for obtaining dark current images includes one or more processors and one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the system to perform various acts. The acts include obtaining a first image frame, generating a first low-pass filtered image by applying a low-pass filter to the first image frame, and generating a first estimated dark current image by subtracting the first low-pass filtered image from the first image frame.

Abstract: A system is configurable to, over a frame capture time period, selectively activate an illuminator to alternately emit light from the illuminator and refrain from emitting light from the illuminator. The system is configurable to, over the frame capture time period, perform a plurality of sequential shutter operations to configure each image sensor pixel of an image sensor array to enable photon detection. The plurality of sequential shutter operations generates, for each image sensor pixel of the image sensor array, a plurality of binary counts indicating whether a photon was detected during each of the plurality of sequential shutter operations. The system is configurable to, based on a first set of binary counts of the plurality of binary counts, generate an ambient light image. The system is configurable to, based on a second set of binary counts of the plurality of binary counts, generate an illuminated image.

Abstract: A system for dark current compensation in SPAD imagery is configurable to capture an image frame with the SPAD array and generate a temporally filtered image by performing a temporal filtering operation using the image frame and at least one preceding image frame. The at least one preceding image frame is captured by the SPAD array at a timepoint that temporally precedes a timepoint associated with the image frame. The system is also configurable to obtain a dark current image frame. The dark current image frame includes data indicating one or more SPAD pixels of the plurality of SPAD pixels that detect an avalanche event without detecting a corresponding photon. The system is also configurable to generate a dark current compensated image by performing a subtraction operation on the temporally filtered image or the image frame based on the dark current image frame.

Abstract: Techniques to temporally filter images via a filtering weight computation are disclosed. A first image having a first timestamp and a second image having a second timestamp are acquired. These images are generated by a camera, and the first timestamp is before the second timestamp. A motion compensation (MC) operation is performed on the first image to produce an MC image. A difference image is generated using the MC image and the second image. The difference image reflects differences in intensities that exist between the two images. A local weight map is generated based on those differences. A global weight map is generated based on certain IMU data. A final weight map is generated based on a combination of the local weight map and the global weight map. The final weight map is used to generate a filtered image.

Abstract: Techniques for generating a temporally filtered image designed to compensate for global motions of a camera and to compensate for local motions of an object are disclosed. A history frame and a current frame are acquired. A global motion compensation operation is performed on the history frame to reproject a pose of the history frame to match the pose of the current frame. The history frame is compared against the current frame to identify pixels that represent moving objects. For each of those pixels, an optical flow vector is computed. The optical flow vectors are then applied to those pixels to shift those pixels to new locations. These new positions, which are in the history frame, correspond to positions that were identified in the current frame. Afterwards, the current frame is temporally filtered with the history frame.

Abstract: A system for low compute high-resolution depth map generation using low-resolution cameras is configured to obtain a stereo pair of images and generate a depth map by performing stereo matching on the stereo pair of images. The system is also configured to obtain a first image comprising first texture information for the environment that has a first image resolution that is higher than an image resolution of images of the stereo pair of images. The system is further configured to generate a reprojected first image by reprojecting the first image to correspond to an image capture perspective associated with the depth map. The reprojection of the first image is based on depth information from the depth map and includes reprojected first texture information for the environment. The system is also configured to generate an upsampled depth map based on the depth map.

Abstract: Techniques for aligning image content are disclosed. The alignment occurs by concurrently identifying corner and line features and by aligning the images based on those features. As a result, relaxed constraints can be used during the alignment process as compared to constraints that are required when aligning images using only corner features. To achieve these benefits, first and second images are acquired. A first set of pixels that correspond to corner features are detected within the images. In parallel, a second set of pixels that correspond to line features are detected within the images. Image content is then aligned from the two images. The aligning process is performed using at least some of the pixels from the first and second sets.