Abstract: Optimizations are provided for generating passthrough visualizations for Head Mounted Displays. The interpupil distance of a user wearing a head-mounted device is determined and a stereo camera pair with a left and right camera is used to capture raw images. The center-line perspectives of the images captured by the left camera have non-parallel alignments with respect to center-line perspectives of any images captured by the right camera. After the raw images are captured, various camera distortion corrections are applied to the images to create corrected images. Epipolar transforms are then applied to the corrected images to create transformed images having parallel center-line perspectives. Thereafter, a depth map is generated of the transformed images. Finally, left and right passthrough visualizations are generated and rendered by reprojecting the transformed left and right images.

Abstract: A computer-implemented technique is described herein for invalidating pixels in a time-of-flight depth-sensing device based on active brightness (AB) measurements. In one implementation, the technique involves, for each sensing element of a sensor: generating frequency-specific sensor readings in response to receiving instances of radiation having plural frequencies (e.g., frequencies f1, f2, and f3); generating a set of active brightness measurements (ABf1, ABf2, and ABf3) associated with the respective frequencies; generating a variation measure that reflects an extent of variation within the set of active brightness measurements; and invalidating a pixel associated with the particular sensing element if the variation measure satisfies a prescribed invalidation condition.

Abstract: Optimizations are provided for generating passthrough visualizations for Head Mounted Displays. The interpupil distance of a user wearing a head-mounted device is determined and a stereo camera pair with a left and right camera is used to capture raw images. The center-line perspectives of the images captured by the left camera have non-parallel alignments with respect to center-line perspectives of any images captured by the right camera. After the raw images are captured, various camera distortion corrections are applied to the images to create corrected images. Epipolar transforms are then applied to the corrected images to create transformed images having parallel center-line perspectives. Thereafter, a depth map is generated of the transformed images. Finally, left and right passthrough visualizations are generated and rendered by reprojecting the transformed left and right images.

Abstract: Optimizations are provided for generating passthrough visualizations for Head Mounted Displays. The interpupil distance of a user wearing a head-mounted device is determined and a stereo camera pair with a left and right camera is used to capture raw images. The center-line perspectives of the images captured by the left camera have non-parallel alignments with respect to center-line perspectives of any images captured by the right camera. After the raw images are captured, various camera distortion corrections are applied to the images to create corrected images. Epipolar transforms are then applied to the corrected images to create transformed images having parallel center-line perspectives. Thereafter, a depth map is generated of the transformed images. Finally, left and right passthrough visualizations are generated and rendered by reprojecting the transformed left and right images.

Abstract: Techniques involving localized depth map generation, the techniques including receiving pixel data for a frame captured by an image sensor, the pixel data including at least one light intensity value, corresponding to an amount of light received by the image sensor during a frame period, for each of a plurality of pixels; identifying a subset of the pixels as being associated with a physical object detected based on at least the pixel data; selecting a region of the frame, the region corresponding to at least the subset of the pixels; and selectively generating a localized depth map for the frame period corresponding to the selected region. A portion of the frame outside of the selected region is not associated with a depth map generated for the frame period.

Abstract: A depth sensing system includes a scanning laser line projector configured to scan a laser line across a scene within the field of view of a differential imaging camera. The differential imaging camera outputs a stream of data that identifies indices for only pixels that have changed in intensity at continuous points in time. As the laser line is scanned across the scene, a line profile is observed by the differential imaging camera due to a disparity between the location of the scanning laser line projector and the differential imaging camera. The differential imaging camera streams data identifying the indices of pixels located along the line profile that have changed in intensity. A 3D depth map is generated based upon known positions of the laser line and the data received from the differential imaging camera. Scanning of the laser line can also be limited to a region of interest.

Abstract: Optimizations are provided for generating passthrough visualizations for Head Mounted Displays. The interpupil distance of a user wearing a head-mounted device is determined and a stereo camera pair with a left and right camera is used to capture raw images. The center-line perspectives of the images captured by the left camera have non-parallel alignments with respect to center-line perspectives of any images captured by the right camera. After the raw images are captured, various camera distortion corrections are applied to the images to create corrected images. Epipolar transforms are then applied to the corrected images to create transformed images having parallel center-line perspectives. Thereafter, a depth map is generated of the transformed images. Finally, left and right passthrough visualizations are generated and rendered by reprojecting the transformed left and right images.

Abstract: Techniques involving localized depth map generation, the techniques including receiving pixel data for a frame captured by an image sensor, the pixel data including at least one light intensity value, corresponding to an amount of light received by the image sensor during a frame period, for each of a plurality of pixels; identifying a subset of the pixels as being associated with a physical object detected based on at least the pixel data; selecting a region of the frame, the region corresponding to at least the subset of the pixels; and selectively generating a localized depth map for the frame period corresponding to the selected region. A portion of the frame outside of the selected region is not associated with a depth map generated for the frame period.

Abstract: A technique is described herein that employs a resource-efficient image capture system. The image capture system includes an active illumination system for emitting electromagnetic radiation within a physical environment. The image capture system also includes a camera system that includes one or more cameras for detecting electromagnetic radiation received from the physical environment, to produce image information. In one implementation, the technique involves using the same image capture system to produce different kinds of image information for consumption by different respective image processing components. The technique can perform this task by allocating timeslots over a span of time for producing the different kinds of image information. In one case, the image processing components include: a pose tracking component; a controller tracking component; and a surface reconstruction component, etc., any subset of which may be active at any given time.

Abstract: A technique is described herein for reducing blur caused by an imaging assembly of a depth camera system. In a runtime phase, the technique involves receiving a sensor image that is generated in response to return radiation reflected from a scene. The return radiation passes through an optical element (such as a visor element) of the imaging assembly, which produces blur due to the scattering of radiation. The technique then deconvolves the sensor image with a kernel, to provide a blur-reduced image. The kernel represents a point spread function that describes the distortion-related characteristics of at least the optical element. The technique then uses the blur-reduced image to calculate a depth image. The technique also encompasses a calibration-phase process for generating the kernel by modeling blur that occurs near an edge of a test object within a test image.

Abstract: A computer-implemented technique is described herein for invalidating pixels in a time-of-flight depth-sensing device based on active brightness (AB) measurements. In one implementation, the technique involves, for each sensing element of a sensor: generating frequency-specific sensor readings in response to receiving instances of radiation having plural frequencies (e.g., frequencies f1, f2, and f3); generating a set of active brightness measurements (ABf1, ABf2, and ABf3) associated with the respective frequencies; generating a variation measure that reflects an extent of variation within the set of active brightness measurements; and invalidating a pixel associated with the particular sensing element if the variation measure satisfies a prescribed invalidation condition.

Abstract: A hybrid sensor is described herein that includes a plurality of photoreceptive element domains. Each domain includes: a first subset of infrared (IR) photoreceptive elements that are selectively receptive to infrared radiation; and a second subset of visible-spectrum photoreceptive elements that are selectively receptive to visible spectrum light. The number of IR photoreceptive elements in the first subset is equal to or greater than a number of visible-spectrum photoreceptive elements in the second subset. In one case, each domain includes a mix of visible-spectrum photoreceptive elements that conforms to a Bayer pattern, or to some other color filer array pattern. Different applications can incorporate the hybrid sensor, including, but not limited to, scene reconstruction applications, object recognition applications, biometric authentication applications, night vision applications, etc.

Abstract: Systems and methods are provided for determining a depth map and a reflectivity map from a structured light image. The depth map can be determined by capturing the structured light image and then using a triangulation method to determine a depth map based on the dots in the captured structured light image. The reflectivity map can be determined based on the depth map and based on performing additional analysis of the dots in the captured structured light image.

Abstract: Systems and methods are provided for determining a depth map and a reflectivity map from a structured light image. The depth map can be determined by capturing the structured light image and then using a triangulation method to determine a depth map based on the dots in the captured structured light image. The reflectivity map can be determined based on the depth map and based on performing additional analysis of the dots in the captured structured light image.