Abstract: Examples are described that relate to correcting line bias in images. One example provides a method comprising receiving, from an imaging device, a plurality of images each comprising a plurality of lines of pixels. The method further comprises, for each image of the plurality of images, for each line of pixels of the plurality of lines of pixels, based at least on one or more pixel values of one or more pixels in the line of pixels, determining a line bias correction for the line, and applying the line bias correction to each pixel in the line, the line bias correction comprising an offset applied to each pixel value in the line of pixels.

Abstract: One aspect of this disclosure includes a method for operating a head-mounted display system that includes an imaging device. The method includes receiving an indication that an ambient light condition in an environment is below a lighting threshold. Responsive to the low lighting condition, an amount of motion of the head-mounted display relative to the environment is determined based on one or more signals received from an inertial measurement unit included in the head-mounted display system. An exposure time, frame rate, and a pixel-binning mode are automatically selected for the imaging device based on the determined amount of motion. Imagery is captured from the environment using the automatically selected exposure time, frame rate, and pixel-binning mode for the imaging device. The captured imagery is then displayed at the head-mounted display system.

Abstract: A system for facilitating the identifying of correspondences between images experiencing motion blur obtains a reference frame captured by a reference camera at a reference camera and obtains a match frame captured by a match camera at a match camera timepoint. The system identifies a motion attribute that includes (1) a reference camera motion attribute associated with the reference camera at the reference camera timepoint, and/or (2) a match camera motion attribute associated with the match camera at the match camera timepoint. The system determines a downsampling resolution using at least as inputs at least one of: the motion attribute, a camera exposure time, a camera field of view, or a camera angular resolution. The system generates a downsampled reference frame and a downsampled match frame based on the downsampling resolution. The system identifies correspondences between the downsampled reference frame and the downsampled match frame.

Abstract: Examples are disclosed that relate to blending different types of images captured by different types of cameras employing different sensing modalities based on a dynamic weighting. The dynamic weighting is calculated based on a dynamic quality proxy that serves as an approximation of image quality that may change from image to image. In one example, a first image of a scene is received from a first camera. A dynamic quality proxy is received. A second image of the scene is received from a second camera with a different sensing modality than the first camera. A composite image blended from the first and second images in proportion to a dynamic weighting that is based on the dynamic quality proxy is output.

Abstract: Techniques for updating a position of overlaid image content using IMU data to reflect subsequent changes in camera positions to minimize latency effects are disclosed. A “system camera” refers to an integrated camera that is a part of an HMD. An “external camera” is a camera that is separated from the HMD. The system camera and the external camera generate images. These images are overlaid on one another and aligned to form an overlaid image. Content from the external camera image is surrounded by a bounding element in the overlaid image. IMU data associated with both the system camera and the external camera is obtained. Based on that IMU data, an amount of movement that the system camera and/or the external camera have moved since the images were originally generated is determined. Based on that movement, the bounding element is shifted to a new position in the overlaid image.

Abstract: An HMD includes a single photon avalanche diode (SPAD) array comprising a plurality of SPAD pixels. The HMD also includes a display positioned to display images for viewing by an eye of a user. The HMD also includes one or more processors and one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the HMD to perform various acts associated with using the SPAD array to capture an image frame of an environment for display to the user.

Abstract: A system for facilitating intensity image capture and time of flight capture. The system includes a single photon avalanche diode (SPAD) array comprising a plurality of SPAD pixels, one or more processors, and one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the system to facilitate intensity image capture and time of flight capture by configuring the system to perform interleaved intensity image capture and time of flight capture operations using the SPAD array.

Abstract: Modifications are performed to cause a style of an image to match a different style. A first image is accessed, where the first image has the first style. A second image is also accessed, where the second image has a second style. Subsequent to a deep neural network (DNN) learning these styles, a copy of the first image is fed as input to the DNN. The DNN modifies the first image copy by transitioning the first image copy from being of the first style to subsequently being of the second style. As a consequence, a modified style of the transitioned first image copy bilaterally matches the second style.

Abstract: Enhanced passthrough images are generated and displayed. A current visibility condition of an environment is determined. Based on the current visibility condition, a first camera or a second camera, which detect light spanning different ranges of illuminance, is selected to generate a passthrough image of the environment. The selected camera is then caused to generate the passthrough image. Additionally, a third camera, which is structured to detect long wave infrared radiation, is caused to generate a thermal image of the environment. Parallax correction is performed by aligning coordinates of the thermal image with corresponding coordinates identified within the passthrough image. Subsequently, the parallax-corrected thermal image is overlaid onto the passthrough image to generate a composite passthrough image, which is then displayed.

Abstract: Mapping common features between images that commonly represent an environment using different light spectrum data is performed. A first image having first light spectrum data is accessed, and a second image having second light spectrum data is accessed. These images are fed as input to a DNN, which then identifies feature points that are common between the two images. A generated mapping lists the feature points and lists coordinates of the feature points from both of the images. Differences between the coordinates of the feature points in the two images are determined. Based on these differences, the second image is warped to cause the coordinates of the feature points in the second image to correspond to the coordinates of the feature points in the first image.

Abstract: Systems are configured for performing GPS-based and sensor-based relocalization. During the relocalization, the systems are configured to obtain radio-based positioning data indicating an estimated position of the system within a mapped environment. The systems are also configured to identify, based on the estimated position, a subset of keyframes of a map of the mapped environment, wherein the map of the mapped environment includes a plurality of keyframes captured from a plurality of locations within the mapped environment, and the plurality of keyframes are associated with anchor points identified within the mapped environment. The systems are further configured to perform relocalization within the mapped environment based on the subset of keyframes.

Abstract: Modifications are performed to cause a style of an image to match a different style. A first image is accessed, where the first image has the first style. A second image is also accessed, where the second image has a second style. Subsequent to a deep neural network (DNN) learning these styles, a copy of the first image is fed as input to the DNN. The DNN modifies the first image copy by transitioning the first image copy from being of the first style to subsequently being of the second style. As a consequence, a modified style of the transitioned first image copy bilaterally matches the second style.

Abstract: A system for updating continuous image alignment of separate cameras identifies a previous alignment matrix associated with a previous frame pair captured at one or more previous timepoints by a reference camera and a match camera. The previous alignment matrix is based on visual correspondences in the previous frame pair. The system also identifies a current matrix associated with a current frame pair captured at one or more current timepoints by the reference camera and the match camera. The current matrix is based on visual correspondences in the current frame pair. The system also identifies a difference value associated with the reference camera or the match camera relative to the one or more previous timepoints and the one or more current timepoints. The system also generates an updated alignment matrix by using the previous alignment matrix, the current matrix, and the difference value as inputs.

Abstract: A system for facilitating the identifying of correspondences between images experiencing motion blur obtains a reference frame captured by a reference camera at a reference camera and obtains a match frame captured by a match camera at a match camera timepoint. The system identifies a motion attribute that includes (1) a reference camera motion attribute associated with the reference camera at the reference camera timepoint, and/or (2) a match camera motion attribute associated with the match camera at the match camera timepoint. The system determines a downsampling resolution using at least as inputs at least one of: the motion attribute, a camera exposure time, a camera field of view, or a camera angular resolution. The system generates a downsampled reference frame and a downsampled match frame based on the downsampling resolution. The system identifies correspondences between the downsampled reference frame and the downsampled match frame.

Abstract: One method comprises receiving a hit signal from a device worn by a first player, receiving a position of the device, receiving an orientation of a launch axis of a virtual-projectile launcher, receiving a position of a second player, and outputting a hit assignment on determining, pursuant to receiving the hit signal, that a recognized object and the second player are coincident at an indicated launch of a virtual projectile. Another method comprises receiving an indication of launch of a virtual projectile by a virtual-projectile launcher of a first player, receiving an image aligned to a launch axis of the virtual-projectile launcher, outputting a hit signal to a server on determining, pursuant to receiving the indication of launch, that a recognized object is imaged in a projectile-delivery area of the image, and outputting a position of the device and an orientation of the launch axis to the server.

Abstract: A head-mounted device (HMD) is structured to include at least one computer vision camera that omits an IR light filter. Consequently, this computer vision's sensor is able to detect IR light, including IR laser light, in the environment. The HMD is configured to generate an image of the environment using the computer vision camera. This image is then fed as input into a machine learning (ML) algorithm that identifies IR laser light, which is detected by the sensor and which is recorded in the image. The HMD then visually displays a notification comprising information corresponding to the detected IR laser light.

Abstract: A system for continuous image alignment of separate cameras identifies a reference camera transformation matrix between a base reference camera pose and an updated reference camera pose. The system also identifies a match camera transformation matrix between a base match camera pose and an updated match camera pose and an alignment matrix based on visual correspondences between one or more reference frames captured by the reference camera and one or more match frames captured by the match camera. The system also generates a motion model configured to facilitate mapping of a set of pixels of a reference frame captured by the reference camera to a corresponding set of pixels of a match frame captured by the match camera based on the reference camera transformation matrix, the match camera transformation matrix, and the alignment matrix.

Abstract: A system for reducing a search area for identifying correspondences identifies an overlap region within a first match frame captured by a match camera. The overlap region includes one or more points of the first match frame that are associated with one or more same portions of an environment as one or more corresponding points of a first reference frame captured by a reference camera. The system obtains a second reference frame captured by the reference camera and a second match frame captured by the match camera. The system identifies a reference camera transformation matrix, and/or a match camera transformation matrix. The system defines a search area within the second match frame based on the overlap region and the reference camera transformation matrix and/or the match camera transformation matrix.

Abstract: Techniques for aligning and stabilizing images generated by an integrated stereo camera pair with images generated by a detached camera are disclosed. A first image is generated using a first stereo camera; a second image is generated using a second stereo camera; and a third image is generated using the detached camera. A first rotation base matrix is computed between the third and first images, and a second rotation base matrix is computed between the third and second images. The third image is aligned to the first image using the first rotation base matrix, and the third image is aligned to the second image using the second rotation base matrix. A first overlaid image is generated by overlaying the third image onto the first image, and a second overlaid image is generated by overlaying the third image onto the second image. The two overlaid images are parallax corrected and displayed.

Abstract: Techniques for aligning images generated by an integrated camera physically mounted to an HMD with images generated by a detached camera physically unmounted from the HMD are disclosed. A 3D feature map is generated and shared with the detached camera. Both the integrated camera and the detached camera use the 3D feature map to relocalize themselves and to determine their respective 6 DOF poses. The HMD receives the detached camera's image of the environment and the 6 DOF pose of the detached camera. A depth map of the environment is accessed. An overlaid image is generated by reprojecting a perspective of the detached camera's image to align with a perspective of the integrated camera and by overlaying the reprojected detached camera's image onto the integrated camera's image.