Abstract: A system for deformation or bending correction in an Augmented Reality (AR) system. Sensors are positioned in a frame of a head-worn AR system to sense forces or pressure acting on the frame by temple pieces attached to the frame. The sensed forces or pressure are used in conjunction with a model of the frame to determine a corrected model of the frame. The corrected model is used to correct video data captured by the AR system and to correct a video virtual overlay that is provided to a user wearing the head-worn AR system.

Abstract: Visual-inertial tracking of an eyewear device using a rolling shutter camera(s). The device includes a position determining system. Visual-inertial tracking is implemented by sensing motion of the device. An initial pose is obtained for a rolling shutter camera and an image of an environment is captured. The image includes feature points captured at a particular capture time. A number of poses for the rolling shutter camera is computed based on the initial pose and sensed movement of the device. The number of computed poses is responsive to the sensed movement of the mobile device. A computed pose is selected for each feature point in the image by matching the particular capture time for the feature point to the particular computed time for the computed pose. The position of the mobile device is determined within the environment using the feature points and the selected computed poses for the feature points.

Abstract: An eyewear device including a strain gauge sensor to determine when the eyewear device is manipulated by a user, such as being put on, taken off, and interacted with. A processor identifies a signature event based on sensor signals received from the strain gauge sensor and a data table of strain gauge sensor measurements corresponding to signature events. The processor controls the eyewear device as a function of the identified signature event, such as powering on a display of the eyewear device as the eyewear device is being put on a user's head, and then turning off the display when the eyewear device is removed from the user's head.

Abstract: Visual-inertial tracking of an eyewear device using sensors. The eyewear device monitors the sensors of a visual inertial odometry system (VIOS) that provide input for determining a position of the device within its environment. The eyewear device determines the status of the VIOS based information from the sensors and adjusts the plurality of sensors (e.g., by turning on/off sensors, changing the sampling rate, of a combination thereof) based on the determined status. The eyewear device then determines the position of the eyewear device within the environment using the adjusted plurality of sensors.

Abstract: An eyewear device having an image processor operable in a camera pipeline for computer vision (CV) and in augmented reality (AR) systems. The image processor is configured to selectively control a plurality of cameras to provide images having a first resolution in the high power AR mode, and to provide the images having a second resolution in the low power CV mode. The first resolution is higher than the second resolution, and the plurality of cameras consume less power in the CV mode than the AR mode. The image processor controls the camera pipeline to process the first resolution high IQ images from the plurality of cameras to operate in the AR mode, and controls the camera pipeline to process the second resolution lower IQ images from the plurality of cameras to operate in the CV mode. Substantial power is saved by reducing the resolution of the images using downscaling in the cameras themselves in the CV mode.

Abstract: An eyewear device with flexible frame for Augmented Reality (AR) is disclosed. At least two sensors and a display are mounted on the flexible frame. When in use, the real time geometry of the eyewear device may change from factory calibrated geometry, resulting in low quality AR rendering. A modeling module is provided to model the real time geometry of the eyewear device on the fly using sensor information of the at least two sensors. The modeled real time geometry is then provided to a rendering module to accurately display the AR to the user.

Abstract: Bending data is used to facilitate tracking operations of an extended reality (XR) device, such as hand tracking or other object tracking operations. The XR device obtains bending data indicative of bending of the XR device to accommodate a body part of a user wearing the XR device. The XR device determines, based on the bending data, whether to use previously identified biometric data in a tracking operation. A mode of the XR device is selected based on this determination. The XR device performs the tracking operation based on the selected mode. The selected mode may be a first mode in which the previously identified biometric data is used in the tracking operation or a second mode which does not apply previously identified biometric data in the tracking operation.

Abstract: A method for improving the startup time of a six-degrees of freedom tracking system is described. An augmented reality system receives a device initialization request and activates a first set of sensors in response to the device initialization request. The augmented reality system receives first tracking data from the first set of sensors. The augmented reality system receives an augmented reality experience request and in response to the augmented reality request, causes display of a set of augmented reality content items based on the first tracking data and simultaneously activates a second set of sensors. The augmented reality system receives second tracking data from the activated second set of sensors. The augmented reality system updates the display of the set of augmented reality content items based on the second tracking data.

Abstract: Bending data is used to facilitate tracking operations of an extended reality (XR) device, such as hand tracking or other object tracking operations. The XR device obtains bending data indicative of bending of the XR device to accommodate a body part of a user wearing the XR device. The XR device determines, based on the bending data, whether to use previously identified biometric data in a tracking operation. A mode of the XR device is selected responsive to determining whether to use the previously identified biometric data. The selected mode is used to initialize the tracking operation. The selected mode may be a first mode in which the previously identified biometric data is used in the tracking operation or a second mode in which the previously identified biometric data is not used in the tracking operation.

Abstract: A method for recalibrating an augmented reality (AR) device includes generating and storing a ground truth map of a real-world environment when the AR device is operating with a high likelihood of having an accurate factory calibration. During operation of the AR device, new map data is generated for the real-world environment. The new map data is compared to the ground truth map to detect potential calibration errors. If calibration errors are detected, a recalibration procedure is executed by determining an optimal path through the real-world environment that allows for observing parameters requiring recalibration. Visual cues are generated to guide a user of the AR device through the optimal path. As the user follows the visual cues, calibration parameters are iteratively adjusted to eliminate detected calibration errors. The recalibration procedure may be presented as an interactive game to improve user engagement, with rewards provided for accurately following guidance.

Abstract: Bending data is used to facilitate tracking operations of an extended reality (XR) device, such as hand tracking or other object tracking operations. The XR device obtains bending data indicative of bending of the XR device to accommodate a body part of a user wearing the XR device. The XR device determines, based on the bending data, whether to use previously identified biometric data in a tracking operation. A mode of the XR device is selected responsive to determining whether to use the previously identified biometric data. The selected mode is used to initialize the tracking operation. The selected mode may be a first mode in which the previously identified biometric data is used in the tracking operation or a second mode in which the previously identified biometric data is not used in the tracking operation.

Abstract: Visual-inertial tracking of an eyewear device using sensors. The eyewear device monitors the sensors of a visual inertial odometry system (VIOS) that provide input for determining a position of the device within its environment. The eyewear device determines the status of the VIOS based information from the sensors and adjusts the plurality of sensors (e.g., by turning on/off sensors, changing the sampling rate, of a combination thereof) based on the determined status. The eyewear device then determines the position of the eyewear device within the environment using the adjusted plurality of sensors.

Abstract: An eyewear device having an image processor operable in a camera pipeline for computer vision (CV) and in augmented reality (AR) systems. The image processor is configured to selectively control a plurality of cameras to provide images having a first resolution in the high power AR mode, and to provide the images having a second resolution in the low power CV mode. The first resolution is higher than the second resolution, and the plurality of cameras consume less power in the CV mode than the AR mode. The image processor controls the camera pipeline to process the first resolution high IQ images from the plurality of cameras to operate in the AR mode, and controls the camera pipeline to process the second resolution lower IQ images from the plurality of cameras to operate in the CV mode. Substantial power is saved by reducing the resolution of the images using downscaling in the cameras themselves in the CV mode.

Abstract: A method for correcting a bending of a flexible device is described. In one aspect, the method includes accessing feature data of a first stereo frame that is generated by stereo optical sensors of the flexible device, the feature data generated based on a visual-inertial odometry (VIO) system of the flexible device, accessing depth map data of the first stereo frame, the depth map data generated based on a depth map system of the flexible device, estimating a pitch-roll bias and a yaw bias based on the features data and the depth map data of the first stereo frame, and generating a second stereo frame after the first stereo frame, the second stereo frame based on the pitch-roll bias and the yaw bias of the first stereo frame.

Abstract: A method for improving the startup time of a six-degrees of freedom tracking system is described. An augmented reality system receives a device initialization request and activates a first set of sensors in response to the device initialization request. The augmented reality system receives first tracking data from the first set of sensors. The augmented reality system receives an augmented reality experience request and in response to the augmented reality request, causes display of a set of augmented reality content items based on the first tracking data and simultaneously activates a second set of sensors. The augmented reality system receives second tracking data from the activated second set of sensors. The augmented reality system updates the display of the set of augmented reality content items based on the second tracking data.

Abstract: A method for calibrating a visual-inertial tracking system is described. A device operates the visual-inertial tracking system without receiving a tracking request from a virtual object display application. In response to operating the visual-inertial tracking system, the device accesses sensor data from sensors at the device. The device identifies, based on the sensor data, a first calibration parameter value of the visual-inertial tracking system and stores the first calibration parameter value. The system detects a tracking request from the virtual object display application. In response to the tracking request, the system accesses the first calibration parameter value and determines a second calibration parameter value from the first calibration parameter value.

Abstract: A method for mitigating motion blur in a visual-inertial tracking system is described. In one aspect, the method includes accessing a first image generated by an optical sensor of the visual tracking system, accessing a second image generated by the optical sensor of the visual tracking system, the second image following the first image, determining a first motion blur level of the first image, determining a second motion blur level of the second image, identifying a scale change between the first image and the second image, determining a first optimal scale level for the first image based on the first motion blur level and the scale change, and determining a second optimal scale level for the second image based on the second motion blur level and the scale change.

Abstract: A method for mitigating motion blur in a visual tracking system is described. In one aspect, a method for selective motion blur mitigation in a visual tracking system includes accessing a first image generated by an optical sensor of the visual tracking system, identifying camera operating parameters of the optical sensor during the optical sensor generating the first image, determining a motion of the optical sensor during the optical sensor generating the first image, determining a motion blur level of the first image based on the camera operating parameters of the optical sensor and the motion of the optical sensor, and determining whether to downscale the first image using a pyramid computation algorithm based on the motion blur level.

Abstract: A method for calibrating a visual-inertial tracking system is described. A device operates the visual-inertial tracking system without receiving a tracking request from a virtual object display application. In response to operating the visual-inertial tracking system, the device accesses sensor data from sensors at the device. The device identifies, based on the sensor data, a first calibration parameter value of the visual-inertial tracking system and stores the first calibration parameter value. The system detects a tracking request from the virtual object display application. In response to the tracking request, the system accesses the first calibration parameter value and determines a second calibration parameter value from the first calibration parameter value.

Abstract: A method for mitigating motion blur in a visual-inertial tracking system is described. In one aspect, the method includes accessing a first image generated by an optical sensor of the visual tracking system, accessing a second image generated by the optical sensor of the visual tracking system, the second image following the first image, determining a first motion blur level of the first image, determining a second motion blur level of the second image, identifying a scale change between the first image and the second image, determining a first optimal scale level for the first image based on the first motion blur level and the scale change, and determining a second optimal scale level for the second image based on the second motion blur level and the scale change.