Abstract: A system for determining the gaze endpoint of a subject, the system comprising: a eye tracking unit adapted to determine the gaze direction of one or more eyes of the subject; a head tracking unit adapted to determine the position comprising location and orientation of the eye tracker with respect to a reference coordinate system; a 3D Structure representation unit, that uses the 3D structure and position of objects of the scene in the reference coordinate system to provide a 3D structure representation of the scene; based on the gaze direction, the eye tracker position and the 3D structure representation, calculating the gaze endpoint on an object of the 3D structure representation of the scene or determining the object itself.

Abstract: Methods and apparatus for providing eye model matching in a device are disclosed. When a user activates a device and the presence of the user's eye is detected, an image of the user's eye is captured. An eye model matching process is then implemented to determine a stored eye model (e.g., an eye model stored after enrollment of the eye on the device) that best matches the eye in the captured image. Determination of the best matching eye model may be based on matching between properties of the user's eye in the captured image (such as cornea and pupil features) and properties of the user's eye determined by the eye model. The best matching eye model may then be implemented in, for example, an eye gaze tracking process. In certain instances, the best matching eye model satisfies a threshold for matching before being implemented in the downstream process.

Abstract: A personalized eye model is used to generate synthetic gaze features at ground-truth eye poses Gg. Corresponding synthetic gaze poses Gs are estimated from the synthetic gaze features using an average eye model. A linear regression is applied between Gg and Gs to generate a gaze correction function. The gaze correction function represents differences between the synthetic gaze Gs of the subject eye at the display and that of the average eye model Gg at the display, but does not contain security- or privacy-sensitive information. Further, the personalized eye model cannot be recovered from the gaze correction function, and thus the gaze correction function can be stored unencrypted and available for use during a cold boot of a device prior to login. On a cold boot of the device, the gaze correction function may be accessed and used with an average eye model to improve gaze-based interactions.

Abstract: Systems and methods are disclosed to enable performance of a lens distance test in head-mounted displays (HMDs) to determine the distance between a user's eye and a lens of the HMD (e.g. the display lens in a virtual or augmented reality device). In embodiments, the HMD is configured to determine a current pose of the eye based on a series of captured eye images. The pose information is used to determine the distance from the apex of the cornea to a closest point on the lens. If the determined distance is too small or too large, an alert or notification is generated instructing to adjust the HMD or change the light seal to achieve better distancing, in order to reduce the risk of eye injury and/or improve user experience. In embodiments, the lens distance test may be repeated during a user session to reevaluate and/or monitor the lens distance.

Abstract: A system for detecting a proximate object includes one or more cameras and one or more illuminators. The proximate object is detected by obtaining image data captured by the one or more cameras, where the image data is captured when at least one of the one or more illuminators are illuminated, determining brightness statistics from the image data, and determining whether the brightness statistics satisfies a predetermined threshold. The proximate object is determined to be detected in accordance with a determination that the brightness statistics satisfies the predetermined threshold.

Abstract: Various implementations disclosed herein include devices, systems, and methods that provide color visual markers that include colored markings that encode data, where the colors of the colored markings are determined by scanning (e.g., detecting the visual marker using a sensor of an electronic device) the visual marker itself. In some implementations, a visual marker is detected in an image of a physical environment. In some implementations, the visual marker is detected in the image by detecting a predefined shape of a first portion of the visual marker in the image. Then, a color-interpretation scheme is determined for interpreting colored markings of the visual marker that encode data by identifying a set of colors at a corresponding set of predetermined locations on the visual marker. Then, the data of the visual marker is decoded using the colored markings and the set of colors of the color-interpretation scheme.

Abstract: Methods and apparatus for using polarized light (e.g., infrared (IR) light) to improve eye-related functions such as iris recognition. An eye camera system includes one or more IR cameras that capture images of the user's eyes that are processed to perform iris recognition, gaze tracking, or other functions. At least one polarizing element may be located in the path of the light which is used to capture images of the user's eye. The user's eye may be illuminated by IR light emitted by one or more LEDs. At least one of the LEDs may be an LED with a polarizing filter. Instead or in addition, at least one polarizer may be located at the eye camera sensor, in the eye camera optics, or as or in an additional optical element located on the light path between the eye camera and the user's eye.

Abstract: Method for subsequently displaying a stream of images in a display area of a display device, wherein a first area within the display area is determined around a point of regard. A first image of the stream of images is displayed in the display area such that a first part of the first image, which is displayed in the first area, is displayed according to a first parameter value of at least one parameter and a second part of the first image, which is displayed in at least one second area outside the first area, is displayed according to a second parameter value of the at least one parameter. Moreover, the determining of the point of regard is performed as predicting the point of regard for a certain future point of time, at which the first image is displayed.

Abstract: Various implementations disclosed herein include an inconspicuous eye enrollment for user devices through the simultaneous or near-simultaneous eye characteristic tracking and eye model updating. The implementations disclosed herein direct light towards an eye of a user to produce glint reflections and obtain, via one or more sensors, image frames comprising depictions of the glint reflections. Based on those image frames, the implementations disclosed herein track eye characteristics and update an eye model of the eye during a period of time between obtaining a first image frame of the image frames and obtaining a second image frame of the image frames.

Abstract: Methods and apparatus for biometric authentication in which two or more biometric features or aspects are captured and analyzed individually or in combination to identify and authenticate a person. An imaging system captures images of a person's iris, eye, periorbital region, and/or other regions of the person's face, and two or more features from the captured images are analyzed individually or in combination to identify and authenticate the person and/or to detect attempts to spoof the biometric authentication. Embodiments may improve the performance of biometric authentication systems, and may help to reduce false positives and false negatives by the biometric authentication algorithms.

Abstract: Methods and apparatus for biometric authentication in which two or more cameras are used to capture images of biometric features or aspects for analysis to identify and authenticate a person. An imaging system includes at least two cameras that are used to capture images of a person's iris, eye, periorbital region, and/or other regions of the person's face, and one or more features from the captured images are analyzed to identify and authenticate the person or to detect attempts to spoof the biometric authentication. Information from two or more images may be combined to process aspects and features extracted from the combined images. Alternatively, one of the images to be used for biometric authentication may be determined, for example using one or more objective criteria to evaluate the quality of the captured images.

Abstract: Methods and apparatus for flexible illumination that improve the performance and robustness of an imaging system are described. Multiple different lighting configurations for the imaging system are pre-generated. Each lighting configuration may specify one or more aspects of lighting. A lookup table may be generated via which each pose is associated with a respective lighting configuration. A user may put on, hold, or otherwise use the device. A process may be initiated in which different lighting configurations may be selected by the controller to capture images of the user's eye, periorbital region, or face at different poses and in different conditions for use by a biometric authentication or gaze tracking process.

Abstract: Methods and apparatus for biometric authentication in which a current eye pose is determined and evaluated to determine if the current pose is satisfactory, and in which the eye pose may be improved by the user manually adjusting the device or their pose/gaze direction in response to a signal from the controller, and/or in which the imaging system is mechanically adjusted at the direction of the controller to improve the current view of the eye.

Abstract: Low-power eye tracking for detecting position and movements of a user’s eyes in a head-mounted device (HMD). An eye tracking system for an HMD may include eye tracking cameras and eye odometers that may reduce power consumption of the system. The eye odometers may be used as a low-power component to track relative movement of the user’s eyes with respect to the HMD between frames captured by the eye tracking cameras, which may allow the frame rate of the eye tracking cameras to be reduced.

Abstract: Methods and apparatus for flexible illumination in which two or more different wavelengths are used in the illumination system. An illumination source may be configured to emit light at two or more different wavelengths, either continuously or selectively. A method may be implemented in which a first wavelength is emitted for capturing an image or images for a first portion of algorithmic processing for biometric authentication, and a second wavelength is emitted for capturing another image or images for a second portion of algorithmic processing for biometric authentication. The camera may sequentially capture two or more images at different wavelengths. As an alternative, the camera may be configured to concurrently capture two or more images at different wavelengths.

Abstract: Various implementations disclosed herein include devices, systems, and methods that assess calibration between eye tracking and other components of a head-mounted device (HMD) using another device, such as a mobile device. For example, an example process may include obtaining first sensor data captured by a first sensor of a first device, the first sensor data including a representation of a portion of a second device, obtaining second sensor data captured by a second sensor of the second device, detecting a position of the first sensor of the first device based on the second sensor data, and assessing a calibration between the portion of the second device and the sensor of the second device based on the first sensor data and the detected position of the first sensor.

Abstract: A head-mounted device may have a head-mounted support structure. Displays may present images to eye boxes at the rear of the head-mounted support structure. Cameras and other sensors may be supported by the head-mounted support structure. Tracking cameras may be used to track the movement of a user's hands or other external objects. In dim ambient lighting conditions, a supplemental illumination system may be activated to provide supplemental illumination for the tracking cameras. A single beam of supplemental illumination may be emitted over a given coverage area or a smaller beam of supplemental illumination may be steered across the given coverage area. Light-emitting devices such as infrared light-emitting diodes and infrared lasers may be used to form infrared light sources for the supplemental illumination system.

Abstract: Methods and apparatus for stray light mitigation in optical systems that include two or more illumination sources (e.g., point light sources such as light-emitting diodes (LEDs)) that illuminate an object to be imaged, and a camera configured to capture images of light from the point light sources reflected by the object when illuminated. To mitigate “occlusions” or artifacts caused by, for example, stray light or reflections of the illumination sources off of other components of the system, multiple images of the object are captured with different groups of the illumination sources enabled. The captured images can then be processed and merged to generate an output image with the occlusions or artifacts caused by stray light or reflections mitigated or eliminated.

Abstract: Operating an outward-facing display device for low light conditions may include receiving information indicative of ambient lighting conditions for the display device's environment; in accordance with a determination that the ambient lighting conditions do not satisfy a brightness criterion, determining a set of pixels in the outward-facing display device corresponding to an area of interest in the environment; and driving the set of pixels at a particular brightness to improve lighting in the area of interest.

Abstract: Low-power eye tracking for detecting position and movements of a user's eyes in a head-mounted device (HMD). An eye tracking system for an HMD may include eye tracking cameras and eye odometers that may reduce power consumption of the system. The eye odometers may be used as a low-power component to track relative movement of the user's eyes with respect to the HMD between frames captured by the eye tracking cameras, which may allow the frame rate of the eye tracking cameras to be reduced.