Abstract: Methods and apparatus for glint-assisted gaze tracking in a VR/AR head-mounted display (HMD). Images of a user's eyes captured by gaze tracking cameras may be analyzed to detect glints (reflections on the cornea of light sources that illuminate the user's eyes) and the pupil. The glints are matched to particular ones of the light sources. The glint-light source matches are used to determine the cornea center of the eye, and the pupil center is determined. The optical axis of the eye is reconstructed from the cornea center and the pupil center, and the visual axis is then reconstructed from the optical axis and a 3D model of the user's eye. The point of gaze on the display is then determined based on the visual axis and a 3D model of the HMD.

Abstract: Methods and apparatus for glint-assisted gaze tracking in a VR/AR head-mounted display (HMD). Images of a user's eyes captured by gaze tracking cameras may be analyzed to detect glints (reflections on the cornea of light sources that illuminate the user's eyes) and the pupil. The glints are matched to particular ones of the light sources. The glint-light source matches are used to determine the cornea center of the eye, and the pupil center is determined. The optical axis of the eye is reconstructed from the cornea center and the pupil center, and the visual axis is then reconstructed from the optical axis and a 3D model of the user's eye. The point of gaze on the display is then determined based on the visual axis and a 3D model of the HMD.

Abstract: Various implementations disclosed herein include devices, systems, and methods that use event camera data to track deformable objects such as faces, hands, and other body parts. One exemplary implementation involves receiving a stream of pixel events output by an event camera. The device tracks the deformable object using this data. Various implementations do so by generating a dynamic representation of the object and modifying the dynamic representation of the object in response to obtaining additional pixel events output by the event camera. In some implementations, generating the dynamic representation of the object involves identifying features disposed on the deformable surface of the object using the stream of pixel events. The features are determined by identifying patterns of pixel events. As new event stream data is received, the patterns of pixel events are recognized in the new data and used to modify the dynamic representation of the object.

Abstract: Methods and apparatus for glint-assisted gaze tracking in a VR/AR head-mounted display (HMD). Images of a user's eyes captured by gaze tracking cameras may be analyzed to detect glints (reflections on the cornea of light sources that illuminate the user's eyes) and the pupil. The glints are matched to particular ones of the light sources. The glint-light source matches are used to determine the cornea center of the eye, and the pupil center is determined. The optical axis of the eye is reconstructed from the cornea center and the pupil center, and the visual axis is then reconstructed from the optical axis and a 3D model of the user's eye. The point of gaze on the display is then determined based on the visual axis and a 3D model of the HMD.

Abstract: Various implementations disclosed herein include devices, systems, and methods that use event camera data to track deformable objects such as faces, hands, and other body parts. One exemplary implementation involves receiving a stream of pixel events output by an event camera. The device tracks the deformable object using this data. Various implementations do so by generating a dynamic representation of the object and modifying the dynamic representation of the object in response to obtaining additional pixel events output by the event camera. In some implementations, generating the dynamic representation of the object involves identifying features disposed on the deformable surface of the object using the stream of pixel events. The features are determined by identifying patterns of pixel events. As new event stream data is received, the patterns of pixel events are recognized in the new data and used to modify the dynamic representation of the object.

Abstract: Methods and apparatus for glint-assisted gaze tracking in a VR/AR head-mounted display (HMD). Images of a user's eyes captured by gaze tracking cameras may be analyzed to detect glints (reflections on the cornea of light sources that illuminate the user's eyes) and the pupil. The glints are matched to particular ones of the light sources. The glint-light source matches are used to determine the cornea center of the eye, and the pupil center is determined. The optical axis of the eye is reconstructed from the cornea center and the pupil center, and the visual axis is then reconstructed from the optical axis and a 3D model of the user's eye. The point of gaze on the display is then determined based on the visual axis and a 3D model of the HMD.

Abstract: Disclosed embodiments pertain to apparatus, systems, and methods for robust feature based tracking. In some embodiments, a score may be computed for a camera captured current image comprising a target object. The score may be based on one or more metrics determined from a comparison of features in the current image and a prior image captured by the camera. The comparison may be based on an estimated camera pose for the current image. In some embodiments, one of a point based, an edge based, or a combined point and edge based feature correspondence method may be selected based on a comparison of the score with a point threshold and/or a line threshold, the point and line thresholds being obtained from a model of the target. The camera pose may be refined by establishing feature correspondences using the selected method between the current image and a model image.

Abstract: A reference in an unknown environment is generated on the fly for positioning and tracking. The reference is produced in a top down process by capturing an image of a planar object with a predefined geometric shape, detecting edge pixels of the planar object, then detecting a plurality of line segments from the edge pixels. The plurality of line segments may then be used to detect the planar object in the image based on the predefined geometric shape. An initial pose of the camera with respect to the planar object is determined and tracked using the edges of the planar object.

Abstract: A method for estimating camera pose includes: obtaining an image of a location captured via a camera, where the image includes a target object and edge line features outside of the target object; and calculating a pose of the camera with respect to the target object based on the edge line features.

Abstract: In several aspects, an electronic device and method index a repository of N documents by W words, by not storing between queries, N*W numbers that are specific to each word i and each document j, normally used to compute a score of relevance to a query, of each document j. Instead, the electronic device and method generate the N*W word-specific -document-specific numbers dynamically at query time, based on a set of W numbers corresponding to the W words, and one or more sets (e.g. x sets) of N numbers corresponding to the N documents. Query-time generation of word-specific-document-specific numbers reduces memory otherwise required, e.g. to store these numbers. Hence, in certain aspects W+xN numbers are maintained between queries, and these numbers are changed incrementally when a new document is added to the set or an existing document is removed. Maintaining W+xN numbers reduces processing otherwise required, to start from scratch.

Abstract: Embodiments disclosed pertain to object tracking based, in part, on occluding contours associated with the tracked object. In some embodiments, a 6 Degrees of Freedom (6-DoF) initial camera pose relative to a tracked object in a first image may be obtained. A 6-DoF updated camera pose relative to the tracked object for a second image subsequent to the first image may then be obtained based, at least, on the initial camera pose and one or more features associated with the tracked object and an occluding contour associated with the tracked object in the second image. The occluding contour associated with the tracked object in the second image may be derived from a closed form function.

Abstract: A method of real-time tracking of an object includes capturing a first and a second image of the object. The object is detected in the first image and movement of the object is tracked between the images. Tracking of the object includes obtaining an initial pose of the camera; projecting an image of a model object onto the second image; determining a gradient profile of the second image from an edge point of the model object along a first direction that is normal to the edge of the model object; computing a radius on the gradient profile; determining a rank order of the peaks of the gradient profile along the radius; comparing the rank order with a predetermined rank order to generate a feature candidate point; and reducing a distance along the first direction between the feature candidate point and the edge point on the edge of the model object.

Abstract: Disclosed embodiments pertain to apparatus, systems, and methods for robust feature based tracking. In some embodiments, a score may be computed for a camera captured current image comprising a target object. The score may be based on one or more metrics determined from a comparison of features in the current image and a prior image captured by the camera. The comparison may be based on an estimated camera pose for the current image. In some embodiments, one of a point based, an edge based, or a combined point and edge based feature correspondence method may be selected based on a comparison of the score with a point threshold and/or a line threshold, the point and line thresholds being obtained from a model of the target. The camera pose may be refined by establishing feature correspondences using the selected method between the current image and a model image.

Abstract: Disclosed embodiments pertain to feature based tracking. In some embodiments, a camera pose may be obtained relative to a tracked object in a first image and a predicted camera pose relative to the tracked object may be determined for a second image subsequent to the first image based, in part, on a motion model of the tracked object. An updated SE(3) camera pose may then be obtained based, in part on the predicted camera pose, by estimating a plane induced homography using an equation of a dominant plane of the tracked object, wherein the plane induced homography is used to align a first lower resolution version of the first image and a first lower resolution version of the second image by minimizing the sum of their squared intensity differences. A feature tracker may be initialized with the updated SE(3) camera pose.

Abstract: A method of real-time tracking of an object includes capturing a first and a second image of the object. The object is detected in the first image and movement of the object is tracked between the images. Tracking of the object includes obtaining an initial pose of the camera; projecting an image of a model object onto the second image; determining a gradient profile of the second image from an edge point of the model object along a first direction that is normal to the edge of the model object; computing a radius on the gradient profile; determining a rank order of the peaks of the gradient profile along the radius; comparing the rank order with a predetermined rank order to generate a feature candidate point; and reducing a distance along the first direction between the feature candidate point and the edge point on the edge of the model object.

Abstract: Embodiments include detection or relocalization of an object in a current image from a reference image, such as using a simple and relatively fast and invariant edge orientation based edge feature extraction, then a weak initial matching combined with a strong contextual filtering framework, and then a pose estimation framework based on edge segments. Embodiments include fast edge-based object detection using instant learning with a sufficiently large coverage area for object re-localization. Embodiments provide a good trade-off between computational efficiency of the extraction and matching processes.

Abstract: In several aspects, an electronic device and method index a repository of N documents by W words, by not storing between queries, N*W numbers that are specific to each word i and each document j, normally used to compute a score of relevance to a query, of each document j. Instead, the electronic device and method generate the N*W word-specific-document-specific numbers dynamically at query time, based on a set of W numbers corresponding to the W words, and one or more sets (e.g. x sets) of N numbers corresponding to the N documents. Query-time generation of word-specific-document-specific numbers reduces memory otherwise required, e.g. to store these numbers. Hence, in certain aspects W+xN numbers are maintained between queries, and these numbers are changed incrementally when a new document is added to the set or an existing document is removed. Maintaining W+xN numbers reduces processing otherwise required, to start from scratch.

Abstract: A method for compensating respiratory motion in coronary fluoroscopic images includes finding a set of transformation parameters of a parametric motion model that maximize an objective function that is a weighted normalized cross correlation function of a reference image acquired at a first time that is warped by the parametric motion model and a first incoming image acquired at a second time subsequent to the first time. The weights are calculated as a ratio of a covariance of the gradients of the reference image and the gradients of the first incoming image with respect to a root of a product of a variance of the gradients of the reference image and the variance of the gradients of the first incoming image. The parametric motion model transforms the reference image to match the first incoming image.

Abstract: A reference in an unknown environment is generated on the fly for positioning and tracking. The reference is produced in a top down process by capturing an image of a planar object with a predefined geometric shape, detecting edge pixels of the planar object, then detecting a plurality of line segments from the edge pixels. The plurality of line segments may then be used to detect the planar object in the image based on the predefined geometric shape. An initial pose of the camera with respect to the planar object is determined and tracked using the edges of the planar object.

Abstract: A method for estimating camera pose includes: obtaining an image of a location captured via a camera, where the image includes a target object and edge line features outside of the target object; and calculating a pose of the camera with respect to the target object based on the edge line features.