Abstract: A Near-Eye-Display (NED) devices that translates combinations of user gaze direction and predetermined facial gestures into user input instructions. The NED device includes an eye tracking system and a display that renders computer-generated images within a user's field-of-view. The eye tracking system may continually track the user's eye movements with a high degree of accuracy to identify specific computer-generated images that a user is focused on. The eye tracking system may also identify various facial gestures such as, for example, left-eye blinks and/or right-eye blinks that are performed while the specific computer-generated images are being focused on. In this way, NED devices are enabled to identify combinations of user gaze direction and predetermined facial gestures and to translate these identified combinations into user input instructions that correspond to specific computer-generated images.

Abstract: A camera system is configured to automatically monitor an area. Depth image(s) of the area are acquired based on active IR light emitted by the camera system and reflected from the area to a sensor array of the camera system. The depth image(s) are computer analyzed to identify a human subject. For each spectral illuminator of the camera system, spectral light image(s) of the area are acquired based on active spectral light in the spectral light sub-band of the spectral illuminator reflected from the area to the sensor array. The spectral light image(s) for the spectral illuminators are computer analyzed to identify an interaction between the human subject and an object in the area. In response to identifying the interaction between the human subject and the object in the area, an action to be performed for the object in the area is computer issued.

Abstract: Examples are disclosed herein relating to signal processing in a time-of-flight (ToF) system. One example provides, a method comprising emitting, via a light source, amplitude-modulated light toward an object, acquiring, via an image sensor comprising a plurality of pixels, a plurality of image frames capturing light emitted from the light source that is reflected by the object, wherein the plurality of image frames are acquired at two or more different frequencies of the amplitude-modulated light and collectively form a multifrequency frame, and for each pixel of the multifrequency frame, determining a brightness level, applying an adaptive denoising process by setting a kernel size based on the brightness level, and performing a phase unwrapping process to determine a depth value for the pixel.

Abstract: Examples are disclosed that relate to methods and systems for determining a depth of a pixel of a depth image. In one example, a method comprises illuminating a scene with amplitude-modulated light of a plurality K of different frequencies. A sensor detects the amplitude-modulated light as reflected from the scene. A plurality K of complex signals k is formed by determining, for each frequency k of the plurality of different frequencies K, a complex signal k comprising a real portion k and an imaginary portion k. A comparison function is used to compare the plurality K of complex signals to a corresponding plurality K of modeled signals, each modeled signal k comprising a modeled real portion k and a modeled imaginary portion k. The method further comprises determining the depth of the pixel based at least on the comparison function and outputting the depth of the pixel.

Abstract: A time-of-flight (ToF) system disclosed herein provides a method of a method of separating a direct component of light collected by a time of flight (ToF) detector from a global component of light collected by the ToF detector, the method comprising acquiring three or more images represented by three or more matrices in response to illuminating a target with a light source using a first spatial pattern at three or more different modulation frequencies, acquiring an additional image represented by an additional matrix in response to illuminating the target with the light source using a second spatial pattern, the second spatial pattern being different than the first spatial pattern, and determining one or more parameters of the direct component of light and the global component of light based on analysis of the three or more matrices and the additional matrix.

Abstract: Techniques for implementing eye tracking using various real-time computational solutions to a three-dimensional eye tracking framework. An exemplary eye tracking system for a NED device includes sensors that are directed toward and angularly offset from a user's eyes in a manner that causes circular features (e.g., irises and/or pupils) of the user's eyes to appear elliptical within sensor planes of the individual sensors. An iris and/or pupil of an eye will appear circular when the eye is looked at straight on (i.e., perpendicular to an optical axis of the eye's lens) but elliptical when observed from an angular offset. The eye tracking systems and methods disclosed herein exploit these principles to track movements of the user's eyes with a higher degree of accuracy than conventional eye tracking systems.

Abstract: A camera system includes one or more spectral illuminators, a tunable optical filter, and a sensor array. Active spectral light emitted from the one or more spectral illuminators towards a scene is dynamically tuned to an illumination sub-band selected from a plurality of different illumination sub-bands. Sequentially for each of a plurality of fluorescing light sub-bands different than the selected illumination sub-band, the tunable optical filter is adjusted to block light from being transmitted from the scene to the sensor array in all but a tested fluorescing light sub-band from the plurality of different fluorescing light sub-bands, and the sensor array is addressed to acquire one or more image of the scene in the tested fluorescing light sub-band.

Abstract: A time-of-flight (ToF) camera is configured to illuminate a subject with IR light. For each sensor of a sensor array, camera space coordinates of a locus of the subject are determined based on the measured IR light. The camera space coordinates include a phase difference between emitted IR light and reflected IR light. A plurality of candidate positions of the locus of the subject are determined in world space based on the camera space coordinates. Each candidate position corresponds to a different phase wrapping of the phase difference. A phase wrapping is determined based on performing a searching algorithm on the different candidate positions. A depth value for the sensor is calculated based on the phase difference and the phase wrapping determined from performing the searching algorithm. A matrix of pixels is outputted from the camera. Each pixel of the matrix includes a depth value.

Abstract: A camera includes one or more spectral illuminators, a tunable optical filter optically intermediate the one or more spectral illuminators and a scene, and a sensor array. The one or more spectral illuminators are configured to emit active spectral light. The tunable optical filter is dynamically adjustable to change a selected sub-band of the active spectral light that illuminates the scene. The sensor array includes a plurality of sensors each configured to measure spectral light reflected from the scene in the selected sub-band.

Abstract: A camera includes one or more spectral illuminators, a tunable optical filter optically intermediate the one or more spectral illuminators and a scene, and a sensor array. The one or more spectral illuminators are configured to emit active spectral light. The tunable optical filter is dynamically adjustable to change a selected sub-band of the active spectral light that illuminates the scene. The sensor array includes a plurality of sensors each configured to measure spectral light reflected from the scene in the selected sub-band.

Abstract: A camera includes a time-of-flight illuminator configured to emit active IR light and a plurality of spectral illuminators, each spectral illuminator configured to emit active spectral light in a different spectral light sub-band. The camera further includes a sensor array that includes a plurality of differential sensors. Each differential sensor is configured to differentially measure both 1) the active IR light, and 2) the active spectral light in each of the different spectral light sub-bands. The camera further includes an output machine operatively connected to the sensor array. The output machine is configured to output a matrix of pixels based on time-multiplexed measurements of the sensor array. Each pixel of the matrix includes 1) a depth value, and 2) a plurality of spectral values. Each of the plurality of spectral values corresponds to a spectral light sub-band of one of the plurality of spectral illuminators.

Abstract: A Near-Eye-Display (NED) devices that translates combinations of user gaze direction and predetermined facial gestures into user input instructions. The NED device includes an eye tracking system and a display that renders computer-generated images within a user's field-of-view. The eye tracking system may continually track the user's eye movements with a high degree of accuracy to identify specific computer-generated images that a user is focused on. The eye tracking system may also identify various facial gestures such as, for example, left-eye blinks and/or right-eye blinks that are performed while the specific computer-generated images are being focused on. In this way, NED devices are enabled to identify combinations of user gaze direction and predetermined facial gestures and to translate these identified combinations into user input instructions that correspond to specific computer-generated images.

Abstract: The technology described herein generically calibrates the eye tracking apparatus in a head-mounted display (HMD) device making user-specific calibration unnecessary. The eye tracking apparatus uses one or more cameras to view the eyes of the user wearing the HMD device. From the images captured of the user's eyes, the generically calibrated HMD is able to determine a gaze direction for the user without the user needing to perform a user-specific calibration. The present technology bypasses the need for understanding user specific eye characteristics, such the shape of pupil and iris (ellipticity), by training a machine learning model on a large number of diverse eye images taken in diverse conditions. The machine learning model may take on the form of a convolutional neural network or some other regression based machine learning technology.

Abstract: Features of the present disclosure implement an analytical formalism to calculate a time of flight (ToF) for a signal in order to identify the distance between the ToF imaging system and a target object. The features of the present disclosure allow a flexible, fast, and accurate solution for phase unwrapping in ToF imaging system. Such techniques minimize the memory usage, accounts for the factors that generally contribute to differences in the performance of the selected modulation frequencies, and mitigates the unwrapping errors that occur in low signal to noise areas.

Abstract: A method to correct a digital image to reverse the effect of signal diffusion among pixels of the digital image. For a target pixel j of the digital image, a set of signal values and a set of signal amplitudes are received, each corresponding to a set of kernel pixels i surrounding and including the target pixel j. For each kernel pixel i, a weighting coefficient is computed based on the signal amplitude of that kernel pixel i and on the signal amplitude of the target pixel j. A linear combination of signal values corresponding to the set of kernel pixels i is computed, wherein the signal value for each pixel i is weighted by the weighting coefficient corresponding to that pixel i. The linear combination is stored in volatile memory of an electronic device as a corrected signal value for the target pixel j.

Abstract: Technologies for performing user-specific calibration of eye tracking systems for Near-Eye-Display (NED) devices. The NED device may sequentially present different virtual stimuli to a user while concurrently capturing instances of eye tracking data. The eye tracking data reveals calibration ellipse centers that uniquely correspond to individual virtual stimuli. The calibration ellipse centers may be used define a polygon grid in association with a sensor plane. The resulting polygon grid is used during operation to interpolate the real-time gaze direction of the user. For example, a real-time instance of eye tracking data may be analyzed to determine which particular polygon of the polygon grid a real-time ellipse center falls within. Then, distances between the real-time ellipse center and the vertices of the particular polygon may be determined. A proportionality factor is then determined based on these distances and is used to interpolate the real-time eye gaze of the user.

Abstract: A Near-Eye-Display (NED) device utilizes eye tracking to interpret gaze direction as user input for hands free positioning of virtual items. The NED device includes an eye tracking system to monitor gaze direction and a display component that renders virtual items within a user's view of a physical real-world environment. The NED device receives an indication that the user desires to adjust a position of the virtual item. In response, the NED device uses tracks the user's eye movements to dynamically change the position at which the display component is rendering the virtual item. In this way, a NED device may identify when a user desires to adjust a position at which a virtual item is being rendered and then may enable the user to make the desired adjustments simply by “dragging” the virtual item with deliberate and controlled eye movements.

Abstract: A Near-Eye-Display (NED) devices that translates combinations of user gaze direction and predetermined facial gestures into user input instructions. The NED device includes an eye tracking system and a display that renders computer-generated images within a user's field-of-view. The eye tracking system may continually track the user's eye movements with a high degree of accuracy to identify specific computer-generated images that a user is focused on. The eye tracking system may also identify various facial gestures such as, for example, left-eye blinks and/or right-eye blinks that are performed while the specific computer-generated images are being focused on. In this way, NED devices are enabled to identify combinations of user gaze direction and predetermined facial gestures and to translate these identified combinations into user input instructions that correspond to specific computer-generated images.

Abstract: Systems and methods are provided for providing an improved optical tracking signal. A system receives an input signal which is based on a reflected optical signal comprising light reflected off of two or more real-world objects. The input signal includes a first region representing light reflected off of a first real-world object, a second region representing light reflected off of a second real-world object, and a transition region between the first region and the second region. The input signal includes jitter in at least one of the first region, the second region, or the transition region. The system also performs a sharpening operation to sharpen the transition region for an output signal created from the input signal. The sharpening operation includes a complex operation utilizing both real and imaginary components of the input signal. The system also outputs the output signal as an improved version of the input signal.

Abstract: Systems and methods are provided for providing an improved optical tracking signal. A system receives an input signal which is based on a reflected optical signal comprising light reflected off of two or more real-world objects. The input signal includes a first region representing light reflected off of a first real-world object, a second region representing light reflected off of a second real-world object, and a transition region between the first region and the second region. The input signal includes jitter in at least one of the first region, the second region, or the transition region. The system also performs a sharpening operation to sharpen the transition region for an output signal created from the input signal. The sharpening operation includes a complex operation utilizing both real and imaginary components of the input signal. The system also outputs the output signal as an improved version of the input signal.