Abstract: A MEMS scanning device (“Device”) includes at least (1) laser projector(s) controlled by a laser drive to project a laser beam, (2) MEMS scanning mirror(s) controlled by a MEMS drive to scan the laser beam to generate a raster scan, (3) a display configured to receive the raster scan, (4) a thermometer configured to detect a current temperature, (5) a display observing camera configured to capture an image of a predetermined area of the display, and (6) a computer-readable media that stores temperature model(s), each of which is custom-built using machine learning. The device uses the display observing camera to capture image(s) of predetermined pattern(s), which are then used to extract feature(s). The extracted feature(s) are compared with ideal feature(s) to identify a discrepancy. When the identified discrepancy is greater than a threshold, the temperature model(s) are updated accordingly.

Abstract: Examples are disclosed that relate to calibration data related to a determined alignment of sensors on a wearable display device. One example provides a wearable display device comprising a frame, a first sensor and a second sensor, one or more displays, a logic system, and a storage system. The storage system comprises calibration data related to a determined alignment of the sensors with the frame in a bent configuration and instructions executable by the logic system. The instructions are executable to obtain a first sensor data and a second sensor data respectfully from the first and second sensors, determine a distance from the wearable display device to a feature based at least upon the first and second sensor data using the calibration data, obtain a stereo image to display based upon the distance from the wearable display device to the feature, and output the stereo image via the displays.

Abstract: A MEMS scanning device (“Device”) includes at least (1) laser projector(s) controlled by a laser drive to project a laser beam, (2) MEMS scanning mirror(s) controlled by a MEMS drive to scan the laser beam to generate a raster scan, (3) a display configured to receive the raster scan, (4) a thermometer configured to detect a current temperature, (5) a display observing camera configured to capture an image of a predetermined area of the display, and (6) a computer-readable media that stores temperature model(s), each of which is custom-built using machine learning. The device uses the display observing camera to capture image(s) of predetermined pattern(s), which are then used to extract feature(s). The extracted feature(s) are compared with ideal feature(s) to identify a discrepancy. When the identified discrepancy is greater than a threshold, the temperature model(s) are updated accordingly.

Abstract: A near-eye display device comprises right and left display projectors, expansion optics, and inertial measurement units (IMUs), in addition to a plurality of angle-sensitive pixel (ASP) elements and a computer. The right and left expansion optics are configured to receive respective display images from the right and left display projectors and to release expanded forms of the display images. The right IMU is fixedly coupled to the right display projector, and the left IMU is fixedly coupled to the left display projector. Each ASP element is responsive to an angle of light of one of the respective display images as received into the right or left expansion optic. The computer is configured to receive output from the right IMU, the left IMU and the plurality of ASP elements, and render display data for the right and left display projectors based in part on the output.

Abstract: Examples are disclosed that relate to calibration data related to a determined alignment of sensors on a wearable display device. One example provides a wearable display device comprising a frame, a first sensor and a second sensor, one or more displays, a logic system, and a storage system. The storage system comprises calibration data related to a determined alignment of the sensors with the frame in a bent configuration and instructions executable by the logic system. The instructions are executable to obtain a first sensor data and a second sensor data respectfully from the first and second sensors, determine a distance from the wearable display device to a feature based at least upon the first and second sensor data using the calibration data, obtain a stereo image to display based upon the distance from the wearable display device to the feature, and output the stereo image via the displays.

Abstract: A near-eye display device comprises right and left display projectors, expansion optics, and inertial measurement units (IMUs), in addition to a plurality of angle-sensitive pixel (ASP) elements and a computer. The right and left expansion optics are configured to receive respective display images from the right and left display projectors and to release expanded forms of the display images. The right IMU is fixedly coupled to the right display projector, and the left IMU is fixedly coupled to the left display projector. Each ASP element is responsive to an angle of light of one of the respective display images as received into the right or left expansion optic. The computer is configured to receive output from the right IMU, the left IMU and the plurality of ASP elements, and render display data for the right and left display projectors based in part on the output.

Abstract: A MEMS scanning device (“Device”) includes at least (1) laser projector(s) controlled by a laser drive to project a laser beam, (2) MEMS scanning mirror(s) controlled by a MEMS drive to scan the laser beam to generate a raster scan, (3) a display configured to receive the raster scan, (4) a thermometer configured to detect a current temperature, (5) a display observing camera configured to capture an image of a predetermined area of the display, and (6) a computer-readable media that stores temperature model(s), each of which is custom-built using machine learning. The device uses the display observing camera to capture image(s) of predetermined pattern(s), which are then used to extract feature(s). The extracted feature(s) are compared with ideal feature(s) to identify a discrepancy. When the identified discrepancy is greater than a threshold, the temperature model(s) are updated accordingly.

Abstract: A system for continuous image alignment of separate cameras identifies a reference camera transformation matrix between a base reference camera pose and an updated reference camera pose. The system also identifies a match camera transformation matrix between a base match camera pose and an updated match camera pose and an alignment matrix based on visual correspondences between one or more reference frames captured by the reference camera and one or more match frames captured by the match camera. The system also generates a motion model configured to facilitate mapping of a set of pixels of a reference frame captured by the reference camera to a corresponding set of pixels of a match frame captured by the match camera based on the reference camera transformation matrix, the match camera transformation matrix, and the alignment matrix.

Abstract: A MEMS scanning device (“Device”) includes at least (1) laser projector(s) controlled by a laser drive to project a laser beam, (2) MEMS scanning mirror(s) controlled by a MEMS drive to scan the laser beam to generate a raster scan, (3) a display configured to receive the raster scan, (4) a thermometer configured to detect a current temperature, (5) a display observing camera configured to capture an image of a predetermined area of the display, and (6) a computer-readable media that stores temperature model(s), each of which is custom-built using machine learning. The device uses the display observing camera to capture image(s) of predetermined pattern(s), which are then used to extract feature(s). The extracted feature(s) are compared with ideal feature(s) to identify a discrepancy. When the identified discrepancy is greater than a threshold, the temperature model(s) are updated accordingly.

Abstract: One disclosed example provides a head-mounted device configured to control a plurality of light sources of a handheld object and acquire image data comprising a sequence of environmental tracking exposures in which the plurality of light sources are controlled to have a lower integrated intensity and handheld object tracking exposures in which the plurality of light sources are controlled to have a higher integrated intensity. The instructions are further executable to detect, via an environmental tracking exposure, one or more features of the surrounding environment, determine a pose of the head-mounted device based upon the one or more features of the surrounding environment detected, detect via a handheld object tracking exposure the plurality of light sources of the handheld object, determine a pose of the handheld object relative to the head-mounted device based upon the plurality of light sources detected, and output the pose of the handheld object.

Abstract: A system for continuous image alignment of separate cameras identifies a reference camera transformation matrix between a base reference camera pose and an updated reference camera pose. The system also identifies a match camera transformation matrix between a base match camera pose and an updated match camera pose and an alignment matrix based on visual correspondences between one or more reference frames captured by the reference camera and one or more match frames captured by the match camera. The system also generates a motion model configured to facilitate mapping of a set of pixels of a reference frame captured by the reference camera to a corresponding set of pixels of a match frame captured by the match camera based on the reference camera transformation matrix, the match camera transformation matrix, and the alignment matrix.

Abstract: A MEMS scanning device (“Device”) includes at least (1) laser projector(s) controlled by a laser drive to project a laser beam, (2) MEMS scanning mirror(s) controlled by a MEMS drive to scan the laser beam to generate a raster scan, (3) a display configured to receive the raster scan, (4) a thermometer configured to detect a current temperature, (5) a display observing camera configured to capture an image of a predetermined area of the display, and (6) a computer-readable media that stores temperature model(s), each of which is custom-built using machine learning. The device uses the display observing camera to capture image(s) of predetermined pattern(s), which are then used to extract feature(s). The extracted feature(s) are compared with ideal feature(s) to identify a discrepancy. When the identified discrepancy is greater than a threshold, the temperature model(s) are updated accordingly.

Abstract: One disclosed example provides a head-mounted device configured to control a plurality of light sources of a handheld object and acquire image data comprising a sequence of environmental tracking exposures in which the plurality of light sources are controlled to have a lower integrated intensity and handheld object tracking exposures in which the plurality of light sources are controlled to have a higher integrated intensity. The instructions are further executable to detect, via an environmental tracking exposure, one or more features of the surrounding environment, determine a pose of the head-mounted device based upon the one or more features of the surrounding environment detected, detect via a handheld object tracking exposure the plurality of light sources of the handheld object, determine a pose of the handheld object relative to the head-mounted device based upon the plurality of light sources detected, and output the pose of the handheld object.

Abstract: One disclosed example provides a head-mounted device configured to control a plurality of light sources of a handheld object and acquire image data comprising a sequence of environmental tracking exposures in which the plurality of light sources are controlled to have a lower integrated intensity and handheld object tracking exposures in which the plurality of light sources are controlled to have a higher integrated intensity. The instructions are further executable to detect, via an environmental tracking exposure, one or more features of the surrounding environment, determine a pose of the head-mounted device based upon the one or more features of the surrounding environment detected, detect via a handheld object tracking exposure the plurality of light sources of the handheld object, determine a pose of the handheld object relative to the head-mounted device based upon the plurality of light sources detected, and output the pose of the handheld object.

Abstract: One disclosed example provides a computing device configured to receive from an image sensor of a head-mounted device environmental tracking exposures and handheld object tracking exposures, determine a pose of the handheld object with respect to the head-mounted device based upon the handheld object tracking exposures, determine a pose of the head-mounted device with respect to a surrounding environment based upon the environmental tracking exposures, derive a pose of the handheld object relative to the surrounding environment based upon the pose of the handheld object with respect to the head-mounted device and the pose of the head-mounted device with respect to the surrounding environment, and output the pose of the handheld object relative to the surrounding environment for controlling a user interface displayed on the head-mounted device.

Abstract: One disclosed example provides a method for determining a pose of a handheld object in a surrounding environment. Optical pose data is stored in an image queue of a first filter. IMU data is received from an IMU of the handheld object and stored in an IMU queue of the first filter. Using at least a portion of the optical pose data and the IMU data, an initial pose of the handheld object is determined and outputted. The method determines that either the image queue or the IMU queue is empty. A second filter comprising the one empty queue and the other non-empty queue is instantiated as a copy of the first filter. Using the data from the non-empty queue in the second filter, the initial pose of the handheld object is updated to an updated pose, and the updated pose is outputted.

Abstract: In embodiments of a camera-based input device, the input device includes an inertial measurement unit that collects motion data associated with velocity and acceleration of the input device in an environment, such as in three-dimensional (3D) space. The input device also includes at least two visual light cameras that capture images of the environment. A positioning application is implemented to receive the motion data from the inertial measurement unit, and receive the images of the environment from the at least two visual light cameras. The positioning application can then determine positions of the input device based on the motion data and the images correlated with a map of the environment, and track a motion of the input device in the environment based on the determined positions of the input device.

Abstract: One disclosed example provides a head-mounted device including a stereo camera arrangement, a logic device configured to execute instructions, and a storage device storing instructions executable by the logic device to, for each camera in the stereo camera arrangement, receive image data of a field of view of the camera, detect light sources of a handheld object in the image data, and based upon the light sources detected, determine a pose of the handheld object. The instructions are executable to, based upon the pose of the handheld object determined for each camera in the stereo camera arrangement, calibrate the stereo camera arrangement.

Abstract: One disclosed example provides a method for determining a pose of a handheld object in a surrounding environment. Optical pose data is stored in an image queue of a first filter. IMU data is received from an IMU of the handheld object and stored in an IMU queue of the first filter. Using at least a portion of the optical pose data and the IMU data, an initial pose of the handheld object is determined and outputted. The method determines that either the image queue or the IMU queue is empty. A second filter comprising the one empty queue and the other non-empty queue is instantiated as a copy of the first filter. Using the data from the non-empty queue in the second filter, the initial pose of the handheld object is updated to an updated pose, and the updated pose is outputted.

Abstract: A display system includes a display alignment tracker configured track the position of a first signal and the position of a second signal. The display alignment tracker optically multiplexes a portion of a first signal and a portion of the second signal into a combined optical signal and measures a differential between the first signal and the second signal.