Abstract: Provided is a process that includes obtaining data indicative of state of a dynamic mechanical system and an environment of the dynamic mechanical system, the data comprising a plurality of channels of data from a plurality of different sensors including a plurality of cameras and other sensors indicative of state of actuators of the dynamic mechanical system; forming a training set from the obtained data by segmenting the data by time and grouping segments from the different channels by time to form units of training data that span different channels among the plurality of channels; training a metric learning model to encode inputs corresponding to the plurality of channels as vectors in an embedding space with self-supervised learning based on the training set; and using the trained metric learning model to control the dynamic mechanical system or another dynamic mechanical system.

Abstract: In some embodiments, a system comprises a head-mounted frame removably coupleable to the user's head; one or more light sources coupled to the head-mounted frame and configured to emit light with at least two different wavelengths toward a target object in an irradiation field of view of the light sources; one or more electromagnetic radiation detectors coupled to the head-mounted member and configured to receive light reflected after encountering the target object; and a controller operatively coupled to the one or more light sources and detectors and configured to determine and display an output indicating the identity or property of the target object as determined by the light properties measured by the detectors in relation to the light properties emitted by the light sources.

Abstract: An apparatus is disclosed for capturing image information. The apparatus includes a waveguide having opposed planar input and output faces. A diffractive optical element (DOE) is formed across the waveguide. The DOE is configured to couple a portion of the light passing through the waveguide into the waveguide. The light coupled into the waveguide is directed via total internal reflection to an exit location on the waveguide. The apparatus further includes a light sensor having an input positioned adjacent the exit location of the waveguide to capture light exiting therefrom and generate output signals corresponding thereto. A processor determines the angle and position of the coupled light with respect to the input face of the waveguide based on the output signals.

Abstract: Head mounted display systems configured to facilitate the exchange of biometric information between the head mounted display system and another computing device are disclosed. The head mounted display system can comprise a virtual or augmented reality device. After displaying a consent request regarding biometric information with the head mounted display system, a response to the consent request that includes a consent indication regarding an aspect of the biometric information can be determined. After obtaining biometric information from a wearer utilizing e.g., a camera of the head mounted display, and processing the biometric information, a biometric information processing result can be generated. The result can be communicated from the head mounted display system to another computing device.

Abstract: Systems and methods for generating a face model for a user of a head-mounted device are disclosed. The head-mounted device can include one or more eye cameras configured to image the face of the user while the user is putting the device on or taking the device off. The images obtained by the eye cameras may be analyzed using a stereoscopic vision technique, a monocular vision technique, or a combination, to generate a face model for the user.

Abstract: An augmented reality (AR) device can be configured to monitor ambient audio data. The AR device can detect speech in the ambient audio data, convert the detected speech into text, or detect keywords such as rare words in the speech. When a rare word is detected, the AR device can retrieve auxiliary information (e.g., a definition) related to the rare word from a public or private source. The AR device can display the auxiliary information for a user to help the user better understand the speech. The AR device may perform translation of foreign speech, may display text (or the translation) of a speaker's speech to the user, or display statistical or other information associated with the speech.

Abstract: Disclosed herein are examples of a wearable display system capable of determining a user interface (UI) event with respect to a virtual UI device (e.g., a button) and a pointer (e.g., a finger or a stylus) using a neural network. The wearable display system can render a representation of the UI device onto an image of the pointer captured when the virtual UI device is shown to the user and the user uses the pointer to interact with the virtual UI device. The representation of the UI device can include concentric shapes (or shapes with similar or the same centers of gravity) of high contrast. The neural network can be trained using training images with representations of virtual UI devices and pointers.

Abstract: An augmented reality device (ARD) can present virtual content which can provide enhanced experiences with the user's physical environment. For example, the ARD can detect a linkage between a person in the FOV of the ARD and a physical object (e.g., a document presented by the person) or detect linkages between the documents. The linkages may be used in identity verification or document verification.

Abstract: A display system comprises a wearable display device for displaying augmented reality content. The display device comprises a display area comprising light redirecting features that are configured to direct light to a user. The display area is at least partially transparent and is configured to provide a view of an ambient environment through the display area. The display device is configured to determine that a reflection of the user is within the user's field of view through the display area. After making this determination, augmented reality content is displayed in the display area with the augmented reality content augmenting the user's view of the reflection. In some embodiments, the augmented reality content may overlie on the user's view of the reflection, thereby allowing all or portions of the reflection to appear to be modified to provide a realistic view of the user with various modifications made to their appearance.

Abstract: A wearable display system can be capable of determining a user interface (UI) event with respect to a virtual UI device (e.g., a button) and a pointer (e.g., a finger or a stylus) using a neural network. The wearable display system can render a representation of the UI device onto an image of the pointer captured when the virtual UI device is shown to the user and the user uses the pointer to interact with the virtual UI device. The representation of the UI device can include concentric shapes (or shapes with similar or the same centers of gravity) of high contrast. The neural network can be trained using training images with representations of virtual UI devices and pointers.

Abstract: Systems and methods for synthesizing an image of the face by a head-mounted device (HMD) are disclosed. The HMD may not be able to observe a portion of the face. The systems and methods described herein can generate a mapping from a conformation of the portion of the face that is not imaged to a conformation of the portion of the face observed. The HMD can receive an image of a portion of the face and use the mapping to determine a conformation of the portion of the face that is not observed. The HMD can combine the observed and unobserved portions to synthesize a full face image.

Abstract: A software compensated robotic system makes use of recurrent neural networks and image processing to control operation and/or movement of an end effector. Images are used to compensate for variations in the response of the robotic system to command signals. This compensation allows for the use of components having lower reproducibility, precision and/or accuracy that would otherwise be practical.

Abstract: Systems and methods pertaining to iris recognition are disclosed. A system is disclosed comprising an iris imager configured to acquire an iris image, and a processor configured to receive a first plurality of iris images from the iris imager, generate a first plurality of iris codes corresponding to the first plurality of iris images, generate a distribution metric corresponding to an iris cell location, generate a first composite iris code using the distribution metric, and generate a first match value using the first composite iris code and a first stored iris code.

Abstract: Methods and devices for estimating position of a device within a 3D environment are described. Embodiments of the methods include sequentially receiving multiple image segments forming an image representing a field of view (FOV) comprising a portion of the environment. The image includes multiple sparse points that are identifiable based in part on a corresponding subset of image segments of the multiple image segments. The method also includes sequentially identifying one or more sparse points of the multiple sparse points when each subset of image segments corresponding to the one or more sparse points is received and estimating a position of the device in the environment based on the identified the one or more sparse points.

Abstract: Systems and methods for blue light adjustment with a wearable display system are provided. Embodiments of the systems and methods for blue light adjustment can include receiving an eye image of an eye exposed to an adjusted level of blue light; detecting a change in a pupillary response by comparison of the received eye image to a first image; determining that the pupillary response corresponds to a biometric characteristic of a human individual; and allowing access to a biometric application based on the pupillary response determination.

Abstract: A wearable display system can automatically recognize a physical remote or a device that the remote serves using computer vision techniques. The wearable system can generate a virtual remote with a virtual control panel viewable and interactable by user of the wearable system. The virtual remote can emulate the functionality of the physical remote. The user can select a virtual remote for interaction, for example, by looking or pointing at the parent device or its remote control, or by selecting from a menu of known devices. The virtual remote may include a virtual button, which is associated with a volume in the physical space. The wearable system can detect that a virtual button is actuated by determining whether a portion of the user's body (e.g., the user's finger) has penetrated the volume associated with the virtual button.

Abstract: A texture projecting light bulb includes an extended light source located within an integrator. The integrator includes at least one aperture configured to allow light to travel out of the interior of the integrator. In various embodiments, the interior of the integrator may be a diffusely reflective surface and the integrator may be configured to produce a uniform light distribution at the aperture to approximate a point source. The integrator may be surrounded by a light bulb enclosure. In various embodiments, the light bulb enclosure may include transparent and opaque regions configured to project a structured pattern of visible and/or infrared light.

Abstract: This disclosure is directed to an image sensor. The image sensor includes a two-dimensional pixel array divided into a plurality of blocks, each of the plurality of blocks comprising pixels arranged in at least two different rows and two different columns, and a shutter mechanism that exposes the plurality of blocks sequentially, with all pixels in each block being exposed synchronously.

Abstract: Systems and methods for eye image set selection, eye image collection, and eye image combination are described. Embodiments of the systems and methods for eye image set selection can include comparing a determined image quality metric with an image quality threshold to identify an eye image passing an image quality threshold, and selecting, from a plurality of eye images, a set of eye images that passes the image quality threshold.

Abstract: An augmented reality device (ARD) can present virtual content which can provide enhanced experiences with the user's physical environment. For example, the ARD can detect a linkage between a person in the FOV of the ARD and a physical object (e.g., a document presented by the person) or detect linkages between the documents. The linkages may be used in identity verification or document verification.