Abstract: A method for displaying a three dimensional (“3D”) image includes rendering a frame of 3D image data. The method also includes analyzing the frame of 3D image data to generate best known depth data. The method further includes using the best known depth data to segment the 3D image data into near and far frames of two dimensional (“2D”) image data corresponding to near and far depths respectively. Moreover, the method includes displaying near and far 2D image frames corresponding to the near and far frames of 2D image data at near and far depths to a user respectively.

Abstract: A computer implemented method of facilitating communication between first and second users includes displaying, by a first head-worn device, a first virtual object to the first user first user wearing the first head-worn device. The method also includes displaying, by a second head-worn device, a second virtual object to the second user wearing the second head-worn device. The method further includes facilitating, by the first and second head-worn devices, communications between the first and second users using the first and second virtual objects to simulate the first and second users being present in a common environment.

Abstract: An apparatus includes a light-emitting diode (LED) driver circuit, one or more LEDs of an LED array, and an electronic switching circuit. The LED driver circuit is configured to generate an electric current. The one or more LEDs are electrically connected to the LED driver circuit. The electronic switching circuit is electrically connected to the one or more LEDs and configured to be placed in one of multiple switching configurations. The electronic switching circuit is further configured to direct a portion of the electric current away from the one or more LEDs, such that a remaining portion of the electric current drives the one or more LEDs. The portion of the electric current corresponds to the one of the multiple switching configurations.

Abstract: A method of performing localization of a handheld device with respect to a wearable device includes capturing, by a first imaging device mounted to the handheld device, a fiducial image containing a number of fiducials affixed to the wearable device and capturing, by a second imaging device mounted to the handheld device, a world image containing one or more features surrounding the handheld device. The method also includes obtaining, by a sensor mounted to the handheld device, handheld data indicative of movement of the handheld device, determining the number of fiducials contained in the fiducial image, and updating a position and an orientation of the handheld device using at least one of the fiducial image or the world image and the handheld data.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for object recognition neural network for amodal center prediction. One of the methods includes receiving an image of an object captured by a camera. The image of the object is processed using an object recognition neural network that is configured to generate an object recognition output. The object recognition output includes data defining a predicted two-dimensional amodal center of the object, wherein the predicted two-dimensional amodal center of the object is a projection of a predicted three-dimensional center of the object under a camera pose of the camera that captured the image.

Abstract: An anti-reflective waveguide assembly comprising a waveguide substrate having a first index of refraction, a plurality of diffractive optical elements disposed upon a first surface of the waveguide and an anti-reflective coating disposed upon a second surface of the waveguide. The anti-reflective coating preferably increases absorption of light through a surface to which it is applied into the waveguide so that at least 97 percent of the light is transmitted. The anti-reflective coating is composed of four layers of material having different indices of refraction that the first index of refraction and an imaginary refractive index less than 1×10?3 but preferably less than 5×10?4.

Abstract: Disclosed herein are systems and methods for processing speech signals in mixed reality applications. A method may include receiving an audio signal; determining, via first processors, whether the audio signal comprises a voice onset event; in accordance with a determination that the audio signal comprises the voice onset event: waking a second one or more processors; determining, via the second processors, that the audio signal comprises a predetermined trigger signal; in accordance with a determination that the audio signal comprises the predetermined trigger signal: waking third processors; performing, via the third processors, automatic speech recognition based on the audio signal; and in accordance with a determination that the audio signal does not comprise the predetermined trigger signal: forgoing waking the third processors; and in accordance with a determination that the audio signal does not comprise the voice onset event: forgoing waking the second processors.

Abstract: An image sensor suitable for use in an augmented reality system to provide low latency image analysis with low power consumption. The image sensor may have multiple pixel cells, each of the pixel cells comprising a photodetector to generate an electric signal based on an intensity of light incident upon the photodetector. The pixel cells may include multiple subsets of pixel cells, each subset of pixel cells including at least one angle-of-arrival to-intensity converter to modulate incident light reaching one or more of the pixel cells in the subset based on an angle of arrival of the incident light. Each pixel cell within the plurality of pixel cells may include differential readout circuitry configured to output a readout signal only when an amplitude of a current electric signal from the photodetector is different from an amplitude of a previous electric signal from the photodetector.

Abstract: Waveguides comprising materials with refractive index greater than or equal to 1.8 and methods of patterning waveguides are disclosed. Patterned waveguides comprising materials with refractive index greater than or equal to 1.8 can be incorporated in display devices, such as, for example wearable display devices to project virtual images to a viewer.

Abstract: Augmented reality glasses can include a first laser, a second laser, and a scanning mirror. The augmented reality glasses can also include a first polarization selective reflector and a second polarization selective reflector. The first polarization selective reflector is arranged to receive light from the first laser and reflect light received from the first laser at a first angle to the scanning mirror, and the second polarization selective reflector is arranged to receive light from the second laser and reflect light received from the second laser at a second angle to the scanning mirror. The augmented reality glasses can also have an eyepiece having an input coupling port arranged to receive light reflected by the scanning mirror.

Abstract: Techniques are disclosed for allowing a user's hands to interact with virtual objects. An image of at least one hand may be received from an image capture devices. A plurality of keypoints associated with at least one hand may be detected. In response to determining that a hand is making or is transitioning into making a particular gesture, a subset of the plurality of keypoints may be selected. An interaction point may be registered to a particular location relative to the subset of the plurality of keypoints based on the particular gesture. A proximal point may be registered to a location along the user's body. A ray may be cast from the proximal point through the interaction point. A multi-DOF controller for interacting with the virtual object may be formed based on the ray.

Abstract: Example display devices include a waveguide configured to propagate visible light under total internal reflection in a direction parallel to a major surface of the waveguide. The waveguide has formed thereon an outcoupling element configured to outcouple a portion of the visible light in a direction normal to the major surface of the waveguide. The example display devices additionally include a polarization-selective notch reflector disposed on a first side of the waveguide and configured to reflect visible light having a first polarization while transmitting the portion of the visible light having a second polarization. The example display devices further include a polarization-independent notch reflector disposed on a second side of the waveguide and configured to reflect visible light having the first polarization and the second polarization, where the polarization-independent notch reflector is configured to convert a polarization of visible light reflecting therefrom.

Abstract: Examples of wearable systems and methods can use multiple inputs (e.g., gesture, head pose, eye gaze, voice, totem, and/or environmental factors (e.g., location)) to determine a command that should be executed and objects in the three-dimensional (3D) environment that should be operated on. The wearable system can detect when different inputs converge together, such as when a user seeks to select a virtual object using multiple inputs such as eye gaze, head pose, hand gesture, and totem input. Upon detecting an input convergence, the wearable system can perform a transmodal filtering scheme that leverages the converged inputs to assist in properly interpreting what command the user is providing or what object the user is targeting.

Abstract: A display subsystem for a virtual image generation system for use by an end user comprises a planar waveguide apparatus, an optical fiber, at least one light source configured for emitting light from a distal end of the optical fiber, and a collimation element mounted to a distal end of the optical fiber for collimating light from the optical fiber. The virtual image generation system further comprises a mechanical drive assembly to which the optical fiber is mounted to the drive assembly. The mechanical drive assembly is configured for displacing the distal end of the optical fiber, along with the collimation element, in accordance with a scan pattern. The virtual image generation system further comprises an optical waveguide input apparatus configured for directing the collimated light from the collimation element down the planar waveguide apparatus, such that the planar waveguide apparatus displays image frames to the end user.

Abstract: An eyewear device for being worn on a head of a user for presenting virtual content to a user comprises an optics system and a frame front operatively coupled to the optics system for presenting virtual content to a user wearing the eyewear device. The eyewear device further comprises left and right opposing temple arms affixed to the frame front, and a torsion band assembly having opposing ends that connect the left and right opposing temple arms together. The eyewear device further comprises at least a first floating boss that protrudes partially into one of the left and right opposing temple arms, such that the first floating boss(es) moves within the one of the left and right opposing temple arms in one or more axes in a constrained manner.

Abstract: Disclosed herein are systems and methods for efficiently rendering audio. A method may include receiving a request to present a first audio track, wherein the first audio track is based on a first audio model comprising a shared model component and a first model component; receiving a request to present a second audio track, wherein the second audio track is based on a second audio model comprising the shared model component and a second model component; rendering a sound based on the first audio track, the second audio track, the shared model component, the first model component, and the second model component; and presenting, via one or more speakers, the an audio signal comprising the rendered sound.

Abstract: A headwear device comprises a frame structure configured for being worn on the head of a user, a vibration voice pickup (VVPU) sensor affixed to the frame structure for capturing vibration originating from a voiced sound of a user and generating a vibration signal, at least one microphone affixed to the frame structure for capturing voiced sound from the user and ambient noise, and at least one processor configured for performing an analysis of the vibration signal, and determining that the user has generated the voice sound based on the analysis of the vibration signal.

Abstract: A wearable display system includes one or more emissive micro-displays, e.g., micro-LED displays. The micro-displays may be monochrome micro-displays or full-color micro-displays. The micro-displays may include arrays of light emitters. Light collimators may be utilized to narrow the angular emission profile of light emitted by the light emitters. Where a plurality of emissive micro-displays is utilized, the micro-displays may be positioned at different sides of an optical combiner, e.g., an X-cube prism which receives light rays from different micro-displays and outputs the light rays from the same face of the cube. The optical combiner directs the light to projection optics, which outputs the light to an eyepiece that relays the light to a user's eye. The eyepiece may output the light to the user's eye with different amounts of wavefront divergence, to place virtual content on different depth planes.

Abstract: The present disclosure relates to display systems and, more particularly, to augmented reality display systems. In one aspect, an adaptive lens assembly includes a lens stack configured to exert polarization-dependent optical power to linearly polarized light. The lens stack includes a birefringent lens and an isotropic lens contacting each other to form a conformal interface therebetween. The adaptive lens assembly is configured to be selectively switched between a plurality of states having different optical powers.

Abstract: A method is disclosed, the method comprising the steps of identifying a first real object in a mixed reality environment, the mixed reality environment having a user; identifying a second real object in the mixed reality environment; generating, in the mixed reality environment, a first virtual object corresponding to the second real object; identifying, in the mixed reality environment, a collision between the first real object and the first virtual object; determining a first attribute associated with the collision; determining, based on the first attribute, a first audio signal corresponding to the collision; and presenting to the user, via one or more speakers, the first audio signal.