Abstract: A virtual image generation system comprises a planar optical waveguide having opposing first and second faces, an in-coupling (IC) element configured for optically coupling a collimated light beam from an image projection assembly into the planar optical waveguide as an in-coupled light beam, a first orthogonal pupil expansion (OPE) element associated with the first face of the planar optical waveguide for splitting the in-coupled light beam into a first set of orthogonal light beamlets, a second orthogonal pupil expansion (OPE) element associated with the second face of the planar optical waveguide for splitting the in-coupled light beam into a second set of orthogonal light beamlets, and an exit pupil expansion (EPE) element associated with the planar optical waveguide for splitting the first and second sets of orthogonal light beamlets into an array of out-coupled light beamlets that exit the planar optical waveguide.

Abstract: A cross reality system enables any of multiple devices to efficiently and accurately access previously stored maps and render virtual content specified in relation to those maps. The cross reality system may include a cloud-based localization service that responds to requests from devices to localize with respect to a stored map. The request may include one or more sets of feature descriptors extracted from an image of the physical world around the device. Those features may be posed relative to a coordinate frame used by the local device. The localization service may identify one or more stored maps with a matching set of features. Based on a transformation required to align the features from the device with the matching set of features, the localization service may compute and return to the device a transformation to relate its local coordinate frame to a coordinate frame of the stored map.

Abstract: Color-selective waveguides, methods for fabricating color-selective waveguides, and augmented reality (AR)/mixed reality (MR) applications including color-selective waveguides are described. The color-selective waveguides can advantageously reduce or block stray light entering a waveguide (e.g., red, green, or blue waveguide), thereby reducing or eliminating back-reflection or back-scattering into the eyepiece.

Abstract: Configurations are disclosed for presenting virtual reality and augmented reality experiences to users. The system may comprise an image capturing device to capture one or more images, the one or more images corresponding to a field of the view of a user of a head-mounted augmented reality device, and a processor communicatively coupled to the image capturing device to extract a set of map points from the set of images, to identify a set of sparse points and a set of dense points from the extracted set of map points, and to perform a normalization on the set of map points.

Abstract: Techniques for resolving hemisphere ambiguity are disclosed. One or more magnetic fields are emitted at a handheld controller. The one or more magnetic fields are detected by one or more sensors positioned relative to a headset. Movement data corresponding to the handheld controller or the headset is detected. During a first time interval, a first position and a first orientation of the handheld controller within a first hemisphere are determined based on the detected one or more magnetic fields, and a first discrepancy is calculated based on the first position, the first orientation, and the movement data. During a second time interval, a second position and a second orientation of the handheld controller within a second hemisphere are determined based on the detected one or more magnetic fields, and a second discrepancy is calculated based on the second position, the second orientation, and the movement data.

Abstract: An eyepiece waveguide includes a set of waveguide layers having a world side and a user side. The eyepiece waveguide also includes a first cover plate having a first optical power and disposed adjacent the world side of the set of waveguide layers and a second cover plate having a second optical power and disposed adjacent the user side of the set of waveguide layers.

Abstract: An optical device includes two or more light sources configured for providing input light beams having different wavelengths. The optical device also has a curved reflective element configured to receive the input light beams to provide two or more reflected light beams. The optical device also has a cylindrical lens for each of the reflected light beams configured to transmit each of the reflected light beams to provide a corresponding collimated output light beam.

Abstract: A display system includes projection optics configured to project a beam of light in a first direction and an eyepiece unit including a first waveguide layer disposed in a first lateral plane and including an incident light surface and an opposing surface opposite the incident light surface and an incoupling diffractive optical element disposed on the incident light surface. The incoupling diffractive optical element is configured to incouple a first portion of the beam of light and to propagate the first portion of the beam of light by total internal reflection in a second direction and transmit a second portion of the beam of light along the first direction. The eyepiece also includes a retroreflector disposed adjacent the opposing surface. The retroreflector is configured to retroreflect the second portion of the beam of light along a reflected direction opposite to the first direction.

Abstract: A method for displaying a three dimensional (“3D”) image includes rendering a frame of 3D image data. The method also includes analyzing the frame of 3D image data to generate best known depth data. The method further includes using the best known depth data to segment the 3D image data into near and far frames of two dimensional (“2D”) image data corresponding to near and far depths respectively. Moreover, the method includes displaying near and far 2D image frames corresponding to the near and far frames of 2D image data at near and far depths to a user respectively.

Abstract: A computer implemented method of facilitating communication between first and second users includes displaying, by a first head-worn device, a first virtual object to the first user first user wearing the first head-worn device. The method also includes displaying, by a second head-worn device, a second virtual object to the second user wearing the second head-worn device. The method further includes facilitating, by the first and second head-worn devices, communications between the first and second users using the first and second virtual objects to simulate the first and second users being present in a common environment.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for object recognition neural network for amodal center prediction. One of the methods includes receiving an image of an object captured by a camera. The image of the object is processed using an object recognition neural network that is configured to generate an object recognition output. The object recognition output includes data defining a predicted two-dimensional amodal center of the object, wherein the predicted two-dimensional amodal center of the object is a projection of a predicted three-dimensional center of the object under a camera pose of the camera that captured the image.

Abstract: An apparatus includes a light-emitting diode (LED) driver circuit, one or more LEDs of an LED array, and an electronic switching circuit. The LED driver circuit is configured to generate an electric current. The one or more LEDs are electrically connected to the LED driver circuit. The electronic switching circuit is electrically connected to the one or more LEDs and configured to be placed in one of multiple switching configurations. The electronic switching circuit is further configured to direct a portion of the electric current away from the one or more LEDs, such that a remaining portion of the electric current drives the one or more LEDs. The portion of the electric current corresponds to the one of the multiple switching configurations.

Abstract: A method of performing localization of a handheld device with respect to a wearable device includes capturing, by a first imaging device mounted to the handheld device, a fiducial image containing a number of fiducials affixed to the wearable device and capturing, by a second imaging device mounted to the handheld device, a world image containing one or more features surrounding the handheld device. The method also includes obtaining, by a sensor mounted to the handheld device, handheld data indicative of movement of the handheld device, determining the number of fiducials contained in the fiducial image, and updating a position and an orientation of the handheld device using at least one of the fiducial image or the world image and the handheld data.

Abstract: Disclosed herein are systems and methods for processing speech signals in mixed reality applications. A method may include receiving an audio signal; determining, via first processors, whether the audio signal comprises a voice onset event; in accordance with a determination that the audio signal comprises the voice onset event: waking a second one or more processors; determining, via the second processors, that the audio signal comprises a predetermined trigger signal; in accordance with a determination that the audio signal comprises the predetermined trigger signal: waking third processors; performing, via the third processors, automatic speech recognition based on the audio signal; and in accordance with a determination that the audio signal does not comprise the predetermined trigger signal: forgoing waking the third processors; and in accordance with a determination that the audio signal does not comprise the voice onset event: forgoing waking the second processors.

Abstract: An anti-reflective waveguide assembly comprising a waveguide substrate having a first index of refraction, a plurality of diffractive optical elements disposed upon a first surface of the waveguide and an anti-reflective coating disposed upon a second surface of the waveguide. The anti-reflective coating preferably increases absorption of light through a surface to which it is applied into the waveguide so that at least 97 percent of the light is transmitted. The anti-reflective coating is composed of four layers of material having different indices of refraction that the first index of refraction and an imaginary refractive index less than 1×10?3 but preferably less than 5×10?4.

Abstract: Waveguides comprising materials with refractive index greater than or equal to 1.8 and methods of patterning waveguides are disclosed. Patterned waveguides comprising materials with refractive index greater than or equal to 1.8 can be incorporated in display devices, such as, for example wearable display devices to project virtual images to a viewer.

Abstract: Example display devices include a waveguide configured to propagate visible light under total internal reflection in a direction parallel to a major surface of the waveguide. The waveguide has formed thereon an outcoupling element configured to outcouple a portion of the visible light in a direction normal to the major surface of the waveguide. The example display devices additionally include a polarization-selective notch reflector disposed on a first side of the waveguide and configured to reflect visible light having a first polarization while transmitting the portion of the visible light having a second polarization. The example display devices further include a polarization-independent notch reflector disposed on a second side of the waveguide and configured to reflect visible light having the first polarization and the second polarization, where the polarization-independent notch reflector is configured to convert a polarization of visible light reflecting therefrom.

Abstract: An image sensor suitable for use in an augmented reality system to provide low latency image analysis with low power consumption. The image sensor may have multiple pixel cells, each of the pixel cells comprising a photodetector to generate an electric signal based on an intensity of light incident upon the photodetector. The pixel cells may include multiple subsets of pixel cells, each subset of pixel cells including at least one angle-of-arrival to-intensity converter to modulate incident light reaching one or more of the pixel cells in the subset based on an angle of arrival of the incident light. Each pixel cell within the plurality of pixel cells may include differential readout circuitry configured to output a readout signal only when an amplitude of a current electric signal from the photodetector is different from an amplitude of a previous electric signal from the photodetector.

Abstract: Techniques are disclosed for allowing a user's hands to interact with virtual objects. An image of at least one hand may be received from an image capture devices. A plurality of keypoints associated with at least one hand may be detected. In response to determining that a hand is making or is transitioning into making a particular gesture, a subset of the plurality of keypoints may be selected. An interaction point may be registered to a particular location relative to the subset of the plurality of keypoints based on the particular gesture. A proximal point may be registered to a location along the user's body. A ray may be cast from the proximal point through the interaction point. A multi-DOF controller for interacting with the virtual object may be formed based on the ray.

Abstract: Augmented reality glasses can include a first laser, a second laser, and a scanning mirror. The augmented reality glasses can also include a first polarization selective reflector and a second polarization selective reflector. The first polarization selective reflector is arranged to receive light from the first laser and reflect light received from the first laser at a first angle to the scanning mirror, and the second polarization selective reflector is arranged to receive light from the second laser and reflect light received from the second laser at a second angle to the scanning mirror. The augmented reality glasses can also have an eyepiece having an input coupling port arranged to receive light reflected by the scanning mirror.