Patents Assigned to Magic Leap, Inc.
  • Publication number: 20210366077
    Abstract: Embodiments shifts the color fields of a rendered image frame based on the eye tracking data (e.g. position of the user's pupils). An MR device obtains a first image frame having a set of color fields. The first image frame corresponds to a first position of the pupils of the viewer. The MR device then determines a second position of the pupils of the viewer based on, for example, data receive from an eye tracking device coupled to the MR device. The MR device generates, based on the first image frame, a second image frame corresponding to the second position of the pupils. The second image frame is generated by shifting color fields by a shift value based on the second position of the pupils of the viewer. The MR device transmits the second image frame to a display device of the MR device to be displayed thereon.
    Type: Application
    Filed: May 20, 2021
    Publication date: November 25, 2021
    Applicant: Magic Leap, Inc.
    Inventor: Andrew Ian Russell
  • Publication number: 20210365785
    Abstract: A method for training a neural network includes receiving a plurality of images and, for each individual image of the plurality of images, generating a training triplet including a subset of the individual image, a subset of a transformed image, and a homography based on the subset of the individual image and the subset of the transformed image. The method also includes, for each individual image, generating, by the neural network, an estimated homography based on the subset of the individual image and the subset of the transformed image, comparing the estimated homography to the homography, and modifying the neural network based on the comparison.
    Type: Application
    Filed: June 7, 2021
    Publication date: November 25, 2021
    Applicant: Magic Leap, Inc.
    Inventors: Daniel DeTone, Tomasz Malisiewicz, Andrew Rabinovich
  • Publication number: 20210366334
    Abstract: Embodiments transform an image frame based on a position of pupils of a viewer to eliminate visual artefacts formed on an image frame displayed on a scanning-type display device. An MR system obtains a first image frame corresponding to a first view perspective associated with a first pupil position. The system receives data from an eye tracking device, determines a second pupil position, and generates a second image frame corresponding to a second view perspective associated with the second pupil position. A first set of pixels of the second image frame are shifted by a first shift value, and a second set of pixels of the second image frame are shifted by a second shift value, where the shift values are calculated based on at least the second pupil position. The system transmits the second image frame to a near-eye display device to be displayed thereon.
    Type: Application
    Filed: May 20, 2021
    Publication date: November 25, 2021
    Applicant: Magic Leap, Inc.
    Inventor: Andrew Ian Russell
  • Publication number: 20210363003
    Abstract: A method for fabricating a cantilever having a device surface, a tapered surface, and an end region includes providing a semiconductor substrate having a first side and a second side opposite to the first side and etching a predetermined portion of the second side to form a plurality of recesses in the second side. Each of the plurality of recesses comprises an etch termination surface. The method also includes anisotropically etching the etch termination surface to form the tapered surface of the cantilever and etching a predetermined portion of the device surface to release the end region of the cantilever.
    Type: Application
    Filed: May 22, 2021
    Publication date: November 25, 2021
    Applicant: Magic Leap, Inc.
    Inventors: Steven Alexander-Boyd Hickman, Sarah Colline McQuaide, Abhijith Rajiv, Brian T. Schowengerdt, Charles David Melville
  • Publication number: 20210364805
    Abstract: An eyepiece for use in front of an eye of a viewer includes a waveguide configured to propagate light therein, and a diffractive optical element optically coupled to the waveguide. The diffractive optical element includes a plurality of first ridges protruding from a surface of the waveguide. Each of the plurality of first ridges has a first height and a first width. The diffractive optical element further includes a plurality of second ridges. Each of the plurality of second ridges protrudes from a respective first ridge and has a second height greater than the first height and a second width less than the first width. The diffractive optical element is configured to diffract a portion of a light beam incident on the diffractive optical element toward the eye as a first order transmission.
    Type: Application
    Filed: June 4, 2021
    Publication date: November 25, 2021
    Applicant: Magic Leap, Inc.
    Inventors: Robert D. Tekolste, Victor K. Liu
  • Publication number: 20210364803
    Abstract: An eyepiece waveguide for an augmented reality display system includes a substrate having a first surface and a second surface and a diffractive input coupling element formed on or in the first surface or the second surface of the substrate. The diffractive input coupling element is configured to receive an input beam of light and to couple the input beam into the substrate as a guided beam. The eyepiece waveguide also includes a diffractive combined pupil expander-extractor (CPE) element formed on or in the first surface or the second surface of the substrate. The diffractive CPE element includes a first portion and a second portion divided by an axis. A first set of diffractive optical elements is disposed in the first portion and oriented at a positive angle with respect to the axis and a second set of diffractive optical elements is disposed in the second portion and oriented at a negative angle with respect to the axis.
    Type: Application
    Filed: May 21, 2021
    Publication date: November 25, 2021
    Applicant: Magic Leap, Inc.
    Inventors: Brian T. Schowengerdt, Mathew D. Watson, Brandon Michael-James Born, Samarth Bhargava, Victor Kai Liu
  • Patent number: 11181974
    Abstract: Techniques are disclosed for performing localization of a handheld device with respect to a wearable device. At least one sensor mounted to the handheld device, such as an inertial measurement unit (IMU), may obtain handheld data indicative of movement of the handheld device with respect to the world. An imaging device mounted to either the handheld device or the wearable device may capture a fiducial image containing a number of fiducials affixed to the other device. The number of fiducials contained in the image are determined. Based on the number of fiducials, at least one of a position and an orientation of the handheld device with respect to the wearable device are updated based on the image and the handheld data in accordance with a first operating state, a second operating state, or a third operating state.
    Type: Grant
    Filed: October 29, 2020
    Date of Patent: November 23, 2021
    Assignee: Magic Leap, Inc.
    Inventors: Zachary C. Nienstedt, Samuel A. Miller, Barak Freedman, Lionel Ernest Edwin, Eric C. Browy, William Hudson Welch, Ron Liraz Lidji
  • Patent number: 11182645
    Abstract: Systems and methods for efficient implementation of a convolutional layer of a convolutional neural network are disclosed. In one aspect, weight values of kernels in a kernel stack of a convolutional layer can be reordered into a tile layout with tiles of runnels. Pixel values of input activation maps of the convolutional layer can be reordered into an interleaved layout comprising a plurality of clusters of input activation map pixels. The output activation maps can be determined using the clusters of the input activation map pixels and kernels tile by tile.
    Type: Grant
    Filed: November 11, 2019
    Date of Patent: November 23, 2021
    Assignee: Magic Leap, Inc.
    Inventors: Ashkan Aliabadi, Gregory David Roberts
  • Patent number: 11182614
    Abstract: An apparatus configured to be worn on a head of a user, includes: a screen configured to present graphics to the user; a camera system configured to view an environment in which the user is located; and a processing unit configured to determine a map based at least in part on output(s) from the camera system, wherein the map is configured for use by the processing unit to localize the user with respect to the environment; wherein the processing unit of the apparatus is also configured to obtain a metric indicating a likelihood of success to localize the user using the map, and wherein the processing unit is configured to obtain the metric by computing the metric or by receiving the metric.
    Type: Grant
    Filed: July 24, 2019
    Date of Patent: November 23, 2021
    Assignee: Magic Leap, Inc.
    Inventors: Divya Sharma, Ali Shahrokni, Anush Mohan, Prateek Singhal, Xuan Zhao, Sergiu Sima, Benjamin Langmann
  • Publication number: 20210356737
    Abstract: A method of fabricating a multi-element fiber scanner includes providing a fiber optic cable having a cladding region and a fiber core and focusing a laser beam at a series of predetermined locations inside the cladding region of the fiber optic cable. The method also includes creating a plurality of damage sites at the predetermined locations, exposing the fiber optic cable to an etchant solution, and preferentially etching the plurality of damage sites to form a base having a base plane and a longitudinal axis orthogonal to the base plane, a retention collar disposed a predetermined distance along the longitudinal axis from the base, a first fiber link including the fiber core, passing through the base plane, and joined to the retention collar, and a plurality of additional links joined to the base, extending from the base to the retention collar, and joined to the retention collar.
    Type: Application
    Filed: May 21, 2021
    Publication date: November 18, 2021
    Applicant: Magic Leap, Inc.
    Inventors: Brian T. Schowengerdt, Mathew D. Watson, Charles David Melville, William Andrew Lee
  • Publication number: 20210356748
    Abstract: A display system includes projection optics configured to project a beam of light in a first direction and an eyepiece unit including a first waveguide layer disposed in a first lateral plane and including an incident light surface and an opposing surface opposite the incident light surface and an incoupling diffractive optical element disposed on the incident light surface. The incoupling diffractive optical element is configured to incouple a first portion of the beam of light and to propagate the first portion of the beam of light by total internal reflection in a second direction and transmit a second portion of the beam of light along the first direction. The eyepiece also includes a retroreflector disposed adjacent the opposing surface. The retroreflector is configured to retroreflect the second portion of the beam of light along a reflected direction opposite to the first direction.
    Type: Application
    Filed: May 27, 2021
    Publication date: November 18, 2021
    Applicant: Magic Leap, Inc.
    Inventor: Hui-Chuan Cheng
  • Publication number: 20210358173
    Abstract: Methods and apparatus for providing a representation of an environment, for example, in an XR system, and any suitable computer vision and robotics applications. A representation of an environment may include one or more planar features. The representation of the environment may be provided by jointly optimizing plane parameters of the planar features and sensor poses that the planar features are observed at. The joint optimization may be based on a reduced matrix and a reduced residual vector in lieu of the Jacobian matrix and the original residual vector.
    Type: Application
    Filed: May 10, 2021
    Publication date: November 18, 2021
    Applicant: Magic Leap, Inc.
    Inventors: Lipu Zhou, Frank Thomas Steinbruecker, Ashwin Swaminathan, Hui Ju, Daniel Esteban Koppel, Konstantinos Zampogiannis, Pooja Piyush Mehta, Vinayram Balakumar
  • Publication number: 20210358224
    Abstract: A virtual image generation system comprises a planar optical waveguide having opposing first and second faces, an in-coupling (IC) element configured for optically coupling a collimated light beam from an image projection assembly into the planar optical waveguide as an in-coupled light beam, a first orthogonal pupil expansion (OPE) element associated with the first face of the planar optical waveguide for splitting the in-coupled light beam into a first set of orthogonal light beamlets, a second orthogonal pupil expansion (OPE) element associated with the second face of the planar optical waveguide for splitting the in-coupled light beam into a second set of orthogonal light beamlets, and an exit pupil expansion (EPE) element associated with the planar optical waveguide for splitting the first and second sets of orthogonal light beamlets into an array of out-coupled light beamlets that exit the planar optical waveguide.
    Type: Application
    Filed: July 30, 2021
    Publication date: November 18, 2021
    Applicant: Magic Leap, Inc.
    Inventors: Brian T. SCHOWENGERDT, Mathew D. WATSON, David TINCH, Ivan Li Chuen YEOH, John Graham Macnamara, Lionel Ernest EDWIN, Michael Anthony KLUG, William Hudson WELCH
  • Publication number: 20210356747
    Abstract: A pupil separation system includes an input surface, a central portion including a set of dichroic mirrors, a first reflective surface disposed laterally with respect to the central portion, and a second reflective surface disposed laterally with respect to the central portion. The pupil separation system also includes an exit face including a central surface operable to transmit light in a first wavelength range, a first peripheral surface adjacent the central surface and operable to transmit light in a second wavelength range, and a second peripheral surface adjacent the central surface and opposite to the first peripheral surface. The second peripheral surface is operable to transmit light in a third wavelength range.
    Type: Application
    Filed: May 13, 2021
    Publication date: November 18, 2021
    Applicant: Magic Leap, Inc.
    Inventors: Ravi Kumar Komandury, Chulwoo Oh, Kevin Messer, Ioannis Papadopoulos
  • Publication number: 20210356654
    Abstract: A method of forming a tapered tip of a polarization-maintaining (PM) fiber includes inserting a tip of the PM fiber into a first etchant solution characterized by a first etching rate for the core of the PM fiber and a second etching rate for the stress members of the PM fiber, the second etching rate being lower than the first etching rate, withdrawing the tip of the PM fiber from the first etchant solution at a withdrawal rate, immersing the tip of the PM fiber in a second etchant solution for a time duration. The second etchant solution is characterized by a third etching rate for the core and a fourth etching rate for the stress members, the fourth etching rate being greater than the third etching rate. The method further includes withdrawing the tip of the PM fiber from the second etchant solution.
    Type: Application
    Filed: May 12, 2021
    Publication date: November 18, 2021
    Applicant: Magic Leap, Inc.
    Inventors: Sarah Colline McQuaide, Brian T. Schowengerdt
  • Publication number: 20210356670
    Abstract: An eyepiece waveguide includes a set of waveguide layers having a world side and a user side. The eyepiece waveguide also includes a first cover plate having a first optical power and disposed adjacent the world side of the set of waveguide layers and a second cover plate having a second optical power and disposed adjacent the user side of the set of waveguide layers.
    Type: Application
    Filed: May 13, 2021
    Publication date: November 18, 2021
    Applicant: Magic Leap, Inc.
    Inventors: Chulwoo Oh, Ravi Kumar Komanduri, Vikramjit Singh, Shuqiang Yang, Frank Y. Xu
  • Patent number: 11176757
    Abstract: An augmented reality (AR) display device can display a virtual assistant character that interacts with the user of the AR device. The virtual assistant may be represented by a robot (or other) avatar that assists the user with contextual objects and suggestions depending on what virtual content the user is interacting with. Animated images may be displayed above the robot's head to display its intents to the user. For example, the robot can run up to a menu and suggest an action and show the animated images. The robot can materialize virtual objects that appear on its hands. The user can remove such an object from the robot's hands and place it in the environment. If the user does not interact with the object, the robot can dematerialize it. The robot can rotate its head to keep looking at the user and/or an object that the user has picked up.
    Type: Grant
    Filed: October 1, 2020
    Date of Patent: November 16, 2021
    Assignee: Magic Leap, Inc.
    Inventors: Kristofer Ryan Whitney, Andrew Moran, Danielle Marie Price, Jonathan Wells Mangagil, Minal Luxman Kalkute
  • Patent number: 11175750
    Abstract: Systems and methods for interacting with virtual objects in a three-dimensional space using a wearable system are disclosed. The wearable system can be programmed to permit user interaction with interactable objects in a field of regard (FOR) of a user. The FOR includes a portion of the environment around the user that is capable of being perceived by the user via the AR system. The system can determine a group of interactable objects in the FOR of the user and determine a pose of the user. The system can update, based on a change in the pose or a field of view (FOV) of the user, a subgroup of the interactable objects that are located in the FOV of the user and receive a selection of a target interactable object from the subgroup of interactable objects. The system can initiate a selection event on the target interactable object.
    Type: Grant
    Filed: November 13, 2019
    Date of Patent: November 16, 2021
    Assignee: Magic Leap, Inc.
    Inventors: James M. Powderly, Savannah Niles, Frank Hamilton, Marshal A. Fontaine, Rony Abovitz, Alysha Naples
  • Patent number: D936096
    Type: Grant
    Filed: February 26, 2020
    Date of Patent: November 16, 2021
    Assignee: Magic Leap, Inc.
    Inventors: Sean Eugene Couture, Gregory Minh Tran
  • Patent number: D936704
    Type: Grant
    Filed: January 27, 2020
    Date of Patent: November 23, 2021
    Assignee: Magic Leap, Inc.
    Inventors: Lorena Pazmino, Karen Stolzenberg, Ian Mankowski, Paul Kim, Christina Lee