Patents by Inventor Andrew Rabinovich

Andrew Rabinovich has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Patent number: 10937188
    Abstract: Systems and methods for cuboid detection and keypoint localization in images are disclosed. In one aspect, a deep cuboid detector can be used for simultaneous cuboid detection and keypoint localization in monocular images. The deep cuboid detector can include a plurality of convolutional layers and non-convolutional layers of a trained convolution neural network for determining a convolutional feature map from an input image. A region proposal network of the deep cuboid detector can determine a bounding box surrounding a cuboid in the image using the convolutional feature map. The pooling layer and regressor layers of the deep cuboid detector can implement iterative feature pooling for determining a refined bounding box and a parameterized representation of the cuboid.
    Type: Grant
    Filed: March 5, 2020
    Date of Patent: March 2, 2021
    Assignee: Magic Leap, Inc.
    Inventors: Tomasz Jan Malisiewicz, Andrew Rabinovich, Vijay Badrinarayanan, Debidatta Dwibedi
  • Patent number: 10909711
    Abstract: A method of determining a pose of an image capture device includes capturing an image using an image capture device. The method also includes generating a data structure corresponding to the captured image. The method further includes comparing the data structure with a plurality of known data structures to identify a most similar known data structure. Moreover, the method includes reading metadata corresponding to the most similar known data structure to determine a pose of the image capture device.
    Type: Grant
    Filed: December 5, 2016
    Date of Patent: February 2, 2021
    Assignee: Magic Leap, Inc.
    Inventors: Brigit Schroeder, Tomasz J. Malisiewicz, Andrew Rabinovich
  • Publication number: 20200410699
    Abstract: Systems and methods are disclosed for training and using neural networks for computing depth maps. One method for training the neural network includes providing an image input to the neural network. The image input may include a camera image of a training scene. The method may also include providing a depth input to the neural network. The depth input may be based on a high-density depth map of the training scene and a sampling mask. The method may further include generating, using the neural network, a computed depth map of the training scene based on the image input and the depth input. The method may further include modifying the neural network based on an error between the computed depth map and the high-density depth map.
    Type: Application
    Filed: September 11, 2020
    Publication date: December 31, 2020
    Applicant: Magic Leap, Inc.
    Inventors: Vijay Badrinarayanan, Zhao Chen, Andrew Rabinovich
  • Publication number: 20200380793
    Abstract: A sensory eyewear system for a mixed reality device can facilitate user's interactions with the other people or with the environment. As one example, the sensory eyewear system can recognize and interpret a sign language, and present the translated information to a user of the mixed reality device. The wearable system can also recognize text in the user's environment, modify the text (e.g., by changing the content or display characteristics of the text), and render the modified text to occlude the original text.
    Type: Application
    Filed: August 6, 2020
    Publication date: December 3, 2020
    Inventors: Eric C. Browy, Michael Janusz Woods, Andrew Rabinovich
  • Publication number: 20200372246
    Abstract: A neural network in multi-task deep learning paradigm for machine vision includes an encoder that further includes a first, a second, and a third tier. The first tier comprises a first-tier unit having one or more first-unit blocks. The second tier receives a first-tier output from the first tier at one or more second-tier units in the second tier, a second-tier unit comprises one or more second-tier blocks, the third tier receives a second-tier output from the second tier at one or more third-tier units in the third tier, and a third-tier block comprises one or more third-tier blocks. The neural network further comprises a decoder operatively the encoder to receive an encoder output from the encoder as well as one or more loss function layers that are configured to backpropagate one or more losses for training at least the encoder of the neural network in a deep learning paradigm.
    Type: Application
    Filed: May 20, 2020
    Publication date: November 26, 2020
    Applicant: MAGIC LEAP, INC.
    Inventors: Prajwal CHIDANANDA, Ayan Tuhinendu SINHA, Adithya Shricharan Srinivasa RAO, Douglas Bertram LEE, Andrew RABINOVICH
  • Publication number: 20200351537
    Abstract: The invention provides a content provisioning system. A mobile device has a mobile device processor. The mobile device mobile device has communication interface connected to the mobile device processor and a first resource device communication interface and under the control of the mobile device processor to receive first content transmitted by the first resource device transmitter The mobile device mobile device has a mobile device output device connected to the mobile device processor and under control of the mobile device processor capable of providing an output that can be sensed by a user.
    Type: Application
    Filed: May 1, 2020
    Publication date: November 5, 2020
    Applicant: Magic Leap, Inc.
    Inventors: Eric C. BROWY, Andrew RABINOVICH, David C. LUNDMARK
  • Publication number: 20200334461
    Abstract: A head-mounted augmented reality (AR) device can include a hardware processor programmed to receive different types of sensor data from a plurality of sensors (e.g., an inertial measurement unit, an outward-facing camera, a depth sensing camera, an eye imaging camera, or a microphone); and determining an event of a plurality of events using the different types of sensor data and a hydra neural network (e.g., face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization).
    Type: Application
    Filed: June 30, 2020
    Publication date: October 22, 2020
    Inventors: Andrew Rabinovich, Tomasz Jan Malisiewicz, Daniel DeTone
  • Publication number: 20200334849
    Abstract: A method of determining a pose of an image capture device includes capturing an image using an image capture device. The method also includes generating a data structure corresponding to the captured image. The method further includes comparing the data structure with a plurality of known data structures to identify a most similar known data structure. Moreover, the method includes reading metadata corresponding to the most similar known data structure to determine a pose of the image capture device.
    Type: Application
    Filed: July 7, 2020
    Publication date: October 22, 2020
    Applicant: Magic Leap, Inc.
    Inventors: Brigit SCHROEDER, Tomasz Jan MALISIEWICZ, Andrew RABINOVICH
  • Publication number: 20200302628
    Abstract: Augmented reality devices and methods for computing a homography based on two images. One method may include receiving a first image based on a first camera pose and a second image based on a second camera pose, generating a first point cloud based on the first image and a second point cloud based on the second image, providing the first point cloud and the second point cloud to a neural network, and generating, by the neural network, the homography based on the first point cloud and the second point cloud. The neural network may be trained by generating a plurality of points, determining a 3D trajectory, sampling the 3D trajectory to obtain camera poses viewing the points, projecting the points onto 2D planes, comparing a generated homography using the projected points to the ground-truth homography and modifying the neural network based on the comparison.
    Type: Application
    Filed: June 8, 2020
    Publication date: September 24, 2020
    Applicant: Magic Leap, Inc.
    Inventors: Daniel DeTone, Tomasz Jan Malisiewicz, Andrew Rabinovich
  • Patent number: 10769858
    Abstract: A sensory eyewear system for a mixed reality device can facilitate user's interactions with the other people or with the environment. As one example, the sensory eyewear system can recognize and interpret a sign language, and present the translated information to a user of the mixed reality device. The wearable system can also recognize text in the user's environment, modify the text (e.g., by changing the content or display characteristics of the text), and render the modified text to occlude the original text.
    Type: Grant
    Filed: February 26, 2020
    Date of Patent: September 8, 2020
    Assignee: Magic Leap, Inc.
    Inventors: Eric C. Browy, Michael Janusz Woods, Andrew Rabinovich
  • Patent number: 10733447
    Abstract: A head-mounted augmented reality (AR) device can include a hardware processor programmed to receive different types of sensor data from a plurality of sensors (e.g., an inertial measurement unit, an outward-facing camera, a depth sensing camera, an eye imaging camera, or a microphone); and determining an event of a plurality of events using the different types of sensor data and a hydra neural network (e.g., face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization).
    Type: Grant
    Filed: July 18, 2019
    Date of Patent: August 4, 2020
    Assignee: Magic Leap, Inc.
    Inventors: Andrew Rabinovich, Tomasz Jan Malisiewicz, Daniel DeTone
  • Patent number: 10726570
    Abstract: Augmented reality devices and methods for computing a homography based on two images. One method may include receiving a first image based on a first camera pose and a second image based on a second camera pose, generating a first point cloud based on the first image and a second point cloud based on the second image, providing the first point cloud and the second point cloud to a neural network, and generating, by the neural network, the homography based on the first point cloud and the second point cloud. The neural network may be trained by generating a plurality of points, determining a 3D trajectory, sampling the 3D trajectory to obtain camera poses viewing the points, projecting the points onto 2D planes, comparing a generated homography using the projected points to the ground-truth homography and modifying the neural network based on the comparison.
    Type: Grant
    Filed: June 27, 2018
    Date of Patent: July 28, 2020
    Assignee: Magic Leap, Inc.
    Inventors: Daniel DeTone, Tomasz Jan Malisiewicz, Andrew Rabinovich
  • Publication number: 20200234051
    Abstract: Systems and methods for estimating a layout of a room are disclosed. The room layout can comprise the location of a floor, one or more walls, and a ceiling. In one aspect, a neural network can analyze an image of a portion of a room to determine the room layout. The neural network can comprise a convolutional neural network having an encoder sub-network, a decoder sub-network, and a side sub-network. The neural network can determine a three-dimensional room layout using two-dimensional ordered keypoints associated with a room type. The room layout can be used in applications such as augmented or mixed reality, robotics, autonomous indoor navigation, etc.
    Type: Application
    Filed: April 9, 2020
    Publication date: July 23, 2020
    Inventors: Chen-Yu Lee, Vijay Badrinarayanan, Tomasz Jan Malisiewicz, Andrew Rabinovich
  • Publication number: 20200226785
    Abstract: Systems and methods for reducing error from noisy data received from a high frequency sensor by fusing received input with data received from a low frequency sensor by collecting a first set of dynamic inputs from the high frequency sensor, collecting a correction input point from the low frequency sensor, and adjusting a propagation path of a second set of dynamic inputs from the high frequency sensor based on the correction input point either by full translation to the correction input point or dampened approach towards the correction input point.
    Type: Application
    Filed: March 27, 2020
    Publication date: July 16, 2020
    Applicant: Magic Leap, Inc.
    Inventors: Michael Janusz Woods, Andrew Rabinovich
  • Publication number: 20200202554
    Abstract: Systems and methods for cuboid detection and keypoint localization in images are disclosed. In one aspect, a deep cuboid detector can be used for simultaneous cuboid detection and keypoint localization in monocular images. The deep cuboid detector can include a plurality of convolutional layers and non-convolutional layers of a trained convolution neural network for determining a convolutional feature map from an input image. A region proposal network of the deep cuboid detector can determine a bounding box surrounding a cuboid in the image using the convolutional feature map. The pooling layer and regressor layers of the deep cuboid detector can implement iterative feature pooling for determining a refined bounding box and a parameterized representation of the cuboid.
    Type: Application
    Filed: March 5, 2020
    Publication date: June 25, 2020
    Inventors: Tomasz Jan Malisiewicz, Andrew Rabinovich, Vijay Badrinarayanan, Debidatta Dwibedi
  • Publication number: 20200193714
    Abstract: A sensory eyewear system for a mixed reality device can facilitate user's interactions with the other people or with the environment. As one example, the sensory eyewear system can recognize and interpret a sign language, and present the translated information to a user of the mixed reality device. The wearable system can also recognize text in the user's environment, modify the text (e.g., by changing the content or display characteristics of the text), and render the modified text to occlude the original text.
    Type: Application
    Filed: February 26, 2020
    Publication date: June 18, 2020
    Inventors: Eric C. Browy, Michael Janusz Woods, Andrew Rabinovich
  • Patent number: 10657376
    Abstract: Systems and methods for estimating a layout of a room are disclosed. The room layout can comprise the location of a floor, one or more walls, and a ceiling. In one aspect, a neural network can analyze an image of a portion of a room to determine the room layout. The neural network can comprise a convolutional neural network having an encoder sub-network, a decoder sub-network, and a side sub-network. The neural network can determine a three-dimensional room layout using two-dimensional ordered keypoints associated with a room type. The room layout can be used in applications such as augmented or mixed reality, robotics, autonomous indoor navigation, etc.
    Type: Grant
    Filed: March 16, 2018
    Date of Patent: May 19, 2020
    Assignee: Magic Leap, Inc.
    Inventors: Chen-Yu Lee, Vijay Badrinarayanan, Tomasz Malisiewicz, Andrew Rabinovich
  • Patent number: 10650552
    Abstract: Systems and methods for reducing error from noisy data received from a high frequency sensor by fusing received input with data received from a low frequency sensor by collecting a first set of dynamic inputs from the high frequency sensor, collecting a correction input point from the low frequency sensor, and adjusting a propagation path of a second set of dynamic inputs from the high frequency sensor based on the correction input point either by full translation to the correction input point or dampened approach towards the correction input point.
    Type: Grant
    Filed: December 29, 2017
    Date of Patent: May 12, 2020
    Assignee: Magic Leap, Inc.
    Inventors: Michael Janusz Woods, Andrew Rabinovich
  • Patent number: 10621747
    Abstract: Systems and methods for cuboid detection and keypoint localization in images are disclosed. In one aspect, a deep cuboid detector can be used for simultaneous cuboid detection and keypoint localization in monocular images. The deep cuboid detector can include a plurality of convolutional layers and non-convolutional layers of a trained convolution neural network for determining a convolutional feature map from an input image. A region proposal network of the deep cuboid detector can determine a bounding box surrounding a cuboid in the image using the convolutional feature map. The pooling layer and regressor layers of the deep cuboid detector can implement iterative feature pooling for determining a refined bounding box and a parameterized representation of the cuboid.
    Type: Grant
    Filed: November 14, 2017
    Date of Patent: April 14, 2020
    Assignee: Magic Leap, Inc.
    Inventors: Tomasz Jan Malisiewicz, Andrew Rabinovich, Vijay Badrinarayanan, Debidatta Dwibedi
  • Publication number: 20200111262
    Abstract: Examples of the disclosure describe systems and methods for generating and displaying a virtual companion. In an example method, a first input from an environment of a user is received at a first time via a first sensor on a head-wearable device. An occurrence of an event in the environment is determined based on the first input. A second input from the user is received via a second sensor on the head-wearable device, and an emotional reaction of the user is identified based on the second input. An association is determined between the emotional reaction and the event. A view of the environment is presented at a second time later than the first time via a see-through display of the head-wearable device. A stimulus is presented at the second time via a virtual companion displayed via the see-through display, wherein the stimulus is determined based on the determined association between the emotional reaction and the event.
    Type: Application
    Filed: October 8, 2019
    Publication date: April 9, 2020
    Inventors: Andrew RABINOVICH, John MONOS