Patents by Inventor Vijay Badrinarayanan
Vijay Badrinarayanan has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).
-
Publication number: 20200234051Abstract: Systems and methods for estimating a layout of a room are disclosed. The room layout can comprise the location of a floor, one or more walls, and a ceiling. In one aspect, a neural network can analyze an image of a portion of a room to determine the room layout. The neural network can comprise a convolutional neural network having an encoder sub-network, a decoder sub-network, and a side sub-network. The neural network can determine a three-dimensional room layout using two-dimensional ordered keypoints associated with a room type. The room layout can be used in applications such as augmented or mixed reality, robotics, autonomous indoor navigation, etc.Type: ApplicationFiled: April 9, 2020Publication date: July 23, 2020Inventors: Chen-Yu Lee, Vijay Badrinarayanan, Tomasz Jan Malisiewicz, Andrew Rabinovich
-
Patent number: 10719951Abstract: Disclosed herein is a wearable display system for capturing retraining eye images of an eye of a user for retraining a neural network for eye tracking. The system captures retraining eye images using an image capture device when user interface (UI) events occur with respect to UI devices displayed at display locations of a display. The system can generate a retraining set comprising the retraining eye images and eye poses of the eye of the user in the retraining eye images (e.g., related to the display locations of the UI devices) and obtain a retrained neural network that is retrained using the retraining set.Type: GrantFiled: September 18, 2018Date of Patent: July 21, 2020Assignee: Magic Leap, Inc.Inventors: Adrian Kaehler, Douglas Bertram Lee, Vijay Badrinarayanan
-
Publication number: 20200202554Abstract: Systems and methods for cuboid detection and keypoint localization in images are disclosed. In one aspect, a deep cuboid detector can be used for simultaneous cuboid detection and keypoint localization in monocular images. The deep cuboid detector can include a plurality of convolutional layers and non-convolutional layers of a trained convolution neural network for determining a convolutional feature map from an input image. A region proposal network of the deep cuboid detector can determine a bounding box surrounding a cuboid in the image using the convolutional feature map. The pooling layer and regressor layers of the deep cuboid detector can implement iterative feature pooling for determining a refined bounding box and a parameterized representation of the cuboid.Type: ApplicationFiled: March 5, 2020Publication date: June 25, 2020Inventors: Tomasz Jan Malisiewicz, Andrew Rabinovich, Vijay Badrinarayanan, Debidatta Dwibedi
-
Patent number: 10657376Abstract: Systems and methods for estimating a layout of a room are disclosed. The room layout can comprise the location of a floor, one or more walls, and a ceiling. In one aspect, a neural network can analyze an image of a portion of a room to determine the room layout. The neural network can comprise a convolutional neural network having an encoder sub-network, a decoder sub-network, and a side sub-network. The neural network can determine a three-dimensional room layout using two-dimensional ordered keypoints associated with a room type. The room layout can be used in applications such as augmented or mixed reality, robotics, autonomous indoor navigation, etc.Type: GrantFiled: March 16, 2018Date of Patent: May 19, 2020Assignee: Magic Leap, Inc.Inventors: Chen-Yu Lee, Vijay Badrinarayanan, Tomasz Malisiewicz, Andrew Rabinovich
-
Patent number: 10621747Abstract: Systems and methods for cuboid detection and keypoint localization in images are disclosed. In one aspect, a deep cuboid detector can be used for simultaneous cuboid detection and keypoint localization in monocular images. The deep cuboid detector can include a plurality of convolutional layers and non-convolutional layers of a trained convolution neural network for determining a convolutional feature map from an input image. A region proposal network of the deep cuboid detector can determine a bounding box surrounding a cuboid in the image using the convolutional feature map. The pooling layer and regressor layers of the deep cuboid detector can implement iterative feature pooling for determining a refined bounding box and a parameterized representation of the cuboid.Type: GrantFiled: November 14, 2017Date of Patent: April 14, 2020Assignee: Magic Leap, Inc.Inventors: Tomasz Jan Malisiewicz, Andrew Rabinovich, Vijay Badrinarayanan, Debidatta Dwibedi
-
Patent number: 10573042Abstract: A wearable device can include an inward-facing imaging system configured to acquire images of a user's periocular region. The wearable device can determine a relative position between the wearable device and the user's face based on the images acquired by the inward-facing imaging system. The relative position may be used to determine whether the user is wearing the wearable device, whether the wearable device fits the user, or whether an adjustment to a rendering location of virtual object should be made to compensate for a deviation of the wearable device from its normal resting position.Type: GrantFiled: September 27, 2017Date of Patent: February 25, 2020Assignee: Magic Leap, Inc.Inventors: Adrian Kaehler, Gary Bradski, Vijay Badrinarayanan
-
Publication number: 20200005462Abstract: Systems and methods for eye image segmentation and image quality estimation are disclosed. In one aspect, after receiving an eye image, a device such as an augmented reality device can process the eye image using a convolutional neural network with a merged architecture to generate both a segmented eye image and a quality estimation of the eye image. The segmented eye image can include a background region, a sclera region, an iris region, or a pupil region. In another aspect, a convolutional neural network with a merged architecture can be trained for eye image segmentation and image quality estimation. In yet another aspect, the device can use the segmented eye image to determine eye contours such as a pupil contour and an iris contour. The device can use the eye contours to create a polar image of the iris region for computing an iris code or biometric authentication.Type: ApplicationFiled: September 13, 2019Publication date: January 2, 2020Inventors: Alexey Spizhevoy, Adrian Kaehler, Vijay Badrinarayanan
-
Patent number: 10445881Abstract: Systems and methods for eye image segmentation and image quality estimation are disclosed. In one aspect, after receiving an eye image, a device such as an augmented reality device can process the eye image using a convolutional neural network with a merged architecture to generate both a segmented eye image and a quality estimation of the eye image. The segmented eye image can include a background region, a sclera region, an iris region, or a pupil region. In another aspect, a convolutional neural network with a merged architecture can be trained for eye image segmentation and image quality estimation. In yet another aspect, the device can use the segmented eye image to determine eye contours such as a pupil contour and an iris contour. The device can use the eye contours to create a polar image of the iris region for computing an iris code or biometric authentication.Type: GrantFiled: May 25, 2017Date of Patent: October 15, 2019Assignee: Magic Leap, Inc.Inventors: Alexey Spizhevoy, Adrian Kaehler, Vijay Badrinarayanan
-
Publication number: 20190286951Abstract: The present disclosure provides an improved approach to implement structure learning of neural networks by exploiting correlations in the data/problem the networks aim to solve. A greedy approach is described that finds bottlenecks of information gain from the bottom convolutional layers all the way to the fully connected layers. Rather than simply making the architecture deeper, additional computation and capacitance is only added where it is required.Type: ApplicationFiled: March 27, 2019Publication date: September 19, 2019Applicant: MAGIC LEAP, INC.Inventors: Andrew RABINOVICH, Vijay BADRINARAYANAN, Daniel DETONE, Srivignesh RAJENDRAN, Douglas Bertram LEE, Tomasz MALISIEWICZ
-
Publication number: 20190289281Abstract: Systems and methods are disclosed for computing depth maps. One method includes capturing, using a camera, a camera image of a runtime scene. The method may also include analyzing the camera image of the runtime scene to determine a plurality of target sampling points at which to capture depth of the runtime scene. The method may further include adjusting a setting associated with a low-density depth sensor based on the plurality of target sampling points. The method may further include capturing, using the low-density depth sensor, a low-density depth map of the runtime scene at the plurality of target sampling points. The method may further include generating a computed depth map of the runtime scene based on the camera image of the runtime scene and the low-density depth map of the runtime scene.Type: ApplicationFiled: March 13, 2019Publication date: September 19, 2019Applicant: Magic Leap, Inc.Inventors: Vijay BADRINARAYANAN, Zhao CHEN, Andrew RABINOVICH, Elad JOSEPH
-
Publication number: 20190147298Abstract: Methods and systems for meta-learning are described for automating learning of child tasks with a single neural network. The order in which tasks are learned by the neural network can affect performance of the network, and the meta-learning approach can use a task-level curriculum for multi-task training. The task-level curriculum can be learned by monitoring a trajectory of loss functions during training. The meta-learning approach can learn to adapt task loss balancing weights in the course of training to get improved performance on multiple tasks on real world datasets. Advantageously, learning to dynamically balance weights among different task losses can lead to superior performance over the use of static weights determined by expensive random searches or heuristics. Embodiments of the meta-learning approach can be used for computer vision tasks or natural language processing tasks, and the trained neural networks can be used by augmented or virtual reality devices.Type: ApplicationFiled: November 9, 2018Publication date: May 16, 2019Inventors: Andrew RABINOVICH, Vijay BADRINARAYANAN, Srivignesh RAJENDRAN, Chen-Yu LEE
-
Publication number: 20190130275Abstract: Systems and methods for training a multitask network is disclosed. In one aspect, training the multitask network includes determining a gradient norm of a single-task loss adjusted by a task weight for each task, with respect to network weights of the multitask network, and a relative training rate for the task based on the single-task loss for the task. Subsequently, a gradient loss function, comprising a difference between (1) the determined gradient norm for each task and (2) a corresponding target gradient norm, can be determined. An updated task weight for the task can be determined and used in the next iteration of training the multitask network, using a gradient of the gradient loss function with respect to the task weight for the task.Type: ApplicationFiled: October 24, 2018Publication date: May 2, 2019Inventors: Zhao Chen, Vijay Badrinarayanan, Andrew Rabinovich
-
Patent number: 10255529Abstract: The present disclosure provides an improved approach to implement structure learning of neural networks by exploiting correlations in the data/problem the networks aim to solve. A greedy approach is described that finds bottlenecks of information gain from the bottom convolutional layers all the way to the fully connected layers. Rather than simply making the architecture deeper, additional computation and capacitance is only added where it is required.Type: GrantFiled: March 13, 2017Date of Patent: April 9, 2019Assignee: Magic Leap, Inc.Inventors: Andrew Rabinovich, Vijay Badrinarayanan, Daniel DeTone, Srivignesh Rajendran, Douglas Bertram Lee, Tomasz Malisiewicz
-
Publication number: 20190087973Abstract: Disclosed herein is a wearable display system for capturing retraining eye images of an eye of a user for retraining a neural network for eye tracking. The system captures retraining eye images using an image capture device when user interface (UI) events occur with respect to UI devices displayed at display locations of a display. The system can generate a retraining set comprising the retraining eye images and eye poses of the eye of the user in the retraining eye images (e.g., related to the display locations of the UI devices) and obtain a retrained neural network that is retrained using the retraining set.Type: ApplicationFiled: September 18, 2018Publication date: March 21, 2019Inventors: Adrian Kaehler, Douglas Lee, Vijay Badrinarayanan
-
Publication number: 20190034765Abstract: Disclosed herein are examples of a wearable display system capable of determining a user interface (UI) event with respect to a virtual UI device (e.g., a button) and a pointer (e.g., a finger or a stylus) using a neural network. The wearable display system can render a representation of the UI device onto an image of the pointer captured when the virtual UI device is shown to the user and the user uses the pointer to interact with the virtual UI device. The representation of the UI device can include concentric shapes (or shapes with similar or the same centers of gravity) of high contrast. The neural network can be trained using training images with representations of virtual UI devices and pointers.Type: ApplicationFiled: May 31, 2018Publication date: January 31, 2019Inventors: Adrian Kaehler, Gary Bradski, Vijay Badrinarayanan
-
Publication number: 20180268220Abstract: Systems and methods for estimating a layout of a room are disclosed. The room layout can comprise the location of a floor, one or more walls, and a ceiling. In one aspect, a neural network can analyze an image of a portion of a room to determine the room layout. The neural network can comprise a convolutional neural network having an encoder sub-network, a decoder sub-network, and a side sub-network. The neural network can determine a three-dimensional room layout using two-dimensional ordered keypoints associated with a room type. The room layout can be used in applications such as augmented or mixed reality, robotics, autonomous indoor navigation, etc.Type: ApplicationFiled: March 16, 2018Publication date: September 20, 2018Inventors: Chen-Yu Lee, Vijay Badrinarayanan, Tomasz Malisiewicz, Andrew Rabinovich
-
Publication number: 20180137642Abstract: Systems and methods for cuboid detection and keypoint localization in images are disclosed. In one aspect, a deep cuboid detector can be used for simultaneous cuboid detection and keypoint localization in monocular images. The deep cuboid detector can include a plurality of convolutional layers and non-convolutional layers of a trained convolution neural network for determining a convolutional feature map from an input image. A region proposal network of the deep cuboid detector can determine a bounding box surrounding a cuboid in the image using the convolutional feature map. The pooling layer and regressor layers of the deep cuboid detector can implement iterative feature pooling for determining a refined bounding box and a parameterized representation of the cuboid.Type: ApplicationFiled: November 14, 2017Publication date: May 17, 2018Inventors: Tomasz Malisiewicz, Andrew Rabinovich, Vijay Badrinarayanan, Debidatta Dwibedi
-
Publication number: 20180096503Abstract: A wearable device can include an inward-facing imaging system configured to acquire images of a user's periocular region. The wearable device can determine a relative position between the wearable device and the user's face based on the images acquired by the inward-facing imaging system. The relative position may be used to determine whether the user is wearing the wearable device, whether the wearable device fits the user, or whether an adjustment to a rendering location of virtual object should be made to compensate for a deviation of the wearable device from its normal resting position.Type: ApplicationFiled: September 27, 2017Publication date: April 5, 2018Inventors: Adrian Kaehler, Gary Bradski, Vijay Badrinarayanan
-
Publication number: 20180089834Abstract: Systems and methods for eye image segmentation and image quality estimation are disclosed. In one aspect, after receiving an eye image, a device such as an augmented reality device can process the eye image using a convolutional neural network with a merged architecture to generate both a segmented eye image and a quality estimation of the eye image. The segmented eye image can include a background region, a sclera region, an iris region, or a pupil region. In another aspect, a convolutional neural network with a merged architecture can be trained for eye image segmentation and image quality estimation. In yet another aspect, the device can use the segmented eye image to determine eye contours such as a pupil contour and an iris contour. The device can use the eye contours to create a polar image of the iris region for computing an iris code or biometric authentication.Type: ApplicationFiled: May 25, 2017Publication date: March 29, 2018Inventors: Alexey Spizhevoy, Adrian Kaehler, Vijay Badrinarayanan
-
Publication number: 20170262737Abstract: The present disclosure provides an improved approach to implement structure learning of neural networks by exploiting correlations in the data/problem the networks aim to solve. A greedy approach is described that finds bottlenecks of information gain from the bottom convolutional layers all the way to the fully connected layers. Rather than simply making the architecture deeper, additional computation and capacitance is only added where it is required.Type: ApplicationFiled: March 13, 2017Publication date: September 14, 2017Applicant: Magic Leap, Inc.Inventors: Andrew Rabinovich, Vijay Badrinarayanan, Daniel DeTone, Srivignesh Rajendran, Douglas Bertram Lee, Tomasz Malisiewicz