Abstract: In an example method of training a neural network for performing visual odometry, the neural network receives a plurality of images of an environment, determines, for each image, a respective set of interest points and a respective descriptor, and determines a correspondence between the plurality of images. Determining the correspondence includes determining one or point correspondences between the sets of interest points, and determining a set of candidate interest points based on the one or more point correspondences, each candidate interest point indicating a respective feature in the environment in three-dimensional space). The neural network determines, for each candidate interest point, a respective stability metric and a respective stability metric. The neural network is modified based on the one or more candidate interest points.

Abstract: A deep neural network provides real-time pose estimation by combining two custom deep neural networks, a location classifier and an ID classifier, with a pose estimation algorithm to achieve a 6D0F location of a fiducial marker. The locations may be further refined into subpixel coordinates using another deep neural network. The networks may be trained using a combination of auto-labeled videos of the target marker, synthetic subpixel corner data, and/or extreme data augmentation. The deep neural network provides improved pose estimations particularly in challenging low-light, high-motion, and/or high-blur scenarios.

Abstract: A head-mounted augmented reality (AR) device can include a hardware processor programmed to receive different types of sensor data from a plurality of sensors (e.g., an inertial measurement unit, an outward-facing camera, a depth sensing camera, an eye imaging camera, or a microphone); and determining an event of a plurality of events using the different types of sensor data and a hydra neural network (e.g., face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization).

Abstract: The present disclosure provides an improved approach to implement structure learning of neural networks by exploiting correlations in the data/problem the networks aim to solve. A greedy approach is described that finds bottlenecks of information gain from the bottom convolutional layers all the way to the fully connected layers. Rather than simply making the architecture deeper, additional computation and capacitance is only added where it is required.

Abstract: A method for training a neural network includes receiving a plurality of images and, for each individual image of the plurality of images, generating a training triplet including a subset of the individual image, a subset of a transformed image, and a homography based on the subset of the individual image and the subset of the transformed image. The method also includes, for each individual image, generating, by the neural network, an estimated homography based on the subset of the individual image and the subset of the transformed image, comparing the estimated homography to the homography, and modifying the neural network based on the comparison.

Abstract: Systems, devices, and methods for training a neural network and performing image interest point detection and description using the neural network. The neural network may include an interest point detector subnetwork and a descriptor subnetwork. An optical device may include at least one camera for capturing a first image and a second image. A first set of interest points and a first descriptor may be calculated using the neural network based on the first image, and a second set of interest points and a second descriptor may be calculated using the neural network based on the second image. A homography between the first image and the second image may be determined based on the first and second sets of interest points and the first and second descriptors. The optical device may adjust virtual image light being projected onto an eyepiece based on the homography.

Abstract: A head-mounted augmented reality (AR) device can include a hardware processor programmed to receive different types of sensor data from a plurality of sensors (e.g., an inertial measurement unit, an outward-facing camera, a depth sensing camera, an eye imaging camera, or a microphone); and determining an event of a plurality of events using the different types of sensor data and a hydra neural network (e.g., face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization).

Abstract: Augmented reality devices and methods for computing a homography based on two images. One method may include receiving a first image based on a first camera pose and a second image based on a second camera pose, generating a first point cloud based on the first image and a second point cloud based on the second image, providing the first point cloud and the second point cloud to a neural network, and generating, by the neural network, the homography based on the first point cloud and the second point cloud. The neural network may be trained by generating a plurality of points, determining a 3D trajectory, sampling the 3D trajectory to obtain camera poses viewing the points, projecting the points onto 2D planes, comparing a generated homography using the projected points to the ground-truth homography and modifying the neural network based on the comparison.

Abstract: In an example method of training a neural network for performing visual odometry, the neural network receives a plurality of images of an environment, determines, for each image, a respective set of interest points and a respective descriptor, and determines a correspondence between the plurality of images. Determining the correspondence includes determining one or point corrspondences between the sets of interest points, and determining a set of candidate interest points based on the one or more point correspondences, each candidate interest point indicating a respective feature in the environment in three-dimensional space). The neural network determines, for each candidate interest point, a respective stability metric and a respective stability metric. The neural network is modified based on the one or more candidate interest points.

Abstract: A method for training a neural network includes receiving a plurality of images and, for each individual image of the plurality of images, generating a training triplet including a subset of the individual image, a subset of a transformed image, and a homography based on the subset of the individual image and the subset of the transformed image. The method also includes, for each individual image, generating, by the neural network, an estimated homography based on the subset of the individual image and the subset of the transformed image, comparing the estimated homography to the homography, and modifying the neural network based on the comparison.

Abstract: A deep neural network provides real-time pose estimation by combining two custom deep neural networks, a location classifier and an ID classifier, with a pose estimation algorithm to achieve a 6D0F location of a fiducial marker. The locations may be further refined into subpixel coordinates using another deep neural network. The networks may be trained using a combination of auto-labeled videos of the target marker, synthetic subpixel corner data, and/or extreme data augmentation. The deep neural network provides improved pose estimations particularly in challenging low-light, high-motion, and/or high-blur scenarios.

Abstract: A head-mounted augmented reality (AR) device can include a hardware processor programmed to receive different types of sensor data from a plurality of sensors (e.g., an inertial measurement unit, an outward-facing camera, a depth sensing camera, an eye imaging camera, or a microphone); and determining an event of a plurality of events using the different types of sensor data and a hydra neural network (e.g., face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization).

Abstract: Systems, devices, and methods for training a neural network and performing image interest point detection and description using the neural network. The neural network may include an interest point detector subnetwork and a descriptor subnetwork. An optical device may include at least one camera for capturing a first image and a second image. A first set of interest points and a first descriptor may be calculated using the neural network based on the first image, and a second set of interest points and a second descriptor may be calculated using the neural network based on the second image. A homography between the first image and the second image may be determined based on the first and second sets of interest points and the first and second descriptors. The optical device may adjust virtual image light being projected onto an eyepiece based on the homography.

Abstract: A method for training a neural network includes receiving a plurality of images and, for each individual image of the plurality of images, generating a training triplet including a subset of the individual image, a subset of a transformed image, and a homography based on the subset of the individual image and the subset of the transformed image. The method also includes, for each individual image, generating, by the neural network, an estimated homography based on the subset of the individual image and the subset of the transformed image, comparing the estimated homography to the homography, and modifying the neural network based on the comparison.

Abstract: The present disclosure provides an improved approach to implement structure learning of neural networks by exploiting correlations in the data/problem the networks aim to solve. A greedy approach is described that finds bottlenecks of information gain from the bottom convolutional layers all the way to the fully connected layers. Rather than simply making the architecture deeper, additional computation and capacitance is only added where it is required.

Abstract: The description relates the feature matching. Our approach establishes pointwise correspondences between challenging image pairs. It takes off-the-shelf local features as input and uses an attentional graph neural network to solve an assignment optimization problem. The deep middle-end matcher acts as a middle-end and handles partial point visibility and occlusion elegantly, producing a partial assignment matrix.

Abstract: Systems, devices, and methods for training a neural network and performing image interest point detection and description using the neural network. The neural network may include an interest point detector subnetwork and a descriptor subnetwork. An optical device may include at least one camera for capturing a first image and a second image. A first set of interest points and a first descriptor may be calculated using the neural network based on the first image, and a second set of interest points and a second descriptor may be calculated using the neural network based on the second image. A homography between the first image and the second image may be determined based on the first and second sets of interest points and the first and second descriptors. The optical device may adjust virtual image light being projected onto an eyepiece based on the homography.

Abstract: The present disclosure provides an improved approach to implement structure learning of neural networks by exploiting correlations in the data/problem the networks aim to solve. A greedy approach is described that finds bottlenecks of information gain from the bottom convolutional layers all the way to the fully connected layers. Rather than simply making the architecture deeper, additional computation and capacitance is only added where it is required.

Abstract: A head-mounted augmented reality (AR) device can include a hardware processor programmed to receive different types of sensor data from a plurality of sensors (e.g., an inertial measurement unit, an outward-facing camera, a depth sensing camera, an eye imaging camera, or a microphone); and determining an event of a plurality of events using the different types of sensor data and a hydra neural network (e.g., face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization).

Abstract: Augmented reality devices and methods for computing a homography based on two images. One method may include receiving a first image based on a first camera pose and a second image based on a second camera pose, generating a first point cloud based on the first image and a second point cloud based on the second image, providing the first point cloud and the second point cloud to a neural network, and generating, by the neural network, the homography based on the first point cloud and the second point cloud. The neural network may be trained by generating a plurality of points, determining a 3D trajectory, sampling the 3D trajectory to obtain camera poses viewing the points, projecting the points onto 2D planes, comparing a generated homography using the projected points to the ground-truth homography and modifying the neural network based on the comparison.