Abstract: Aspects of the present disclosure relate to instant boundary creation for a virtual reality (VR) experience. An artificial reality (XR) system, while rendering an augmented reality (AR) or mixed reality (MR) environment, can scan and gather visual characteristic data for a user's real-world environment in the background. When the XR system detects an intent to enter VR mode (e.g., by launching a VR application), the XR system can generate a recommendation for a boundary for the real-world space, which can include a type of interaction mode (e.g., moveable or stationary mode). Based on the user's response to the recommendation, the XR system can prompt the user to scan the real-world space further and/or manually adjust the boundary, while continuing to scan and gather visual characteristic data in the background. Some implementations can alternatively or additionally automatically relocalize a real-world space based on a generated boundary.

Abstract: A cross reality system enables any of multiple devices to efficiently and accurately access previously persisted maps of very large scale environments and render virtual content specified in relation to those maps. The cross reality system may quickly determine whether a 2D set of features derived from images acquired with a portable device match a set of 3D features of an environment map and, if so, determine the relative pose of the feature sets. The pose may be used in quickly and accurately localizing the portable device to the environment map. Pairs of features in the 2D and 3D features sets may be identified based on matching feature descriptors and may be scored in a neural network trained to assess the quality of the match. Poses may be identified based on subsets of the matching features weighted towards pairs of features with high quality.

Abstract: For artificial reality (XR) applications that need a guardian, aspects of the present disclosure can automatically determine where boundaries should be set in the real-world environment. Using a machine learning model, the automatic boundary system can detect the floor plane, then generate a height map of the real-world environment with the floor at zero. The height map can include positive heights (e.g., objects on the floor) and negative heights (e.g., downward leading stairs). The automatic boundary system can then generate a boundary for the area based on the detected heights by: applying thresholds to disregard objects at certain heights, applying thresholds to disregard open areas at certain widths, excluding oddly shaped areas that would constantly trigger the boundary if the user were nearby, etc. The user can further manually adjust the generated boundary in both directions (e.g., bringing it closer or further from the user).

Abstract: An apparatus configured to be head-worn by a user, includes: a screen configured to present graphics for the user; a camera system configured to view an environment in which the user is located; and a processing unit coupled to the camera system, the processing unit configured to: obtain locations of features for an image of the environment, wherein the locations of the features are identified by a neural network; determine a region of interest for one of the features in the image, the region of interest having a size that is less than a size of the image; and perform a corner detection using a corner detection algorithm to identify a corner in the region of interest.

Abstract: An apparatus configured to be head-worn by a user, includes: a screen configured to present graphics for the user; a camera system configured to view an environment in which the user is located; and a processing unit coupled to the camera system, the processing unit configured to: obtain locations of features for an image of the environment, wherein the locations of the features are identified by a neural network; determine a region of interest for one of the features in the image, the region of interest having a size that is less than a size of the image; and perform a corner detection using a corner detection algorithm to identify a corner in the region of interest.

Abstract: An apparatus configured to be head-worn by a user, includes: a screen configured to present graphics for the user; a camera system configured to view an environment in which the user is located; and a processing unit coupled to the camera system, the processing unit configured to: obtain locations of features for an image of the environment, wherein the locations of the features are identified by a neural network; determine a region of interest for one of the features in the image, the region of interest having a size that is less than a size of the image; and perform a corner detection using a corner detection algorithm to identify a corner in the region of interest.

Abstract: Systems and methods for predicting head pose for a rendering engine of an augmented or virtual reality device can include a recurrent neural network (RNN) that accepts a time series of head pose data and outputs a predicted head pose. The recurrent neural network can include one or more long short term memory (LSTM) units or gated recurrent units (GRUs). A fully connected (FC) layer can accept input from the RNN and output a 3 degree-of-freedom (DOF) head pose (e.g., angular orientation or spatial position) or a 6 DOF head pose (e.g., both angular orientation and spatial position). The rendering engine can use the predicted head pose to generate and display virtual content to the user at the time the user looks toward the position of the virtual content, which reduces system latency and improves user experience.

Abstract: Systems and methods for predicting head pose for a rendering engine of an augmented or virtual reality device can include a recurrent neural network (RNN) that accepts a time series of head pose data and outputs a predicted head pose. The recurrent neural network can include one or more long short term memory (LSTM) units or gated recurrent units (GRUs). A fully connected (FC) layer can accept input from the RNN and output a 3 degree-of-freedom (DOF) head pose (e.g., angular orientation or spatial position) or a 6 DOF head pose (e.g., both angular orientation and spatial position). The rendering engine can use the predicted head pose to generate and display virtual content to the user at the time the user looks toward the position of the virtual content, which reduces system latency and improves user experience.

Abstract: Systems and methods for predicting head pose for a rendering engine of an augmented or virtual reality device can include a recurrent neural network (RNN) that accepts a time series of head pose data and outputs a predicted head pose. The recurrent neural network can include one or more long short term memory (LSTM) units or gated recurrent units (GRUs). A fully connected (FC) layer can accept input from the RNN and output a 3 degree-of-freedom (DOF) head pose (e.g., angular orientation or spatial position) or a 6 DOF head pose (e.g., both angular orientation and spatial position). The rendering engine can use the predicted head pose to generate and display virtual content to the user at the time the user looks toward the position of the virtual content, which reduces system latency and improves user experience.

Abstract: A cross reality system enables any of multiple devices to efficiently and accurately access previously persisted maps of very large scale environments and render virtual content specified in relation to those maps. The cross reality system may quickly determine whether a 2D set of features derived from images acquired with a portable device match a set of 3D features of an environment map and, if so, determine the relative pose of the feature sets. The pose may be used in quickly and accurately localizing the portable device to the environment map. Pairs of features in the 2D and 3D features sets may be identified based on matching feature descriptors and may be scored in a neural network trained to assess the quality of the match. Poses may be identified based on subsets of the matching features weighted towards pairs of features with high quality.

Abstract: Systems and methods for predicting head pose for a rendering engine of an augmented or virtual reality device can include a recurrent neural network (RNN) that accepts a time series of head pose data and outputs a predicted head pose. The recurrent neural network can include one or more long short term memory (LSTM) units or gated recurrent units (GRUs). A fully connected (FC) layer can accept input from the RNN and output a 3 degree-of-freedom (DOF) head pose (e.g., angular orientation or spatial position) or a 6 DOF head pose (e.g., both angular orientation and spatial position). The rendering engine can use the predicted head pose to generate and display virtual content to the user at the time the user looks toward the position of the virtual content, which reduces system latency and improves user experience.

Abstract: An apparatus configured to be head-worn by a user, includes: a screen configured to present graphics for the user; a camera system configured to view an environment in which the user is located; and a processing unit coupled to the camera system, the processing unit configured to: obtain locations of features for an image of the environment, wherein the locations of the features are identified by a neural network; determine a region of interest for one of the features in the image, the region of interest having a size that is less than a size of the image; and perform a corner detection using a corner detection algorithm to identify a corner in the region of interest.