Abstract: Some aspects of the present disclosure are directed to providing partial pass-through on an artificial reality (XR) device, such as a head-mounted display (HMD). Some implementations can allow a user to turn on selective areas of pass-through on the XR device, such that one or more portions of a real-world environment physically surrounding the user can be seen. In some implementations, a user can control the area of pass-through by specifying a percentage of the real-world environment he wishes to see, e.g., 50%. In some implementations, a user can control the area of pass-through by specifying an area of XR device she wishes to see, e.g., the top portion of the view. In some implementations, a user can control the area of pass-through by specifying a physical object in the real-world environment he wishes to see, e.g., a real-world desk.

Abstract: Aspects of the present disclosure can trigger an action based on a motion detected by an artificial reality (XR) device, such as a head-mounted display (HMD). The XR device can display an XR experience to a user. While displaying the XR experience, the XR device can detect a physical interaction with the XR device using one of more sensors (e.g., sensors of an inertial measurement unit (IMU)). The physical interaction can generate a movement profile captured by the one or more sensors. The XR device can identify the physical interaction as a particular motion (e.g., one or more taps on the XR device) by applying a machine learning model to the movement profile. In response to identifying the particular motion, the XR device can trigger an action on the XR device (e.g., activating pass-through on the XR device).

Abstract: Instead of having a constant boundary, artificial reality (XR) applications that can run in boundaryless mode (e.g., in augmented or mixed reality) can be given controls by selective boundary system on an XR system to customize partial boundaries. For example, an application can specify to select particular boundaries when certain object types are in the real-world environment. In another example, an application can transition to different boundary modes when certain application events occur (e.g., when transitioning from virtual reality to mixed reality). On the backend, the selective boundary system can provide the application with an API that exposes boundary information. Based on this information, the application can use the API to create new boundaries or turn on or off certain boundaries based on requests by the application.

Abstract: For artificial reality (XR) applications that need a guardian, aspects of the present disclosure can automatically determine where boundaries should be set in the real-world environment. Using a machine learning model, the automatic boundary system can detect the floor plane, then generate a height map of the real-world environment with the floor at zero. The height map can include positive heights (e.g., objects on the floor) and negative heights (e.g., downward leading stairs). The automatic boundary system can then generate a boundary for the area based on the detected heights by: applying thresholds to disregard objects at certain heights, applying thresholds to disregard open areas at certain widths, excluding oddly shaped areas that would constantly trigger the boundary if the user were nearby, etc. The user can further manually adjust the generated boundary in both directions (e.g., bringing it closer or further from the user).

Abstract: Aspects of the present disclosure can trigger an action based on a motion detected by an artificial reality (XR) device, such as a head-mounted display (HMD). The XR device can display an XR experience to a user. While displaying the XR experience, the XR device can detect a physical interaction with the XR device using one of more sensors (e.g., sensors of an inertial measurement unit (IMU)). The physical interaction can generate a movement profile captured by the one or more sensors. The XR device can identify the physical interaction as a particular motion (e.g., one or more taps on the XR device) by applying a machine learning model to the movement profile. In response to identifying the particular motion, the XR device can trigger an action on the XR device (e.g., activating pass-through on the XR device).