Abstract: Aspects of the present disclosure provide a “framework of frameworks” to help developers build artificial reality (XR) applications, including two-dimensional and three-dimensional content, using disparate rendering frameworks. These rendering frameworks can output data to the XR environment, but cannot talk to each other. Thus, some implementations can provide an intermediary framework to coordinate communication and rendering of content between the various systems. The intermediary framework can provide input routing by detecting an event with respect to a piece of content, and routing input data only to the system associated with that piece of content. The intermediary framework can also allow a node within an augment associated with one system to receive notifications of events in another node within the augment associated with another system.

Abstract: A method of coordinating artificial-reality (AR) interactions by presenting augmented-reality interaction guides is provided. The method includes, after receiving a user input requesting an augmented-reality assisted interaction to be directed to a physical surface, presenting, via the AR headset, an augmented-reality interaction guide that is (i) co-located with the physical surface, and (ii) presented with a first orientation relative to the AR headset. The method includes obtaining data indicating that a user interaction has caused a modification to the physical surface. And the method includes, responsive to obtaining additional data, via the AR headset, indicating movement of the physical surface relative to the orientation of the AR headset: presenting the augmented-reality interaction guide to appear at the physical surface with a second orientation relative, different than the first orientation.

Abstract: In some implementations, the disclosed systems and methods can compute the position of the user interface elements in three-dimensional space, project the position into two-dimensional camera space, and dynamically determine the scale of the user interface elements per frame as either static or dynamic based on the distance between the user and the virtual object. In some implementations, the disclosed systems and methods can simulate a three-dimensional (3D) XR experience on the 2D interface (such as a laptop, mobile phone, tablet, etc.), by placing a virtual XR head-mounted display (HMD) into the XR environment, and providing feeds of the field-of view of the virtual XR HMD to the 2D interface.