Abstract: Techniques for displaying a virtual object in an enhanced reality setting in accordance with a physical muting mode being active are described. In some examples, a system obtains context data for one or more physical elements in a physical setting, wherein the context data includes first context data and second context data that is different from the first context data. In some examples, in response to obtaining the context data for the one or more physical elements in the physical setting, a system causes display of a virtual object that represents the one or more physical elements using the first context data without using the second context data, in accordance with a determination that a physical muting mode is active.

Abstract: A method includes obtaining first pass-through image data characterized by a first pose. The method includes obtaining respective pixel characterization vectors for pixels in the first pass-through image data. The method includes identifying a feature of an object within the first pass-through image data in accordance with a determination that pixel characterization vectors for the feature satisfy a feature confidence threshold. The method includes displaying the first pass-through image data and an AR display marker that corresponds to the feature. The method includes obtaining second pass-through image data characterized by a second pose. The method includes transforming the AR display marker to a position associated with the second pose in order to track the feature. The method includes displaying the second pass-through image data and maintaining display of the AR display marker that corresponds to the feature of the object based on the transformation.

Abstract: A method may include receiving a communication from a device at an artificial intelligence controller including state information for a software application component running on the device, the state information including information corresponding to at least one potential state change available to the software application component, and metrics associated with at least one end condition, interpreting the state information using the artificial intelligence controller, and selecting an artificial intelligence algorithm from a plurality of artificial intelligence algorithms for use by the software application component based on the interpreted state information; and transmitting, to the device, an artificial intelligence algorithm communication, the artificial intelligence algorithm communication indicating the selected artificial intelligence algorithm for use in the software application component on the device.

Abstract: A method includes displaying a home ER environment characterized by home ER world coordinates, including one or more diorama-view representation of one or more respective ER environments. Each diorama-view representation includes ER objects arranged in a spatial relationship according to corresponding ER world coordinates. In some implementations, in response to detecting an input directed to a first diorama-view representation, the method includes transforming the home ER environment. Transforming the home ER environment includes transforming the spatial relationship between a subset of the ER objects as a function of the home ER world coordinates and corresponding ER world coordinates.

Abstract: A group communications platform facilitates that sharing of an application environment with other users. The platform may receive a request to initiate a group session for a local user and a remote user. An out-of-process network connection with a system communication channel between a local computing device associated with the local user and a remote computing device associated with the remote user may be established for the group session. A system call may be received from a local instance of a first application on the local computing device to transfer local data to a remote instance of the first application on the remote computing device via the out-of-process network connection. The local data may be transferred to the remote instance of the first application on the remote computing device via the out-of-process network connection and the system communication channel.

Abstract: A method includes obtaining first pass-through image data characterized by a first pose. The method includes obtaining respective pixel characterization vectors for pixels in the first pass-through image data. The method includes identifying a feature of an object within the first pass-through image data in accordance with a determination that pixel characterization vectors for the feature satisfy a feature confidence threshold. The method includes displaying the first pass-through image data and an AR display marker that corresponds to the feature. The method includes obtaining second pass-through image data characterized by a second pose. The method includes transforming the AR display marker to a position associated with the second pose in order to track the feature. The method includes displaying the second pass-through image data and maintaining display of the AR display marker that corresponds to the feature of the object based on the transformation.

Abstract: Facilitating collaboration in a multi-user communication session may include, at a first device, providing a toolkit of system-level tools for interacting within the session with a second device. A first user interaction with a first tool is detected which results in a virtual modification to an environment of the first device. In response to the first user interaction, the virtual modification is provided for presentation in a second environment representation of the session corresponding to the second device.

Abstract: Various implementations disclosed herein include devices, systems, and methods for presenting communication data. In various implementations, a first device associated with a first person includes a display, a non-transitory memory and one or more processors coupled with the display and the non-transitory memory. In some implementations, a method includes obtaining communication data associated with a second device corresponding to a second person. In some implementations, the method includes determining whether the first device and the second device are in a shared environment. In some implementations, the method includes in response to determining that the first device and the second device are not in a shared environment, displaying a representation of the second person based on the communication data associated with the second device.

Abstract: Methods and apparatus for spatial audio navigation that may, for example, be implemented by mobile multipurpose devices. A spatial audio navigation system provides navigational information in audio form to direct users to target locations. The system uses directionality of audio played through a binaural audio device to provide navigational cues to the user. A current location, target location, and map information may be input to pathfinding algorithms to determine a real world path between the user’s current location and the target location. The system may then use directional audio played through a headset to guide the user on the path from the current location to the target location. The system may implement one or more of several different spatial audio navigation methods to direct a user when following a path using spatial audio-based cues.

Abstract: Presentation of objects within the multiuser communication session is changed based on a shared or unshared status, or whether a user is interacting with the object. The movement of a user of a first device is monitored within a first physical environment comprising shared objects and unshared objects, where the shared objects are visible to the user and an additional user of a second device in a second physical environment, the first device and the second device are active in a multiuser communication session, and the unshared objects are visible to the user and are not visible to the additional user. An interaction of the user with an unshared object is detected. In accordance with the detection of the interaction of the user with the unshared object, a representation of at least a portion of the unshared object is provided for presentation by the second device.

Abstract: The disclosure pertains to techniques for collaborating in a multi-user communications environment. One such technique includes receiving, at a first communication device, data associated with a multi-user communication session between a first user of the first communication device and a second user of a second communication device, presenting, at the first communication device, a non-extended reality graphical user interface (GUI), the non-extended reality GUI including a non-extended reality representation of a virtual object included in the multi-user communication session and a representation of the second user based on the data associated with the multi-user communication session, and updating, at the first communication device, the non-extended reality GUI to illustrate an interaction between the representation of the second user and the virtual object in response to the data indicating the interaction.

Abstract: A method includes displaying a home ER environment characterized by home ER world coordinates, including one or more diorama-view representation of one or more respective ER environments. Each diorama-view representation includes ER objects arranged in a spatial relationship according to corresponding ER world coordinates. In some implementations, in response to detecting an input directed to a first diorama-view representation, the method includes transforming the home ER environment. Transforming the home ER environment includes transforming the spatial relationship between a subset of the ER objects as a function of the home ER world coordinates and corresponding ER world coordinates.

Abstract: Various implementations disclosed herein more accurately or efficiently determine to which of multiple potential virtual objects user input should be directed in a 3D graphical environment. In some implementations, this involves using a rule that accounts for the types of the virtual objects to which a particular event may correspond. For example, a direction of intent may be identified and a rule used to determine to which of multiple potential virtual objects to associate an event.

Abstract: Avatars may be displayed in a multiuser communication session using various spatial modes. One technique for presenting avatars includes presenting avatars such that an attention direction of the avatar is retargeted to match the intent of the remote user corresponding to the avatar. Another technique for presenting avatars includes a pinned mode in which a spatial relationship between one or more avatars remains displayed in a consistent spatial relationship to a local user regardless of movements of the local user. Another technique for presenting avatars includes providing user-selectable presentation modes between a room scale mode and a stationary mode for presenting a representation of a multiuser communication session.

Abstract: A group communications platform facilitates that sharing of an application environment with other users. The platform may receive a request to initiate a group session for a local user and a remote user. An out-of-process network connection with a system communication channel between a local computing device associated with the local user and a remote computing device associated with the remote user may be established for the group session. A system call may be received from a local instance of a first application on the local computing device to transfer local data to a remote instance of the first application on the remote computing device via the out-of-process network connection. The local data may be transferred to the remote instance of the first application on the remote computing device via the out-of-process network connection and the system communication channel.

Abstract: Various implementations disclosed herein include devices, systems, and methods that are capable of executing an application on a head-mounted device (HMD) having a first image sensor in a first image sensor configuration. In some implementations, the application is configured for execution on a device including a second image sensor in a second image sensor configuration different than the first image sensor configuration. In some implementations, a request is received from the executing application for image data from the second image sensor. Responsive to the request at the HMD, a pose of a virtual image sensor is determined, image data is generated based on the pose of the virtual image sensor, and the generated image data is provided to the executing application.

Abstract: Systems and methods for localization for mobile devices are described. Some implementations may include accessing motion data captured using one or more motion sensors; determining, based on the motion data, a coarse localization, wherein the coarse localization includes a first estimate of position; obtaining one or more feature point maps, wherein the feature point maps are associated with a position of the coarse localization; accessing images captured using one or more image sensors; determining, based on the images, a fine localization pose by localizing into a feature point map of the one or more feature point maps, wherein the fine localization pose includes a second estimate of position and an estimate of orientation; generating, based on the fine localization pose, a virtual object image including a view of a virtual object; and displaying the virtual object image.

Abstract: This disclosure relates to managing a feature point map. The managing can include adding feature points to the feature point map using feature points captured by multiple devices and/or by a single device at different times. The resulting feature point map may be referred to as a global feature point map, storing feature points from multiple feature point maps. The global feature point map may be stored in a global feature point database accessible by a server so that the server may provide the global feature point map to devices requesting feature point maps for particular locations. In such examples, the global feature point map may allow a user to localize in locations without having to scan for feature points in the locations. In this manner, the user can localize in locations in which the user has never previously visited.

Abstract: Some examples of the disclosure are directed to selective display of avatars corresponding to users of electronic devices in a multi-user communication session. In some examples, when immersive content is shared in the communication session, the avatars remain displayed when presenting the content in the three-dimensional environment. In some examples, when perspective-limited immersive content is shared in the communication session, the avatars cease being displayed when presenting the content in the three-dimensional environment. In some examples, when content presented in a full-screen mode is shared in the communication session, the avatars remain displayed when presenting the content in the full-screen mode in the three-dimensional environment. In some examples, when object-bounded content is shared in the communication session, the avatars remain displayed when presenting the object-bounded content in the three-dimensional environment.

Abstract: In one implementation, a method of providing display device sharing and interactivity in simulated reality is performed at a first electronic device including one or more processors and a non-transitory memory. The method includes obtaining a gesture input to a first display device in communication with the first electronic device from a first user, where the first display device includes a first display. The method further includes transmitting a representation of the first display to a second electronic device in response to obtaining the gesture input. The method additionally includes receiving an input message directed to the first display device from the second electronic device, where the input message includes an input directive obtained by the second electronic device from a second user. The method also includes transmitting the input message to the first display device for execution by the first display device.