Abstract: Devices, systems, and methods that provide a view of a three-dimensional (3D) environment (e.g., a viewer's room) with a portal providing views of another user's (e.g., a sender's) background environment during a communication session. For example, a process at a first electronic device may include presenting a view of a first 3D environment. Data representing a second 3D environment based at least in part on sensor data captured in a physical environment of a second electronic device may be obtained. Portal content based on the data representing the second 3D environment and a viewpoint within the first 3D environment may be determined. A portal with the portal content may be displayed in the view of the first 3D environment, where the portal content depicts a portion of the second 3D environment viewed through the portal from the viewpoint.

Abstract: A method includes receiving a signal that indicates a location of a control device that is configured to change an operating state of a controlled device. The method also includes identifying a first visible device and a second visible device in one or more images, matching the first visible device with the control device based on a location of the first visible device matching the location of the control device, matching the second visible device with the controlled device, pairing the control device with a host device, and controlling the control device using the host device to change the operating state of the controlled device.

Abstract: A method includes receiving one or more signals that each indicate a device type for a respective remote device, identifying one or more visible devices in one or more images, matching a first device from the one or more visible devices with a first signal from the one or more signals based on a device type of the first device matching a device type for the first signal and based on a visible output of the first device, pairing the first device with a second device, and controlling a function of the first device using the second device.

Abstract: Implementations of the subject technology provide application-free extended reality systems and methods. In an application-free extended reality environment, data may be visualized for a user using a data visualization utility that is provided by an operating system of a computing device. The operating system can also provide, separately from the data visualization utility, interaction tools for interacting data that is displayed in the mixed reality environment by the data visualization utility.

Abstract: A device for providing operating system managed group communication sessions may include a memory and at least one processor. The at least one processor may be configured to receive, by an operating system level process executing on a device and from an application process executing on a device, a request to initiate a group session between a user associated with the device and another user. The at least one processor may be further configured to identify, by the operating system level process, another device associated with the other user. The at least one processor may be further configured to initiate, by the operating system level process, the group session with the user via the other device. The at least one processor may be further configured to manage, by the operating system level process, the group session.

Abstract: Various implementations disclosed herein include devices, systems, and methods that track a movement of an input device. For example, an example process may include determine a pose of a tracking device in a physical environment based on first sensor data from an image sensor. The process then may receive, from the tracking device, first positional data corresponding to a first relative positioning between the tracking device and an input device in the physical environment, where the first positional data is determined based on second sensor data obtained via a sensor on the tracking device. The process then may track movement of the input device in the physical environment based at least in part on the first positional data and the pose of the tracking device. The process then may determine an input for the electronic device based at least in part on tracking the movement of the input device.

Abstract: Various implementations disclosed herein include devices, systems, and methods that provide color visual markers that include colored markings that encode data, where the colors of the colored markings are determined by scanning (e.g., detecting the visual marker using a sensor of an electronic device) the visual marker itself. In some implementations, a visual marker is detected in an image of a physical environment. In some implementations, the visual marker is detected in the image by detecting a predefined shape of a first portion of the visual marker in the image. Then, a color-interpretation scheme is determined for interpreting colored markings of the visual marker that encode data by identifying a set of colors at a corresponding set of predetermined locations on the visual marker. Then, the data of the visual marker is decoded using the colored markings and the set of colors of the color-interpretation scheme.

Abstract: A method includes obtaining first pass-through image data characterized by a first pose. The method includes obtaining respective pixel characterization vectors for pixels in the first pass-through image data. The method includes identifying a feature of an object within the first pass-through image data in accordance with a determination that pixel characterization vectors for the feature satisfy a feature confidence threshold. The method includes displaying the first pass-through image data and an AR display marker that corresponds to the feature. The method includes obtaining second pass-through image data characterized by a second pose. The method includes transforming the AR display marker to a position associated with the second pose in order to track the feature. The method includes displaying the second pass-through image data and maintaining display of the AR display marker that corresponds to the feature of the object based on the transformation.

Abstract: A method includes obtaining first pass-through image data characterized by a first pose. The method includes obtaining respective pixel characterization vectors for pixels in the first pass-through image data. The method includes identifying a feature of an object within the first pass-through image data in accordance with a determination that pixel characterization vectors for the feature satisfy a feature confidence threshold. The method includes displaying the first pass-through image data and an AR display marker that corresponds to the feature. The method includes obtaining second pass-through image data characterized by a second pose. The method includes transforming the AR display marker to a position associated with the second pose in order to track the feature. The method includes displaying the second pass-through image data and maintaining display of the AR display marker that corresponds to the feature of the object based on the transformation.

Abstract: Various implementations disclosed herein include devices, systems, and methods that determine how to present a three-dimensional (3D) photo in an extended reality (XR) environment (e.g., in 3D, 2D, blurry, or not at all) based on viewing position of a user active in the XR environment relative to a placement of the 3D photo in the XR environment. In some implementations, at an electronic device having a processor, a 3D photo that is an incomplete 3D representation created based on one or more images captured by an image capture device is obtained. In some implementations, a viewing position of the electronic device relative to a placement position of the 3D photo is determined, and a presentation mode for the 3D photo is determined based on the viewing position. In some implementations, the 3D photo is provided at the placement position based on the presentation mode in the XR environment.

Abstract: Various implementations disclosed herein include devices, systems, and methods that are capable of executing an application on a head-mounted device (HMD) having a first image sensor in a first image sensor configuration. In some implementations, the application is configured for execution on a device including a second image sensor in a second image sensor configuration different than the first image sensor configuration. In some implementations, a request is received from the executing application for image data from the second image sensor. Responsive to the request at the HMD, a pose of a virtual image sensor is determined, image data is generated based on the pose of the virtual image sensor, and the generated image data is provided to the executing application.

Abstract: Some examples of the disclosure are directed to methods for spatial placement of avatars in a communication session. In some examples, while a first electronic device is presenting a three-dimensional environment, the first electronic device may receive an input corresponding to a request to enter a communication session with a second electronic device. In some examples, in response to receiving the input, the first electronic device may scan an environment surrounding the first electronic device. In some examples, the first electronic device may identify a placement location in the three-dimensional environment at which to display a virtual object representing a user of the second electronic device. In some examples, the first electronic device displays the virtual object representing the user of the second electronic device at the placement location in the three-dimensional environment. Some examples of the disclosure are directed to methods for spatial refinement in the communication session.

Abstract: A method includes receiving one or more signals that each indicate a device type for a respective remote device, identifying one or more visible devices in one or more images, matching a first device from the one or more visible devices with a first signal from the one or more signals based on a device type of the first device matching a device type for the first signal and based on a visible output of the first device, pairing the first device with a second device, and controlling a function of the first device using the second device.

Abstract: A method includes obtaining a location and a device type for one or more remote devices, and identifying one or more visible devices in one or more images, the one or more visible devices having a location and a device type. The method also includes matching a first visible device from the one or more visible devices with a first remote device from the one or more remote devices based on a location and a device type of the first visible device matching a location and a device type of the first remote device, obtaining a user input, and controlling a function of the first remote device based on the user input.

Abstract: A method includes obtaining a location and a device type for one or more remote devices, and identifying one or more visible devices in one or more images, the one or more visible devices having a location and a device type. The method also includes matching a first visible device from the one or more visible devices with a first remote device from the one or more remote devices based on a location and a device type of the first visible device matching a location and a device type of the first remote device, obtaining a user input, and controlling a function of the first remote device based on the user input.

Abstract: A device for providing operating system managed group communication sessions may include a memory and at least one processor. The at least one processor may be configured to receive, by an operating system level process executing on a device and from an application process executing on a device, a request to initiate a group session between a user associated with the device and another user. The at least one processor may be further configured to identify, by the operating system level process, another device associated with the other user. The at least one processor may be further configured to initiate, by the operating system level process, the group session with the user via the other device. The at least one processor may be further configured to manage, by the operating system level process, the group session.

Abstract: A method and system control navigation of a 3D CAD model in an augmented reality space. The 3D CAD model is rendered and appears as if the 3D CAD model is present in a physical space at true scale. A constrained translate or rotation mode is activated. In response to the activating, an axis triad is rendered. For translate, the axis triad is three lines extending from a point in each principal axis direction, and division markers are displayed along the lines at predefined division distances. One of the division markers is selected and the model is moved the predefined division distance of the selected division marker along the first line. For rotation, the axis triad is three perpendicular rings, and upon selecting one of the rings, an input gesture causes the model to rotate around the selected axis in the direction of the input gesture.

Abstract: A method includes identifying remote devices, at a host device, based on received signals that indicate locations and device types for the remote devices. The method also includes identifying visible devices in images of a location and matching a first visible device to a first remote device. The first visible device is matched with the first remote device based on presence of the first visible device within a search area of the images, the search area of the images is determined based on the location for the first remote device, the first visible device is matched with the first remote device based on the device type for the first remote device, and the first visible device is matched with the first remote device based on a machine recognizable indicator that is output by the first visible device. The method also includes pairing the first remote device with the host device.

Abstract: Various implementations disclosed herein include devices, systems, and methods that determine the relative positioning (e.g., offset) between a mobile electronic device and a visual marker. In some implementations, the determined relative positioning and a known position of the visual marker are used to determine a position (e.g., geo coordinates) of the mobile electronic device that is more accurate than existing techniques. In some implementations, the determined relative positioning is used with a position of the mobile electronic device to crowd source the stored position of the visual marker. In some implementations, the determined relative positioning and a position of the visual marker are used to determine a position of an object detected in an image by the mobile electronic device. In some implementations at an electronic device having a processor, locally-determined locations of a visual marker are received from mobile electronic devices that scan a visual marker.

Abstract: A method includes identifying remote devices, at a host device, based on received signals that indicate locations and device types for the remote devices. The method also includes identifying visible devices in images of a location and matching a first visible device to a first remote device. The first visible device is matched with the first remote device based on presence of the first visible device within a search area of the images, the search area of the images is determined based on the location for the first remote device, the first visible device is matched with the first remote device based on the device type for the first remote device, and the first visible device is matched with the first remote device based on a machine recognizable indicator that is output by the first visible device. The method also includes pairing the first remote device with the host device.