Abstract: Various techniques pertaining to methods, systems, and computer program products a spatial persistence process that places a virtual object relative to a physical object for an extended-reality display device based at least in part upon a persistent coordinate frame (PCF). A determination is made to decide whether a drift is detected for the virtual object relative to the physical object. upon or after detection of the drift or deviation, the drift or deviation is corrected at least by updating a tracking map into an updated tracking map and further at least by updating the persistent coordinate frame (PCF) based at least in part upon the updated tracking map, wherein the persistent coordinate frame (PCF) comprises six degrees of freedom relative to the map coordinate system.

Abstract: A method of efficiently and accurately computing a pose of an image with respect to other image information. The image may be acquired with a camera on a portable device and the other information may be a map, such that the computation of pose localizes the device relative to the map. Such a technique may be applied in a cross reality system to enable devices to efficiently and accurately access previously persisted maps. Localizing with respect to a map may enable multiple cross reality devices to render virtual content at locations specified in relation to those maps, providing an enhanced experience for uses of the system. The method may be used in other devices and for other purposes, such as for navigation of autonomous vehicles.

Abstract: Various techniques pertaining to methods, systems, and computer program products a spatial persistence process that places a virtual object relative to a physical object for an extended-reality display device based at least in part upon a persistent coordinate frame (PCF). A determination is made to decide whether a drift is detected for the virtual object relative to the physical object. upon or after detection of the drift or deviation, the drift or deviation is corrected at least by updating a tracking map into an updated tracking map and further at least by updating the persistent coordinate frame (PCF) based at least in part upon the updated tracking map, wherein the persistent coordinate frame (PCF) comprises six degrees of freedom relative to the map coordinate system.

Abstract: A method of efficiently and accurately computing a pose of an image with respect to other image information. The image may be acquired with a camera on a portable device and the other information may be a map, such that the computation of pose localizes the device relative to the map. Such a technique may be applied in a cross reality system to enable devices to efficiently and accurately access previously persisted maps. Localizing with respect to a map may enable multiple cross reality devices to render virtual content at locations specified in relation to those maps, providing an enhanced experience for uses of the system. The method may be used in other devices and for other purposes, such as for navigation of autonomous vehicles.

Abstract: Various techniques pertaining to methods, systems, and computer program products identify, for a first portable device at a first location in a three-dimensional (3D) physical environment, a first sparse map, the first portable device connected to a remote computing system for accessing sharable world data of the 3D physical environment that comprises a physical object, wherein the plurality electronic devices include the plurality of portable devices having respective sparse maps installed thereupon. A second sparse map is identified for a second portable device connected to the remote computing system for accessing the sharable world data. The second portable device performs content persistence for a virtual content relative to the physical object at least by using a merged map determined by merging a version of the first sparse map and a version of the second sparse map based at least in part upon one or more persistence coordinate frames.

Abstract: A method to culling parts of a 3D reconstruction volume is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data with low usage of computational resources and storage spaces. The method includes culling parts of the 3D reconstruction volume against a depth image. The depth image has a plurality of pixels, each of which represents a distance to a surface in a scene. In some embodiments, the method includes culling parts of the 3D reconstruction volume against a frustum. The frustum is derived from a field of view of an image sensor, from which image data to create the 3D reconstruction is obtained.

Abstract: A method to efficiently update and manage outputs of real time or offline 3D reconstruction and scanning in a mobile device having limited resource and connection to the Internet is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data, in either single user applications or multi-user applications sharing and updating the same 3D reconstruction data. The method includes a block-based 3D data representation that allows local update and maintains neighbor consistency at the same time, and a multi-layer caching mechanism that retrieves, prefetches, and stores 3D data efficiently for XR applications. Between sessions of an XR device, blocks may be persisted on the device or in remote storage in one or more cache layers. The device may, upon starting a new session, selectively use the blocks from one or more layers of the cache.

Abstract: A method to culling parts of a 3D reconstruction volume is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data with low usage of computational resources and storage spaces. The method includes culling parts of the 3D reconstruction volume against a depth image. The depth image has a plurality of pixels, each of which represents a distance to a surface in a scene. In some embodiments, the method includes culling parts of the 3D reconstruction volume against a frustum. The frustum is derived from a field of view of an image sensor, from which image data to create the 3D reconstruction is obtained.

Abstract: Various techniques pertaining to methods, systems, and computer program products identify, for a first portable device at a first location in a three-dimensional (3D) physical environment, a first sparse map, the first portable device connected to a remote computing system for accessing sharable world data of the 3D physical environment that comprises a physical object, wherein the plurality electronic devices include the plurality of portable devices having respective sparse maps installed thereupon. A second sparse map is identified for a second portable device connected to the remote computing system for accessing the sharable world data. The second portable device performs content persistence for a virtual content relative to the physical object at least by using a merged map determined by merging a version of the first sparse map and a version of the second sparse map based at least in part upon one or more persistence coordinate frames.

Abstract: An extended or cross reality system includes a computing device communicably connected to a plurality of portable electronic devices via a network component, a repository accessible by the computing device and the plurality of portable electronic devices, and a dense map merge component. The extended or cross reality system determines a representation for multiple portions of a 3D environment based at least in part upon on a set of dense maps received from the plurality of portable devices, wherein the set of dense maps is grouped into multiple subgroups based at least in part upon pose data pertaining to the set of dense maps or surface information in the set of dense maps. The extended or cross reality system storing the representation as at least a portion of a shared persistent dense map.

Abstract: A method to efficiently update and manage outputs of real time or offline 3D reconstruction and scanning in a mobile device having limited resource and connection to the Internet is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data, in either single user applications or multi-user applications sharing and updating the same 3D reconstruction data. The method includes a block-based 3D data representation that allows local update and maintains neighbor consistency at the same time, and a multi-layer caching mechanism that retrieves, prefetches, and stores 3D data efficiently for XR applications. Between sessions of an XR device, blocks may be persisted on the device or in remote storage in one or more cache layers. The device may, upon starting a new session, selectively use the blocks from one or more layers of the cache.

Abstract: A method to reconstruct an environment with depth information is provided. The method makes available to a wide variety of XR applications fresh and accurate 3D reconstruction data of environments with low processing time and low usage of computational resources and storage spaces. The method includes capturing depth information about an environment upon a change within a field-of-view of a user caused by, for example, motions of headpose, user location, and/or objects, and updating a representation of the environment based on depths associated with different confidence levels by the captured depth information. The method may include updating a geometry of the environment based on depths associated with confidence levels higher than a threshold. The method may include, upon a change of the 3D environment, inferring the removal of surfaces in the environment based on depths associated with confidence levels lower than the threshold.

Abstract: An augmented reality/mixed reality system that provides a more immersive user experience. That experience is provided with increased speed of update for occlusion data by using depth sensor data augmented with lower-level reconstruction data. When operating in real-time dynamic environments, changes in the physical world can be reflected quickly in the occlusion data. Occlusion rendering using live depth data augmented with lower-level 3D reconstruction data, such as a raycast point cloud, can greatly reduce the latency for visual occlusion processing. Generating occlusion data in this way may provide faster operation of an XR system using less computing resources and enabling the system to be packaged in a battery operated wearable device.

Abstract: A method to efficiently update and manage outputs of real time or offline 3D reconstruction and scanning in a mobile device having limited resource and connection to the Internet is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data, in either single user applications or multi-user applications sharing and updating the same 3D reconstruction data. The method includes a block-based 3D data representation that allows local update and maintains neighbor consistency at the same time, and a multi-layer caching mechanism that retrieves, prefetches, and stores 3D data efficiently for XR applications.

Abstract: A method to efficiently update and manage outputs of real time or offline 3D reconstruction and scanning in a mobile device having limited resource and connection to the Internet is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data, in either single user applications or multi-user applications sharing and updating the same 3D reconstruction data. The method includes a block-based 3D data representation that allows local update and maintains neighbor consistency at the same time, and a multi-layer caching mechanism that retrieves, prefetches, and stores 3D data efficiently for XR applications. Between sessions of an XR device, blocks may be persisted on the device or in remote storage in one or more cache layers. The device may, upon starting a new session, selectively use the blocks from one or more layers of the cache.

Abstract: A method of operating a computing system to generate a model of an environment represented by a mesh is provided. The method allows to update 3D meshes to client applications in real time with low latency to support on the fly environment changes. The method provides 3D meshes adaptive to different levels of simplification requested by various client applications. The method provides local update, for example, updating the mesh parts that are changed since last update. The method also provides 3D meshes with planarized surfaces to support robust physics simulations. The method includes segmenting a 3D mesh into mesh blocks. The method also includes performing a multi-stage simplification on selected mesh blocks. The multi-stage simplification includes a pre-simplification operation, a planarization operation, and a post-simplification operation.

Abstract: Methods and apparatus for providing a representation of an environment, for example, in an XR system, and any suitable computer vision and robotics applications. A representation of an environment may include one or more planar features. The representation of the environment may be provided by jointly optimizing plane parameters of the planar features and sensor poses that the planar features are observed at. The joint optimization may be based on a reduced matrix and a reduced residual vector in lieu of the Jacobian matrix and the original residual vector.

Abstract: A method of efficiently and accurately computing a pose of an image with respect to other image information. The image may be acquired with a camera on a portable device and the other information may be a map, such that the computation of pose localizes the device relative to the map. Such a technique may be applied in a cross reality system to enable devices to efficiently and accurately access previously persisted maps. Localizing with respect to a map may enable multiple cross reality devices to render virtual content at locations specified in relation to those maps, providing an enhanced experience for uses of the system. The method may be used in other devices and for other purposes, such as for navigation of autonomous vehicles.

Abstract: An extended or cross reality system includes a computing device communicably connected to a plurality of portable electronic devices via a network component, a repository accessible by the computing device and the plurality of portable electronic devices, and a dense map merge component. The extended or cross reality system determines a representation for multiple portions of a 3D environment based at least in part upon on a set of dense maps received from the plurality of portable devices, wherein the set of dense maps is grouped into multiple subgroups based at least in part upon pose data pertaining to the set of dense maps or surface information in the set of dense maps. The extended or cross reality system storing the representation as at least a portion of a shared persistent dense map.

Abstract: A method to culling parts of a 3D reconstruction volume is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data with low usage of computational resources and storage spaces. The method includes culling parts of the 3D reconstruction volume against a depth image. The depth image has a plurality of pixels, each of which represents a distance to a surface in a scene. In some embodiments, the method includes culling parts of the 3D reconstruction volume against a frustum. The frustum is derived from a field of view of an image sensor, from which image data to create the 3D reconstruction is obtained.