Abstract: Embodiments include systems and methods for creation of a 3D mesh from a video stream or a sequence of frames. A sparse point cloud is first created from the video stream, which is then densified per frame by comparison with spatially proximate frames. A 3D mesh is then created from the densified depth maps, and the mesh is textured by projecting the images from the video stream or sequence of frames onto the mesh. Metric scale of the depth maps may be estimated where direct measurements are not able to be measured or calculated using a machine learning depth estimation network.

Abstract: Embodiments include systems and methods for determining a 6D pose estimate associated with an image of a physical 3D object captured in a video stream. An initial 6D pose estimate is inferred and then further iteratively refined. The video stream may be frozen to allow the user to tap or touch a display to indicate a location of the user-input keypoints. The resulting 6D pose estimate is used to assist in replacing or superimposing the physical 3D object with digital or virtual content in an augmented reality (AR) frame.

Abstract: Multiple devices may use cameras, sensors, and other inputs to record characteristics of an environment and generate an Augmented Reality (AR) model of the environment. Although devices may implement AR models in different incompatible systems, such as using positioning systems with different scales, unit sizes, etc., one or more devices may coordinate to determine one or more transform to be applied by one or more devices to establish a common framework for referencing AR objects in the models for the environment. With the common framework, and a language to facilitate establishing the common framework, devices may share rich content between the devices. This, for example, allows a tablet device presenting an AR space to “grab” an object out of a display present in the environment, and place the grabbed object into the AR space while maintaining proper relative dimensions and other characteristics of the object after the transfer from one space to the other.