Abstract: Various implementations disclosed herein generate a mesh representing the surfaces in a physical environment. The mesh is generated using multi-resolution voxels based on detected depth information, e.g., from a depth camera. The techniques may use multiple hash tables to store the multi-resolution voxel data. For example, the hash tables may store each voxel's 3D position and a truncated signed distance field (TSDF) value corresponding to each voxels' distance to a nearest surface. Each of the multiple hash tables may include data corresponding to a different level of resolution and those resolutions may depend upon distance/noise or other factors. For example, voxels close to a depth camera may have a finer resolution and smaller size compared to voxels that are further from the depth camera. Techniques disclosed herein may involve using a meshing algorithm that combines multi-resolution voxel information stored in multiple hash tables to generate a single mesh.

Abstract: In one implementation, a method of performing perspective correction is performed by a device including an image sensor, a display, one or more processors, and non-transitory memory. The method includes capturing, using the image sensor, an image of a physical environment. The method includes obtaining a depth map including a plurality of depths respectively associated with a plurality of pixels of the image of the physical environment, wherein the depth map includes, for a particular pixel at a particular pixel location representing a dynamic object in the physical environment, a particular depth corresponding to a distance between the image sensor and a static object in the physical environment behind the dynamic object. The method includes transforming, using the one or more processors, the image of the physical environment based on the depth map. The method includes displaying, on the display, the transformed image.

Abstract: In one implementation, a method of determining a display location is performed by a device including one or more processors and non-transitory memory. The method includes obtaining a camera set of two-dimensional coordinates of a user input object in a physical environment. The method includes obtaining depth information of the physical environment excluding the user input object. The method includes transforming the camera set of two-dimensional coordinates into a display set of two-dimensional coordinates based on the depth information of the physical environment excluding the user input object.

Abstract: In one implementation, a method includes: obtaining a representation for a volumetric region and obtaining SR content with a first set of dimensions; adapting the SR content by modifying one or more dimensions of the SR content from the first set of dimensions to a second set of dimensions based on one or more portions of the representation of the volumetric region; and causing presentation of the adapted SR content with the second set of dimensions via the display device.

Abstract: In one implementation, a method includes: obtaining locality data characterizing objects and relative spatial information of a volumetric region around a user; synthesizing a mesh map of the volumetric region based on the locality data; selecting synthesized reality (SR) content based on the mesh map, wherein the SR content satisfies a dimensional variance threshold relative to one or more portions of the mesh map; compositing at least a portion of the SR content with the mesh map in order to generate composite SR content; and presenting the composite SR content to the user in order to occlude at least a portion of a visual presentation of the volumetric region. In some implementations, the SR content is adapted to fit the one or more portions of the mesh map. In some implementations, the SR content is updated as the user location changes or the user interacts with the SR content.

Abstract: Various implementations disclosed herein generate a mesh representing the surfaces in a physical environment. The mesh is generated using multi-resolution voxels based on detected depth information, e.g., from a depth camera. The techniques may use multiple hash tables to store the multi-resolution voxel data. For example, the hash tables may store each voxel's 3D position and a truncated signed distance field (TSDF) value corresponding to each voxels' distance to a nearest surface. Each of the multiple hash tables may include data corresponding to a different level of resolution and those resolutions may depend upon distance/noise or other factors. For example, voxels close to a depth camera may have a finer resolution and smaller size compared to voxels that are further from the depth camera. Techniques disclosed herein may involve using a meshing algorithm that combines multi-resolution voxel information stored in multiple hash tables to generate a single mesh.

Abstract: In one implementation, a method includes: obtaining locality data characterizing objects and relative spatial information of a volumetric region around a user; synthesizing a mesh map of the volumetric region based on the locality data; selecting synthesized reality (SR) content based on the mesh map, wherein the SR content satisfies a dimensional variance threshold relative to one or more portions of the mesh map; compositing at least a portion of the SR content with the mesh map in order to generate composite SR content; and presenting the composite SR content to the user in order to occlude at least a portion of a visual presentation of the volumetric region. In some implementations, the SR content is adapted to fit the one or more portions of the mesh map. In some implementations, the SR content is updated as the user location changes or the user interacts with the SR content.

Abstract: In one implementation, a method includes: obtaining locality data characterizing objects and relative spatial information of a volumetric region around a user; synthesizing a mesh map of the volumetric region based on the locality data; selecting synthesized reality (SR) content based on the mesh map, wherein the SR content satisfies a dimensional variance threshold relative to one or more portions of the mesh map; compositing at least a portion of the SR content with the mesh map in order to generate composite SR content; and presenting the composite SR content to the user in order to occlude at least a portion of a visual presentation of the volumetric region. In some implementations, the SR content is adapted to fit the one or more portions of the mesh map. In some implementations, the SR content is updated as the user location changes or the user interacts with the SR content.

Abstract: A method for estimating the speed of movement of a first video camera when it captures a current image of a three-dimensional scene, includes storing a reference image corresponding to an image of the same scene captured by a second video camera in a different pose, the reference image including pixels. The method also includes storing the current image, the current image including pixels containing the measurement of a physical quantity measured by that pixel, that physical quantity being the same as the physical quantity measured by the pixels of the reference image. The method further includes storing for each pixel of the reference image or of the current image the measurement of a depth that separates that pixel from the point of the scene photographed by that pixel, estimating the pose of the first camera, and estimating the speed of movement of the first camera.

Abstract: Various implementations disclosed herein generate a mesh representing the surfaces in a physical environment. The mesh is generated using multi-resolution voxels based on detected depth information, e.g., from a depth camera. The techniques may use multiple hash tables to store the multi-resolution voxel data. For example, the hash tables may store each voxel's 3D position and a truncated signed distance field (TSDF) value corresponding to each voxels' distance to a nearest surface. Each of the multiple hash tables may include data corresponding to a different level of resolution and those resolutions may depend upon distance/noise or other factors. For example, voxels close to a depth camera may have a finer resolution and smaller size compared to voxels that are further from the depth camera. Techniques disclosed herein may involve using a meshing algorithm that combines multi-resolution voxel information stored in multiple hash tables to generate a single mesh.

Abstract: Various implementations are disclosed of producing a 3-dimensional model of a scene. Various method, electronic device, or system implementations use RGB-D camera to provide RGB-D video content or periodic aligned RGB images and depth images to localize camera spatial position(s) defined in a three dimensional (3D) coordinate system or reconstruct a 3D virtual representation of a current camera frame in the 3D coordinate system, each in real time.

Abstract: In one implementation, a method includes: obtaining locality data characterizing objects and relative spatial information of a volumetric region around a user; synthesizing a mesh map of the volumetric region based on the locality data; selecting synthesized reality (SR) content based on the mesh map, wherein the SR content satisfies a dimensional variance threshold relative to one or more portions of the mesh map; compositing at least a portion of the SR content with the mesh map in order to generate composite SR content; and presenting the composite SR content to the user in order to occlude at least a portion of a visual presentation of the volumetric region. In some implementations, the SR content is adapted to fit the one or more portions of the mesh map. In some implementations, the SR content is updated as the user location changes or the user interacts with the SR content.

Abstract: A method for estimating the speed a first video camera when it captures a current image of a three-dimensional scene, the current image including pixels. The method includes storing a reference image corresponding to an image of the same scene captured by a second video camera in a different pose, the reference image including pixels. The method also includes storing the current image, containing for each pixel of the current image the measurement of a physical magnitude measured by that pixel, which is the same as the physical magnitude measured by the pixels of the reference image. The method further includes storing for each pixel of the reference image or of the current image the measurement of a depth that separates that pixel from the point of the scene photographed by that pixel, estimating the pose and speed of the first video camera.

Abstract: Various implementations are disclosed of producing a 3-dimensional model of a scene. Various method, electronic device, or system implementations use RGB-D camera to provide RGB-D video content or periodic aligned RGB images and depth images to localize camera spatial position(s) defined in a three dimensional (3D) coordinate system or reconstruct a 3D virtual representation of a current camera frame in the 3D coordinate system, each in real time.

Abstract: A method for estimating the speed of movement of a first video camera when it captures a current image of a three-dimensional scene, includes storing a reference image corresponding to an image of the same scene captured by a second video camera in a different pose, the reference image including pixels. The method also includes storing the current image, the current image including pixels containing the measurement of a physical quantity measured by that pixel, that physical quantity being the same as the physical quantity measured by the pixels of the reference image. The method further includes storing for each pixel of the reference image or of the current image the measurement of a depth that separates that pixel from the point of the scene photographed by that pixel, estimating the pose of the first camera, and estimating the speed of movement of the first camera.

Abstract: A method for estimating the speed a first video camera when it captures a current image of a three-dimensional scene, the current image including pixels. The method includes storing a reference image corresponding to an image of the same scene captured by a second video camera in a different pose, the reference image including pixels. The method also includes storing the current image, containing for each pixel of the current image the measurement of a physical magnitude measured by that pixel, which is the same as the physical magnitude measured by the pixels of the reference image. The method further includes storing for each pixel of the reference image or of the current image the measurement of a depth that separates that pixel from the point of the scene photographed by that pixel, estimating the pose and speed of the first video camera.

Abstract: A method for estimating the speed a first video camera when it captures a current image of a three-dimensional scene, the current image including pixels. The method includes storing a reference image corresponding to an image of the same scene captured by a second video camera in a different pose, the reference image including pixels. The method also includes storing the current image, containing for each pixel of the current image the measurement of a physical magnitude measured by that pixel, which is the same as the physical magnitude measured by the pixels of the reference image. The method further includes storing for each pixel of the reference image or of the current image the measurement of a depth that separates that pixel from the point of the scene photographed by that pixel, estimating the pose and speed of the first video camera.

Abstract: A method for estimating the speed a first video camera when it captures a current image of a three-dimensional scene, the current image including pixels. The method includes storing a reference image corresponding to an image of the same scene captured by a second video camera in a different pose, the reference image including pixels. The method also includes storing the current image, containing for each pixel of the current image the measurement of a physical magnitude measured by that pixel, which is the same as the physical magnitude measured by the pixels of the reference image. The method further includes storing for each pixel of the reference image or of the current image the measurement of a depth that separates that pixel from the point of the scene photographed by that pixel, estimating the pose and speed of the first video camera.

Abstract: This method comprises the estimation of the speed XvR of displacement of a camera by searching for the speed XvR which minimizes a discrepancy directly between: -a first value of a physical quantity at the level of a first point (p*) of a reference image, and -a second value of the same physical quantity at the level of a second point (p v2) of a current image, the first value of the physical quantity at the level of the first point (p*) of the reference image being constructed: -by selecting neighbour points of the first point (p*) as a function of the speed XvR and of a time to equal to the exposure time of the first camera, then -by averaging the values of the physical quantity at the level of the neighbour points selected and of the first point in such a way as to generate a new value of the physical quantity at the level of the first point.