Abstract: A method of interacting with an artificial-reality (AR) content at an AR headset that includes cameras and displays is described. The method includes, while the AR headset has a first position within a physical environment that includes an object, if a distance of the object is within a threshold collision distance from the AR headset, presenting a meshed representation of the object. The meshed representation is displayed at first respective locations on the displays such that the meshed representation is viewable within the artificial reality. After the AR headset moves to a second position with a different distance, and if the different distance is within the threshold collision distance from the AR headset, moving the meshed representation of the object to second respective locations within the artificial reality. The second respective locations correspond to the position of the object in the physical environment.

Abstract: A system generates a plurality of spatial points based on depth measurements of physical objects. The system determines, based on the plurality of spatial points, an occupancy score for each voxel within a plurality of voxels. The system identifies, based on a gaze of the user, a first set of occupied voxels that are in a field of view of the user and a second set of occupied voxels that are outside the field of view of the user. The system updates the occupancy scores of the first set of occupied voxels by temporally decaying one or more of the plurality of spatial points within the first set of occupied voxels. The system maintains the occupancy scores of the second set of occupied voxels. The system detects intrusions in a predefined subspace within a physical space based on the updated occupancy scores of the first set of occupied voxels.

Abstract: In particular embodiments, a computing system may divide at least a portion of a physical space surrounding a user into a plurality of three-dimensional (3D) regions, wherein each of the 3D regions is associated with an area of a plurality of areas in a plane. The system may generate estimated locations of features of objects in the portion of the physical space. Based on the estimated locations, the system may determine an occupancy state of each of the plurality of 3D regions. Then based on the occupancy states of the plurality of 3D regions, the system may determine that one or more of the plurality of areas have respective airspaces that are likely unoccupied by objects.

Abstract: A mixed reality (MR) simulation system includes a console and a head mounted device (HMD). The MR system captures stereoscopic images from a real-world environment using outward-facing stereoscopic cameras mounted to the HMD. The MR system preprocesses the stereoscopic images to maximize contrast and then extracts a set of features from those images, including edges or corners, among others. For each feature, the MR system generates one or more two-dimensional (2D) polylines. Then, the MR system triangulates between 2D polylines found in right side images and corresponding 2D polylines found in left side images to generate a set of 3D polylines. The MR system interpolates between 3D vertices included in the 3D polylines or extrapolates additional 3D vertices, thereby generating a geometric reconstruction of the real-world environment. The MR system may map textures derived from the real-world environment onto the geometric representation faster than the geometric reconstruction is updated.

Abstract: In one embodiment, a method includes segmenting a layout of a physical space surrounding a user into physical segments; generating, based on the physical segments, virtual paths for a virtual environment through which the user can navigate by traveling the physical segments; identifying, based on a current location of the user with respect to the physical space, a portion of the physical segments for which to enable an intrusion detection feature; detecting a physical object in the portion of the physical segments that corresponds to a particular virtual path of the virtual paths; and in response to the detecting, displaying a representation of the physical object in the particular virtual path.

Abstract: A mixed reality (MR) simulation system includes a console and a head mounted device (HMD). The MR system captures stereoscopic images from a real-world environment using outward-facing stereoscopic cameras mounted to the HMD. The MR system preprocesses the stereoscopic images to maximize contrast and then extracts a set of features from those images, including edges or corners, among others. For each feature, the MR system generates one or more two-dimensional (2D) polylines. Then, the MR system triangulates between 2D polylines found in right side images and corresponding 2D polylines found in left side images to generate a set of 3D polylines. The MR system interpolates between 3D vertices included in the 3D polylines or extrapolates additional 3D vertices, thereby generating a geometric reconstruction of the real-world environment. The MR system may map textures derived from the real-world environment onto the geometric representation faster than the geometric reconstruction is updated.

Abstract: In particular embodiments, a computing system may divide at least a portion of a physical space surrounding a user into a plurality of three-dimensional (3D) regions, wherein each of the 3D regions is associated with an area of a plurality of areas in a plane. The system may generate estimated locations of features of objects in the portion of the physical space. Based on the estimated locations, the system may determine an occupancy state of each of the plurality of 3D regions. Then based on the occupancy states of the plurality of 3D regions, the system may determine that one or more of the plurality of areas have respective airspaces that are likely unoccupied by objects.

Abstract: A system generates a plurality of spatial points based on depth measurements of physical objects. The system determines, based on the plurality of spatial points, an occupancy score for each voxel within a plurality of voxels. The system identifies, based on a gaze of the user, a first set of occupied voxels that are in a field of view of the user and a second set of occupied voxels that are outside the field of view of the user. The system updates the occupancy scores of the first set of occupied voxels by temporally decaying one or more of the plurality of spatial points within the first set of occupied voxels. The system maintains the occupancy scores of the second set of occupied voxels. The system detects intrusions in a predefined subspace within a physical space based on the updated occupancy scores of the first set of occupied voxels.

Abstract: In particular embodiments, a computing system may divide at least a portion of a physical space surrounding a user into a plurality of three-dimensional (3D) regions, wherein each of the 3D regions is associated with an area of a plurality of areas in a plane. The system may generate estimated locations of features of objects in the portion of the physical space. Based on the estimated locations, the system may determine an occupancy state of each of the plurality of 3D regions. Then based on the occupancy states of the plurality of 3D regions, the system may determine that one or more of the plurality of areas have respective airspaces that are likely unoccupied by objects.

Abstract: A system generates a plurality of spatial points based on depth measurements of physical objects. The system determines, based on the plurality of spatial points, an occupancy score for each voxel within a plurality of voxels. The system identifies, based on a gaze of the user, a first set of occupied voxels that are in a field of view of the user and a second set of occupied voxels that are outside the field of view of the user. The system updates the occupancy scores of the first set of occupied voxels by temporally decaying one or more of the plurality of spatial points within the first set of occupied voxels. The system maintains the occupancy scores of the second set of occupied voxels. The system detects intrusions in a predefined subspace within a physical space based on the updated occupancy scores of the first set of occupied voxels.

Abstract: In one embodiment for generating passthrough, a computing system may access images of an environment captured by cameras of a device worn by a user. The system may generate, based on the images, depth measurements of objects in the environment. The system may generate a mesh covering a field of view of the user and then update the mesh based on the depth measurements to represent a contour of the objects in the environment. The system may determine a first viewpoint of a first eye of the user and render a first output image based on the first viewpoint and the updated mesh. The system may then display the first output image on a first display of the device, the first display being configured to be viewed by the first eye of the user.

Abstract: In one embodiment, a method includes displaying, through a head-mounted display (HMD), virtual objects to a user wearing the HMD. The method then accesses a boundary definition that corresponds to a boundary within a physical space surrounding the user and generates a plurality of spatial points based on depth measurements of physical objects within the physical space. Based on the spatial points, a location at which a physical object is likely to exist is determined. The method determines whether the location of the physical object is inside the boundary definition and, in response to this determination, issues an alert to the user.

Abstract: A system generates a plurality of spatial points based on depth measurements of physical objects. The system determines, based on the plurality of spatial points, an occupancy score for each voxel within a plurality of voxels. The system identifies, based on a gaze of the user, a first set of occupied voxels that are in a field of view of the user and a second set of occupied voxels that are outside the field of view of the user. The system updates the occupancy scores of the first set of occupied voxels by temporally decaying one or more of the plurality of spatial points within the first set of occupied voxels. The system maintains the occupancy scores of the second set of occupied voxels. The system detects intrusions in a predefined subspace within a physical space based on the updated occupancy scores of the first set of occupied voxels.

Abstract: In one embodiment, a method includes segmenting a layout of a physical space surrounding a user into physical segments; generating, based on the physical segments, virtual paths for a virtual environment through which the user can navigate by traveling the physical segments; identifying, based on a current location of the user with respect to the physical space, a portion of the physical segments for which to enable an intrusion detection feature; detecting a physical object in the portion of the physical segments that corresponds to a particular virtual path of the virtual paths; and in response to the detecting, displaying a representation of the physical object in the particular virtual path.

Abstract: In one embodiment, a method includes generating a plurality of spatial points based on depth measurements of physical objects within a physical space surrounding a user and determining, based on the spatial points, a location at which a physical object is likely to exist. The method then renders, based on the location of the physical object, a virtual space representing the physical space. This virtual space may include a virtual object representing the physical object. The method displays the virtual space to the user, and, while displaying the virtual space, receives input from the user indicating a boundary of a subspace within the virtual space, and detects that at least a portion of the virtual object is within the subspace. Finally, the method updates the virtual space to indicate that the portion of the virtual object is within the subspace.

Abstract: In one embodiment, a method includes displaying, through a head-mounted display (HMD), virtual objects to a user wearing the HMD. The method then accesses a boundary definition that corresponds to a boundary within a physical space surrounding the user and generates a plurality of spatial points based on depth measurements of physical objects within the physical space. Based on the spatial points, a location at which a physical object is likely to exist is determined. The method determines whether the location of the physical object is inside the boundary definition and, in response to this determination, issues an alert to the user.

Abstract: In one embodiment, a method includes generating a plurality of spatial points based on depth measurements of physical objects within a physical space surrounding a user and determining, based on the spatial points, a location at which a physical object is likely to exist. The method then renders, based on the location of the physical object, a virtual space representing the physical space. This virtual space may include a virtual object representing the physical object. The method displays the virtual space to the user, and, while displaying the virtual space, receives input from the user indicating a boundary of a subspace within the virtual space, and detects that at least a portion of the virtual object is within the subspace. Finally, the method updates the virtual space to indicate that the portion of the virtual object is within the subspace.

Abstract: In particular embodiments, a computing system may divide at least a portion of a physical space surrounding a user into a plurality of three-dimensional (3D) regions, wherein each of the 3D regions is associated with an area of a plurality of areas in a plane. The system may generate estimated locations of features of objects in the portion of the physical space. Based on the estimated locations, the system may determine an occupancy state of each of the plurality of 3D regions. Then based on the occupancy states of the plurality of 3D regions, the system may determine that one or more of the plurality of areas have respective airspaces that are likely unoccupied by objects.

Abstract: A mixed reality (MR) simulation system includes a console and a head mounted device (HMD). The MR system captures stereoscopic images from a real-world environment using outward-facing stereoscopic cameras mounted to the HMD. The MR system preprocesses the stereoscopic images to maximize contrast and then extracts a set of features from those images, including edges or corners, among others. For each feature, the MR system generates one or more two-dimensional (2D) polylines. Then, the MR system triangulates between 2D polylines found in right side images and corresponding 2D polylines found in left side images to generate a set of 3D polylines. The MR system interpolates between 3D vertices included in the 3D polylines or extrapolates additional 3D vertices, thereby generating a geometric reconstruction of the real-world environment. The MR system may map textures derived from the real-world environment onto the geometric representation faster than the geometric reconstruction is updated.

Abstract: A mixed reality (MR) simulation system includes a console and a head mounted device (HMD). The MR system captures stereoscopic images from a real-world environment using outward-facing stereoscopic cameras mounted to the HMD. The MR system preprocesses the stereoscopic images to maximize contrast and then extracts a set of features from those images, including edges or corners, among others. For each feature, the MR system generates one or more two-dimensional (2D) polylines. Then, the MR system triangulates between 2D polylines found in right side images and corresponding 2D polylines found in left side images to generate a set of 3D polylines. The MR system interpolates between 3D vertices included in the 3D polylines or extrapolates additional 3D vertices, thereby generating a geometric reconstruction of the real-world environment. The MR system may map textures derived from the real-world environment onto the geometric representation faster than the geometric reconstruction is updated.