Abstract: An artificial reality environment (XRE) schema is defined that supports anel-controlling interactions between various artificial reality actors. The XRE schema includes a set of definitions for an XRE, independent of type of artificial reality device. The definitions in the XRE schema can include standards for both interfaces and data objects. The XRE schema can define XR elements in terms of entities and components of a space, organized according to a hierarchy. Each entity can represent a real or virtual object or space, within the XRE, defined by a name and a collection of one or more components. Each component (as part of an entity) can define aspects and expose information about the entity. The XRE schema can specify structures that that allow actors (e.g., producers, instantiators, and consumers) to define and perform actions in relation to XRE elements.

Abstract: In particular embodiments, a computing system may capture access a set of depth measurements and an image of a scene generated using one or more sensors of an artificial reality device. The system may generate, based on the image, a plurality of segmentation masks respectively associated with a plurality of object types. The system may segment, using the plurality of segmentation masks, the set of depth measurements into subsets of depth measurements respectively associated with the plurality of object types. The system may determine, for each object type, a three-dimensional (3D) model that best fits the subset of depth measurements corresponding to the object type. The system may refine, using 3D models determined for the plurality of object types, the subsets of depth measurements respectively associated with the plurality of object types and use refined depth measurements for mixed reality rendering.

Abstract: In one embodiment, a method by a computing system comprising a color camera and two monochrome cameras respectively associated with two eyes of a user includes computing a point cloud corresponding to a visible environment based at least on two stereoscopic grayscale images respectively captured by the two monochrome cameras, generating a mesh corresponding to the visible environment based on the computed point cloud, and generating two stereoscopic colorized images to be respectively displayed to the two eyes of the user, where each of the two stereoscopic colorized images is generated using (1) the mesh, (2) luminance information from one of the two stereoscopic grayscale images that is associated with the eye to which the stereoscopic colorized image is to be displayed, and (3) color information from a color input image captured by the color camera.

Abstract: Aspects of the present disclosure are directed to defining an artificial reality environment (XRE) and controlling interactions between various artificial reality actors with a defined XRE schema. The XRE schema includes a set of definitions for an XRE, independent of type of artificial reality device. The definitions in the XRE schema can include standards for both interfaces and data objects. The XRE schema can define XR elements in terms of entities and components of a space, organized according to a hierarchy. Each entity can represent a real or virtual object or space, within the XRE, defined by a name and a collection of one or more components. Each component (as part of an entity) can define aspects and expose information about the entity. The XRE schema can specify structures that that allow actors (e.g., producers, instantiators, and consumers) to define and perform actions in relation to XRE elements.

Abstract: In one embodiment for generating passthrough, a computing system may access images of an environment captured by cameras of a device worn by a user. The system may generate, based on the images, depth measurements of objects in the environment. The system may generate a mesh covering a field of view of the user and then update the mesh based on the depth measurements to represent a contour of the objects in the environment. The system may determine a first viewpoint of a first eye of the user and render a first output image based on the first viewpoint and the updated mesh. The system may then display the first output image on a first display of the device, the first display being configured to be viewed by the first eye of the user.