Abstract: Various implementations disclosed herein include devices, systems, and methods that uses object relationships represented in the object relationship model to adjust the position of objects. For example, an example process may include obtaining a three-dimensional (3D) representation of a physical environment that was generated based on sensor data obtained during a scanning process, detecting positions of a set of objects in the physical environment based on the 3D representation, generating a object relationship model for the 3D representation of the physical environment based on the detected positions of the set of objects, wherein the object relationship model represents the set of objects and relationships between the objects, and determining a refined 3D representation of the physical environment by refining the position of at least one object in the set of objects based on the object relationship model and an alignment rule associated with a relationship in the object relationship model.

Abstract: In one implementation, a method of generating a scene graph includes generating the scene graph using an end-to-end scene graph generator comprising an integrated neural network. For example, in various implementations, the method includes obtaining an image representing a plurality of objects. The method includes determining a relationship vector indicating spatial relationships between a particular object of the plurality of objects and others of the plurality of objects. The method includes determining, based on the relationship vector, an object type of the particular object.

Abstract: In one implementation, a method of generating a scene graph includes generating the scene graph using an end-to-end scene graph generator comprising an integrated neural network. For example, in various implementations, the method includes obtaining an image representing a plurality of objects. The method includes determining a relationship vector indicating spatial relationships between a particular object of the plurality of objects and others of the plurality of objects. The method includes determining, based on the relationship vector, an object type of the particular object.

Abstract: Implementations disclosed herein provide systems and methods that determine relationships between objects based on an original semantic mesh of vertices and faces that represent the 3D geometry of a physical environment. Such an original semantic mesh may be generated and used to provide input to a machine learning model that estimates relationships between the objects in the physical environment. For example, the machine learning model may output a graph of nodes and edges indicating that a vase is on top of a table or that a particular instance of a vase, V1, is on top of a particular instance of a table, T1.

Abstract: Implementations disclosed herein provide systems and methods that determine relationships between objects based on an original semantic mesh of vertices and faces that represent the 3D geometry of a physical environment. Such an original semantic mesh may be generated and used to provide input to a machine learning model that estimates relationships between the objects in the physical environment. For example, the machine learning model may output a graph of nodes and edges indicating that a vase is on top of a table or that a particular instance of a vase, V1, is on top of a particular instance of a table, T1.

Abstract: An exemplary process obtains sensor data for a physical environment, generates a local scene graph for the physical environment based on the sensor data, wherein the local scene graph represents a set of objects and relationships between the objects, matches the local scene graph with a principal scene graph of a set of principal scene graphs, and executes one or more scripted actions involving the objects based on a narrative associated with the matched principal scene graph. In some implementations, the set of principal scene graphs is generated by generating local scene graphs for a plurality of environments, and generating individual scene graphs each representative of local scene graphs.

Abstract: Various implementations disclosed herein include devices, systems, and methods that uses object relationships represented in the scene graph to adjust the position of objects. For example, an example process may include obtaining a three-dimensional (3D) representation of a physical environment that was generated based on sensor data obtained during a scanning process, detecting positions of a set of objects in the physical environment based on the 3D representation, generating a scene graph for the 3D representation of the physical environment based on the detected positions of the set of objects, wherein the scene graph represents the set of objects and relationships between the objects, and determining a refined 3D representation of the physical environment by refining the position of at least one object in the set of objects based on the scene graph and an alignment rule associated with a relationship in the scene graph.

Abstract: In one implementation, a method of generating an object model is performed at a device including one or more processors and non-transitory memory. The method includes obtaining a point cloud including a plurality of points, wherein each of the plurality of points is associated with a plurality of sets of coordinates in a three-dimensional space at a respective plurality of time periods. The method includes generating an object model including a plurality of vertices corresponding to the plurality of points and a plurality of edges between respective pairs of the plurality of vertices. The method includes generating, based on the sets of coordinates of a particular point, a characterization vector of a particular vertex corresponding to the particular point, wherein the characterization vector includes a joint label and joint data with respect to one or more of the plurality of edges of the particular vertex.

Abstract: Various implementations disclosed herein include devices, systems, and methods that determine a user location within a physical environment. For example, an example process may include obtaining an object relationship model representing positional relationships between objects of a set of objects within a physical environment. The process may further include obtaining sensor data of the physical environment. The process may further include detecting an interaction of a user with an object of the set of objects within the physical environment based on the sensor data. The process may further include determining a location of the user within the physical environment based on a location associated with the object in the physical environment.

Abstract: In one implementation, a method of achieving an objective using a scene graph includes identifying a set of transition couples of a plurality of transition couples between sets of spatial relationships of the scene graph, wherein each spatial relationship of a respective first set of spatial relationships indicated by each transition couple of the set of transition couples is included in an initial set of spatial relationships or an respective second set of spatial relationships indicated by a previous transition couple and wherein a particular spatial relationship is included in a respective second set of spatial relationships of a last transition couple of the set of transition couples.

Abstract: Implementations disclosed herein provide systems and methods that match a current relationship model associated with a user's current environment to a prior relationship model for a prior environment to determine that the user is in the same environment. The current relationship model is compared with the prior relationship model based on matching characteristics of the current relationship model with characteristics of the prior relationship model.

Abstract: In an exemplary process, a speech input including a referenced virtual object is received. Based on the speech input, a first reference set is obtained. The first reference set is then compared to a plurality of second reference sets. Based on the comparison, a second reference set from the plurality of second reference sets is obtained. The second reference set may be identified based on a matching score between the first reference set and the second reference set. An object is then identified based on the second reference set. Based on the identified object, the reference virtual object is displayed.

Abstract: In one implementation, a method of generating a plane hypothesis is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes obtaining an image of a scene including a plurality of pixels. The method includes obtaining a plurality of points of a point cloud based on the image of the scene. The method includes obtaining an object classification set based on the image of the scene. Each element of the object classification set includes a plurality of pixels respectively associated with a corresponding object in the scene. The method includes detecting a plane within the scene by identifying a subset of the plurality of points of the point cloud that correspond to a particular element of the object classification set.

Abstract: Systems and processes for locating objects and environmental mapping are provided. For example, a device may receive an input including a reference to a first object, wherein the input includes a request for a location of the first object, and the first object is located in a physical environment. The device may determine a location of the first object, and in response to receiving the input, if first criteria are met, provide a description of the location, wherein a criterion of the one or more first criteria is met when the location is available, and the description includes a relationship between the first object and a reference within the physical environment. If second criteria are met, the device may forgo providing the description of the location, wherein a criterion of the one or more second criteria is met when the location is not available.

Abstract: Generating a virtual representation of an interaction includes determining a potential user interaction with a physical object in a physical environment, determining an object type associated with the physical object, and obtaining an object-centric affordance region for the object type, wherein the object-centric affordance region indicates, for each of one or more regions of the object type, a likelihood of user contact. The object-centric affordance region is mapped to a geometry of the physical object to obtain an instance-specific affordance region, is used to render the virtual representation of the interaction with the physical object.

Abstract: Various implementations disclosed herein include devices, systems, and methods that uses object relationships represented in the scene graph to adjust the position of objects. For example, an example process may include obtaining a three-dimensional (3D) representation of a physical environment that was generated based on sensor data obtained during a scanning process, detecting positions of a set of objects in the physical environment based on the 3D representation, generating a scene graph for the 3D representation of the physical environment based on the detected positions of the set of objects, wherein the scene graph represents the set of objects and relationships between the objects, and determining a refined 3D representation of the physical environment by refining the position of at least one object in the set of objects based on the scene graph and an alignment rule associated with a relationship in the scene graph.

Abstract: In one implementation, a method of including a person in a CGR experience or excluding the person from the CGR experience is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes, while presenting a CGR experience, capturing an image of scene; detecting, in the image of the scene, a person; and determining an identity of the person. The method includes determining, based on the identity of the person, whether to include the person in the CGR experience or exclude the person from the CGR experience. The method includes presenting the CGR experience based on the determination.

Abstract: In one implementation, a method of providing a portion of a three-dimensional scene model includes storing, in the non-transitory memory, a three-dimensional scene model of a physical environment including a plurality of points, wherein each of the plurality of points is associated with a set of coordinates in a three-dimensional space, wherein a subset of the plurality of points is associated with a hierarchical data set including a plurality of layers. The method includes receiving, from an objective-effectuator, a request for a portion of the three-dimensional scene model, wherein the portion of the three-dimensional scene model includes less than all of the plurality of points or less than all of the plurality of layers. The method includes obtaining, by the processor from the non-transitory memory, the portion of the three-dimensional scene model. The method includes providing, to the objective-effectuator, the portion of the three-dimensional scene model.

Abstract: Various implementations disclosed herein include devices, systems, and methods that enable faster and more efficient real-time physical object recognition, information retrieval, and updating of a CGR environment. In some implementations, the CGR environment is provided at a first device based on a classification of the physical object, image or video data including the physical object is transmitted by the first device to a second device, and the CGR environment is updated by the first device based on a response associated with the physical object received from the second device.

Abstract: In one implementation, a method of generating a plane hypothesis is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes obtaining an image of a scene including a plurality of pixels. The method includes obtaining a plurality of points of a point cloud based on the image of the scene. The method includes obtaining an object classification set based on the image of the scene. Each element of the object classification set includes a plurality of pixels respectively associated with a corresponding object in the scene. The method includes detecting a plane within the scene by identifying a subset of the plurality of points of the point cloud that correspond to a particular element of the object classification set.