Abstract: An exemplary process obtains sensor data corresponding to a physical environment including one or more physical objects. A physical property of the one or more physical objects is determined based on the sensor data. A presentation mode associated with a knowledge domain is determined. An extended reality environment including a view of the physical environment and a visualization selected based on the determined presentation mode is provided. The visualization includes virtual content associated with the knowledge domain. The virtual content is provided based on display characteristics specified by the presentation mode that depend upon the physical property of the one or more objects.

Abstract: Various implementations disclosed herein include devices, systems, and methods that uses object relationships represented in the object relationship model to adjust the position of objects. For example, an example process may include obtaining a three-dimensional (3D) representation of a physical environment that was generated based on sensor data obtained during a scanning process, detecting positions of a set of objects in the physical environment based on the 3D representation, generating a object relationship model for the 3D representation of the physical environment based on the detected positions of the set of objects, wherein the object relationship model represents the set of objects and relationships between the objects, and determining a refined 3D representation of the physical environment by refining the position of at least one object in the set of objects based on the object relationship model and an alignment rule associated with a relationship in the object relationship model.

Abstract: In one implementation, a method includes: identifying a plurality of plot-effectuators and a plurality of environmental elements within a scene associated with a portion of video content; determining one or more spatial relationships between the plurality of plot-effectuators and the plurality of environmental elements within the scene; synthesizing a representation of the scene based at least in part on the one or more spatial relationships; extracting a plurality of action sequences corresponding to the plurality of plot-effectuators based at least in part on the portion of the video content; and generating a corresponding synthesized reality (SR) reconstruction of the scene by driving a plurality of digital assets, associated with the plurality of plot-effectuators, within the representation of the scene according to the plurality of action sequences.

Abstract: A method is performed at an electronic device with one or more processors and a non-transitory memory. The method includes obtaining a plurality of volumetric regions of a physical environment based on a first representation of the physical environment at a first time. Each of the plurality of volumetric regions includes a corresponding portion of the physical environment. The method includes determining a first feature property based on a query. The method includes identifying a first volumetric region of the first plurality of volumetric regions based on determining that the first volumetric region satisfies a criterion with respect to the first feature property.

Abstract: Systems, methods, and computer readable media to implementing an end-to-end room layout estimation are described. A room layout estimation engine performs feature extraction on an image frame to generate a first set of coefficients for a first room layout class and a second set of coefficients for a second room layout class. Afterwards, the room layout estimation engine generates a first set of planes according to the first set of coefficients and a second set of planes according to the second set of coefficients. The room layout estimation engine generates a first prediction plane according to the first set of planes and a second prediction plane according to the second set of planes. Afterwards, the room layout estimation engine merges the first prediction plane and the second prediction plane to generate a predicted room layout for the room.

Abstract: Various implementations disclosed herein include devices, systems, and methods that perform a video event segmentation process to segment video events that include a living entity interacting with an object. For example, a process may obtain frames of a video depicting a living entity and objects within a three-dimensional (3D) environment. The process may further identify the objects depicted in the frames and identifying an event based on the living entity and the objects. The event may involve the living entity and a subset of the objects. The process may further identify the subset of the objects involved in the event and segment the living entity and the subset of the objects involved in the event in the frames. The segmenting process may include identifying portions of the frames corresponding to the living entity and the subset of the objects.

Abstract: In one implementation, a method of generating a scene graph includes generating the scene graph using an end-to-end scene graph generator comprising an integrated neural network. For example, in various implementations, the method includes obtaining an image representing a plurality of objects. The method includes determining a relationship vector indicating spatial relationships between a particular object of the plurality of objects and others of the plurality of objects. The method includes determining, based on the relationship vector, an object type of the particular object.

Abstract: In one implementation, a method of generating a scene graph includes generating the scene graph using an end-to-end scene graph generator comprising an integrated neural network. For example, in various implementations, the method includes obtaining an image representing a plurality of objects. The method includes determining a relationship vector indicating spatial relationships between a particular object of the plurality of objects and others of the plurality of objects. The method includes determining, based on the relationship vector, an object type of the particular object.

Abstract: Implementations disclosed herein provide systems and methods that determine relationships between objects based on an original semantic mesh of vertices and faces that represent the 3D geometry of a physical environment. Such an original semantic mesh may be generated and used to provide input to a machine learning model that estimates relationships between the objects in the physical environment. For example, the machine learning model may output a graph of nodes and edges indicating that a vase is on top of a table or that a particular instance of a vase, V1, is on top of a particular instance of a table, T1.

Abstract: In one implementation, a method includes: obtaining a user input to view SR content associated with video content; if the video content includes a first scene when the user input was detected: obtaining first SR content for a first time period of the video content associated with the first scene; obtaining a task associated with the first scene; and causing presentation of the first SR content and a first indication of the task associated with the first scene; and if the video content includes a second scene when the user input was detected: obtaining second SR content for a second time period of the video content associated with the second scene; obtaining a task associated with the second scene; and causing presentation of the second SR content and a second indication of the task associated with the second scene.

Abstract: Implementations disclosed herein provide systems and methods that determine relationships between objects based on an original semantic mesh of vertices and faces that represent the 3D geometry of a physical environment. Such an original semantic mesh may be generated and used to provide input to a machine learning model that estimates relationships between the objects in the physical environment. For example, the machine learning model may output a graph of nodes and edges indicating that a vase is on top of a table or that a particular instance of a vase, V1, is on top of a particular instance of a table, T1.

Abstract: In one implementation, a method includes: obtaining a user input to view SR content associated with video content; if the video content includes a first scene when the user input was detected: obtaining first SR content for a first time period of the video content associated with the first scene; obtaining a task associated with the first scene; and causing presentation of the first SR content and a first indication of the task associated with the first scene; and if the video content includes a second scene when the user input was detected: obtaining second SR content for a second time period of the video content associated with the second scene; obtaining a task associated with the second scene; and causing presentation of the second SR content and a second indication of the task associated with the second scene.

Abstract: An exemplary process obtains sensor data for a physical environment, generates a local scene graph for the physical environment based on the sensor data, wherein the local scene graph represents a set of objects and relationships between the objects, matches the local scene graph with a principal scene graph of a set of principal scene graphs, and executes one or more scripted actions involving the objects based on a narrative associated with the matched principal scene graph. In some implementations, the set of principal scene graphs is generated by generating local scene graphs for a plurality of environments, and generating individual scene graphs each representative of local scene graphs.

Abstract: Various implementations disclosed herein include devices, systems, and methods that uses object relationships represented in the scene graph to adjust the position of objects. For example, an example process may include obtaining a three-dimensional (3D) representation of a physical environment that was generated based on sensor data obtained during a scanning process, detecting positions of a set of objects in the physical environment based on the 3D representation, generating a scene graph for the 3D representation of the physical environment based on the detected positions of the set of objects, wherein the scene graph represents the set of objects and relationships between the objects, and determining a refined 3D representation of the physical environment by refining the position of at least one object in the set of objects based on the scene graph and an alignment rule associated with a relationship in the scene graph.

Abstract: Various implementations manage an electronic asset by creating a representation of an electronic asset and its variants. This may be accomplished by identifying variants of an electronic asset, identifying a portion of a feature space associated with the asset and variants, and providing a representation corresponding to that portion of feature space. A fixed function classifier may be used to determine the points in the feature space for the electronic asset and its variants. The set of points produced for an asset and its variants using such a fixed function classifier will be near one another in feature space. Moreover, the area around such points will also represent points for other similar variations of the asset and thus, the portion of the feature space around the points can be considered the area of ownership for the electronic asset, e.g., it defines a boundary of what the creator is asserting is his or her creation.

Abstract: In one implementation, a method of achieving an objective using a scene graph includes identifying a set of transition couples of a plurality of transition couples between sets of spatial relationships of the scene graph, wherein each spatial relationship of a respective first set of spatial relationships indicated by each transition couple of the set of transition couples is included in an initial set of spatial relationships or an respective second set of spatial relationships indicated by a previous transition couple and wherein a particular spatial relationship is included in a respective second set of spatial relationships of a last transition couple of the set of transition couples.

Abstract: Implementations disclosed herein provide systems and methods that match a current relationship model associated with a user's current environment to a prior relationship model for a prior environment to determine that the user is in the same environment. The current relationship model is compared with the prior relationship model based on matching characteristics of the current relationship model with characteristics of the prior relationship model.

Abstract: In an exemplary process, a speech input including a referenced virtual object is received. Based on the speech input, a first reference set is obtained. The first reference set is then compared to a plurality of second reference sets. Based on the comparison, a second reference set from the plurality of second reference sets is obtained. The second reference set may be identified based on a matching score between the first reference set and the second reference set. An object is then identified based on the second reference set. Based on the identified object, the reference virtual object is displayed.

Abstract: In one implementation, a method of generating a plane hypothesis is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes obtaining an image of a scene including a plurality of pixels. The method includes obtaining a plurality of points of a point cloud based on the image of the scene. The method includes obtaining an object classification set based on the image of the scene. Each element of the object classification set includes a plurality of pixels respectively associated with a corresponding object in the scene. The method includes detecting a plane within the scene by identifying a subset of the plurality of points of the point cloud that correspond to a particular element of the object classification set.

Abstract: Location mapping and navigation user interfaces may be generated and presented via mobile computing devices. A mobile device may detect its location and orientation using internal systems, and may capture image data using a device camera. The mobile device also may retrieve map information from a map server corresponding to the current location of the device. Using the image data captured at the device, the current location data, and the corresponding local map information, the mobile device may determine or update a current orientation reading for the device. Location errors and updated location data also may be determined for the device, and a map user interface may be generated and displayed on the mobile device using the updated device orientation and/or location data.