Abstract: In one implementation, a method of generating a plane hypothesis is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes obtaining an image of a scene including a plurality of pixels. The method includes obtaining a plurality of points of a point cloud based on the image of the scene. The method includes obtaining an object classification set based on the image of the scene. Each element of the object classification set includes a plurality of pixels respectively associated with a corresponding object in the scene. The method includes detecting a plane within the scene by identifying a subset of the plurality of points of the point cloud that correspond to a particular element of the object classification set.

Abstract: In one implementation, a method of remedying a medical impairment of a user is performed by a device including a processor, non-transitory memory, one or more biometric sensors, an image sensor, and a display. The method includes detecting, based on data from at least one of the image sensor and the one or more biometric sensors, a medical impairment of a user of the head-mounted device from a plurality of potential medical impairments associated with a plurality of remedies. The method includes selecting, from the plurality of remedies, a remedy of the medical impairment of the user. The method includes controlling the display to effect the remedy of the medical impairment of the user.

Abstract: Techniques are disclosed herein for implementing a novel, low latency, guidance map-free video matting system, e.g., for use in extended reality (XR) platforms. The techniques may be designed to work with low resolution auxiliary inputs (e.g., binary segmentation masks) and to generate alpha mattes (e.g., alpha mattes configured to segment out any object(s) of interest, such as human hands, from a captured image) in near real-time and in a computationally efficient manner. Further, in a domain-specific setting, the system can function on a captured image stream alone, i.e., it would not require any auxiliary inputs, thereby reducing computational costs—without compromising on visual quality and user comfort. Once an alpha matte has been generated, various alpha-aware graphical processing operations may be performed on the captured images according to the generated alpha mattes (e.g.

Abstract: In some implementations, representations of applications are identified, positioned, and configured in a computer generated reality (CGR) environment based on context. The location at which the representation of the application is positioned may be based on the context of the CGR environment. The context may be determined based on non-image data that is separate from image data of the physical environment being captured for the CGR environment. As examples, the non-image data may relate to the user, a user preferences, a user attribute, a user gesture, motion, activity, or interaction, semantics related to user input or an external source of information, the current time, date, or time period, information from another device involved in the CGR, etc.

Abstract: In one implementation, a method of including a person in a CGR experience or excluding the person from the CGR experience is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes, while presenting a CGR experience, capturing an image of scene; detecting, in the image of the scene, a person; and determining an identity of the person. The method includes determining, based on the identity of the person, whether to include the person in the CGR experience or exclude the person from the CGR experience. The method includes presenting the CGR experience based on the determination.

Abstract: Various implementations disclosed herein include devices, systems, and methods that enable faster and more efficient real-time physical object recognition, information retrieval, and updating of a CGR environment. In some implementations, the CGR environment is provided at a first device based on a classification of the physical object, image or video data including the physical object is transmitted by the first device to a second device, and the CGR environment is updated by the first device based on a response associated with the physical object received from the second device.

Abstract: In one implementation, a method of generating a plane hypothesis is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes obtaining an image of a scene including a plurality of pixels. The method includes obtaining a plurality of points of a point cloud based on the image of the scene. The method includes obtaining an object classification set based on the image of the scene. Each element of the object classification set includes a plurality of pixels respectively associated with a corresponding object in the scene. The method includes detecting a plane within the scene by identifying a subset of the plurality of points of the point cloud that correspond to a particular element of the object classification set.

Abstract: In one implementation, consumption of media content (such as video, audio, or text) is supplemented with an immersive synthesized reality (SR) map based on the media content. In various implementations described herein, the SR map includes a plurality of SR environment representations which, when selected by a user, cause display of a corresponding SR environment.

Abstract: In one implementation, a method of generating a plane hypothesis is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes obtaining an image of a scene including a plurality of pixels. The method includes obtaining a plurality of points of a point cloud based on the image of the scene. The method includes obtaining an object classification set based on the image of the scene. Each element of the object classification set includes a plurality of pixels respectively associated with a corresponding object in the scene. The method includes detecting a plane within the scene by identifying a subset of the plurality of points of the point cloud that correspond to a particular element of the object classification set.

Abstract: In one implementation, consumption of media content (such as video, audio, or text) is supplemented with an immersive synthesized reality (SR) map based on the media content. In various implementations described herein, the SR map includes a plurality of SR environment representations which, when selected by a user, cause display of a corresponding SR environment.

Abstract: In one implementation, consumption of media content (such as video, audio, or text) is supplemented with an immersive synthesized reality (SR) map based on the media content. In various implementations described herein, the SR map includes a plurality of SR environment representations which, when selected by a user, cause display of a corresponding SR environment.

Abstract: In some implementations, a method includes: obtaining a logical representation of a fixed function node; generating, by concerted operation of the logical representation of the fixed function node and a machine learning node that interfaces with the logical representation of the fixed function node, a candidate result based on a set of image data frames; determining whether error criteria are satisfied based at least in part on a comparison between the candidate result and a predetermined result for the set of image data frames; and, in response to determining that the error criteria are satisfied, modifying at least one of: a first portion of operating parameters of the machine learning node associated with operations of the machine learning node; and a second portion of operating parameters of the machine learning node associated with interfacing operations between the machine learning node and the logical representation of the fixed function node.

Abstract: In one implementation, a method of estimating the orientation of an object in an image is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes obtaining an image of a scene including a plurality of pixels at a respective plurality of pixel locations and having a respective plurality of pixel values. The method includes determining a first set of pixels locations corresponding to a 2D boundary surrounding an object represented in the image and determining, based on the first set of pixel locations, a second set of pixel locations corresponding to a 3D boundary surrounding the object.

Abstract: In one implementation, a method of including a person in a CGR experience or excluding the person from the CGR experience is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes, while presenting a CGR experience, capturing an image of scene; detecting, in the image of the scene, a person; and determining an identity of the person. The method includes determining, based on the identity of the person, whether to include the person in the CGR experience or exclude the person from the CGR experience. The method includes presenting the CGR experience based on the determination.

Abstract: In one implementation, a method of including a person in a CGR experience or excluding the person from the CGR experience is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes, while presenting a CGR experience, capturing an image of scene; detecting, in the image of the scene, a person; and determining an identity of the person. The method includes determining, based on the identity of the person, whether to include the person in the CGR experience or exclude the person from the CGR experience. The method includes presenting the CGR experience based on the determination.

Abstract: In one implementation, consumption of media content (such as video, audio, or text) is supplemented with an immersive synthesized reality (SR) map based on the media content. In various implementations described herein, the SR map includes a plurality of SR environment representations which, when selected by a user, cause display of a corresponding SR environment.

Abstract: Various implementations disclosed herein include devices, systems, and methods that enable faster and more efficient real-time physical object recognition, information retrieval, and updating of a CGR environment. In some implementations, the CGR environment is provided at a first device based on a classification of the physical object, image or video data including the physical object is transmitted by the first device to a second device, and the CGR environment is updated by the first device based on a response associated with the physical object received from the second device.

Abstract: In one implementation, a method of generating a plane hypothesis is performed by a device including one or more processors, non-transitory memory, and a scene camera. The method includes obtaining an image of a scene including a plurality of pixels. The method includes obtaining a plurality of points of a point cloud based on the image of the scene. The method includes obtaining an object classification set based on the image of the scene. Each element of the object classification set includes a plurality of pixels respectively associated with a corresponding object in the scene. The method includes detecting a plane within the scene by identifying a subset of the plurality of points of the point cloud that correspond to a particular element of the object classification set.

Abstract: Various implementations disclosed herein include devices, systems, and methods that enable faster and more efficient real-time physical object recognition, information retrieval, and updating of a CGR environment. In some implementations, the CGR environment is provided at a first device based on a classification of the physical object, image or video data including the physical object is transmitted by the first device to a second device, and the CGR environment is updated by the first device based on a response associated with the physical object received from the second device.

Abstract: In one implementation, a method of generating a plane hypothesis is performed by a head-mounted device (HMD) including one or more processors, non-transitory memory, and a scene camera. The method includes obtaining an image of a scene including a plurality of pixels. The method include obtaining a point cloud based on the image of the scene and generating an object classification set based on the image of the scene, each element of the object classification set including a respective plurality of pixels classified as a respective object in the scene. The method includes generating a plane hypothesis based on the point cloud and the object classification set.