Abstract: Some examples of the disclosure are directed to systems and methods for performing one or more actions in response to detecting a product in a physical environment. For example, performing the one or more actions include presenting one or more virtual elements in an environment. For example, performing the one or more actions include storing information associated with the product. In some examples, the one or more actions are performed in accordance with one or more criteria being satisfied. For example, the one or more criteria include a criterion that is satisfied when one or more indications of user interest in the product are detected. In some examples, one or more virtual elements are presented in an environment indicating a location of a product within a region of the physical environment. The methods of the disclosure are optionally performed at an electronic device including a head-mounted display.

Abstract: A head-mounted device may be used to control one or more external electronic devices. Gaze input and camera images may be used to determine a point of gaze relative to a display for an external electronic device. The external electronic device may receive information regarding the user's gaze input from the head-mounted device and may highlight one out of multiple user interface elements that is targeted by the gaze input. The head-mounted device may receive input such as keystroke information from an accessory device and relay the input to an external electronic device that is being viewed. The head-mounted device may receive a display configuration request, determine layout information for displays in the physical environment of the head-mounted device, and transmit the layout information to an external device associated with the displays.

Abstract: Some examples of the disclosure are directed to systems and methods for minimizing display of an object from a maximized state to a minimized state in a three-dimensional environment. In some examples, an electronic device presents a computer-generated environment that includes a virtual object displayed in a maximized state in the three-dimensional environment. While displaying the three-dimensional environment, the electronic device detects that a first event has occurred. In response to detecting that the first event has occurred, in accordance with a determination that the first event satisfies one or more criteria, the electronic device displays the virtual object in a minimized state in the three-dimensional environment from a viewpoint of a user of the electronic device. In accordance with a determination that the first event does not satisfy the one or more criteria, the electronic device maintains display of the virtual object in the maximized state from the viewpoint.

Abstract: Some examples of the disclosure are directed to systems and methods for capturing and caching one or more first optical captures of an object in a physical environment, and subsequently capturing one or more second optical captures after one or more portions of a user are detected to be directed to the first object. When the one or more portions of the user are determined to satisfy certain criteria (e.g., occluding a first region of the first object), the electronic device performs one or more operations on the one or more first optical captures including recognizing, generating representations of, displaying related information, and/or saving informational content associated with the first object, including informational content occluded by the one or more portions of a user.

Abstract: Some examples of the disclosure are directed to systems and methods for presenting one or more user interface elements including informational content related to information (e.g., textual information) found in an indicated region of the physical environment. In some examples, after confirming that one or more portions of a user satisfy one or more criteria (e.g., one or more portions of a user indicating a region a physical environment including textual information), the electronic device initiates image processing to generate a representation of the region of the physical environment, and optionally provides informational content related to the identified region of the physical environment.

Abstract: Examples of the disclosure are directed to systems and methods for acquiring and transferring content associated with objects that are displayed in a three-dimensional environment. While a computer system displays a three-dimensional environment that includes a first object and a first electronic device, the computer system detects that a user is performing a first gesture directed at the first object. In response to detecting the first gesture directed at the first object, the computer system collects information associated with the first object. The computer system detects the user performing a second gesture directed to a first electronic device (e.g., a laptop or other computing device). In response to detecting the second gesture directed to the first electronic device, the computer system transmits the collected information associated with the first object to the first electronic device.

Abstract: Some examples of the disclosure are directed to systems and methods for presenting one or more user interface elements including second information related to first information found in an indicated region of the physical environment. In some examples, after confirming that the electronic device meets one or more first criteria, the electronic device captures one or more first images of the physical environment, including a first region which includes information. After confirming that one or more second criteria are satisfied (e.g., corresponding to an object-interaction gesture), the electronic device initiates image processing to generate second information related to the first information.

Abstract: Disclosed herein are example processes for providing translations of foreign language content based on context information. For example, in response to receiving language content in a foreign language and in accordance with a determination that the context in which the language content is received satisfies certain criteria, a translation of the language content is provided to a user.

Abstract: Various implementations that use user activity determined via a first device (e.g., an HMD, tablet, etc.) to provide augmentations corresponding to video content (e.g., a TV show, movie, etc.) that is being presented on a second device (e.g., a TV, monitor, etc.). The first device (e.g., HMD) detects that a user activity (e.g., a user's gaze, gesture, interest) is directed to a portion of a surface corresponding to a video content viewing area provided by the second device (e.g., the TV, monitor, etc.). An element of the video content currently being displayed at that portion of the surface by the second device is identified and used to provide an augmentation that is displayed by the first or second device.

Abstract: Some examples of the disclosure are directed to systems and methods for performing one or more actions in response to detecting a product in a physical environment. For example, performing the one or more actions include presenting one or more virtual elements in an environment. For example, performing the one or more actions include storing information associated with the product. In some examples, the one or more actions are performed in accordance with one or more criteria being satisfied. For example, the one or more criteria include a criterion that is satisfied when one or more indications of user interest in the product are detected. In some examples, one or more virtual elements are presented in an environment indicating a location of a product within a region of the physical environment. The methods of the disclosure are optionally performed at an electronic device including a head-mounted display.

Abstract: Methods for transitioning from a first user interface that corresponds to a first session of a first computer system to a second user interface of a second computer system in response to a change in contextual information. In some embodiments, the change in contextual information includes a first change in a relative pose between the second computer system and a reference point corresponding to the first computer system. In some embodiments, the second user interface corresponds to a second session of the first application operating on the second computer system based at least in part on the first session of the first application.

Abstract: A head-mounted device may include a camera with a target depth range. The camera may capture images of physical objects that include text for optical character recognition (OCR). In one arrangement, no images from the camera are presented to the user during OCR. Accordingly, it may not be apparent to the user when the images captured by the camera are blurry, when the physical object is outside the target depth range, or when OCR is not able to be performed with a high accuracy. To allow the user to easily determine when the physical object is outside the target depth range, the head-mounted device may provide audio and/or visual feedback. Presenting visual feedback may include displaying text instructions, adjusting an appearance of a visual indicator that is aligned with the physical object, or displaying a virtual object at a depth that is within the target depth range.

Abstract: A head-mounted device may be used to control one or more external electronic devices. Gaze input and camera images may be used to determine a point of gaze relative to a display for an external electronic device. The external electronic device may receive information regarding the user's gaze input from the head-mounted device and may highlight one out of multiple user interface elements that is targeted by the gaze input. The head-mounted device may receive input such as keystroke information from an accessory device and relay the input to an external electronic device that is being viewed. The head-mounted device may receive a display configuration request, determine layout information for displays in the physical environment of the head-mounted device, and transmit the layout information to an external device associated with the displays.

Abstract: A head-mounted device may be used to perform a visual search on a physical environment around the head-mounted device. A user may wish to visually search one out of multiple physical objects in the physical environment. To clearly show the user which physical object was the target of a visual search, the head-mounted device may present a thumbnail of the candidate physical object on a display. The thumbnail may be cropped and/or zoomed using an image from a camera on the head-mounted device. By displaying a thumbnail that is taken by a camera on the head-mounted device, the thumbnail will directly match the physical object in the user's physical environment, eliminating ambiguity regarding which physical object is the subject of the visual search.

Abstract: A method includes, while presenting a representation of a physical environment of the device, obtaining contextual data that indicates a characteristic of the physical environment, the device or a user of the device, and obtaining an indication of historical content associated with the device. The method includes determining, based on the characteristic indicated by the contextual data and the historical content, a user interest value that indicates a user interest level in the physical environment. The method includes storing an image captured by an image sensor in response to the user interest value being greater than a threshold interest value.

Abstract: A method includes, while presenting a representation of a physical environment of the device, obtaining contextual data that indicates a characteristic of the physical environment, the device or a user of the device, and obtaining an indication of historical content associated with the device. The method includes determining, based on the characteristic indicated by the contextual data and the historical content, a user interest value that indicates a user interest level in the physical environment. The method includes storing an image captured by an image sensor in response to the user interest value being greater than a threshold interest value.

Abstract: Embodiments disclosed herein are directed to devices, systems, and methods for presenting a magnified view in an extended reality environment. Specifically, a magnified view includes a zoom reticle that is presented at a display location of a display. The zoom reticle includes magnified content that includes a magnified portion of a user's field of view. For example, the magnified content may be generated from image data selected from a corresponding portion of a field of view of a camera. The position of the zoom reticle on the display, as well as the portion of the field of view that is magnified, may vary in different circumstances such as described herein.

Abstract: Some examples of the disclosure are directed to systems and methods for performing one or more actions in response to detecting a product in a physical environment. For example, performing the one or more actions include presenting one or more virtual elements in an environment. For example, performing the one or more actions include storing information associated with the product. In some examples, the one or more actions are performed in accordance with one or more criteria being satisfied. For example, the one or more criteria include a criterion that is satisfied when one or more indications of user interest in the product are detected. In some examples, one or more virtual elements are presented in an environment indicating a location of a product within a region of the physical environment. The methods of the disclosure are optionally performed at an electronic device including a head-mounted display.

Abstract: Some examples of the disclosure are directed to systems and methods for presenting one or more user interface elements including suggested content based on detection of physical objects in a computer-generated environment. In some examples, the electronic device captures one or more images of a physical environment, including a first object (e.g., a fillable form). In some examples, in accordance with detecting, in the one or more images, a first region (e.g., a first fillable field) of the first object and that one or more first criteria are satisfied, the electronic device displays a first user interface element including first suggested text, wherein the first user interface element is displayed at a first location based on the first region.

Abstract: Electronic devices, methods, and program storage devices for achieving improved optical character recognition (OCR) operations are disclosed. Performing OCR operations on captured images, e.g., images captured by cameras that are affixed to a user's body (e.g., from mixed reality devices, such as smart HMDs) requires a low-power, robust camera design. Obtaining high spatial resolution in such captured images faces many challenges. However, images with higher spatial resolution can be created by combining information extracted from multiple images captured by such devices, leveraging information obtained from positional sensors of such devices, and performing SR post-processing operations. Such higher spatial resolution images may then be used to enable high-acuity OCR capabilities. The solutions disclosed herein also compensate for the missing ability of such devices due to the lack of a vestibulo-ocular reflex (i.e.