Abstract: A method for sharing a display of a mixed reality experience is described. In one aspect, the method includes pairing a head-wearable device with a handheld device to establish a shared coordinate system, and displaying, based on a pose of the handheld device, a virtual object of a first mixed reality application operating at the head-wearable device, with a second mixed reality application operating at the handheld device using the shared coordinate system.

Abstract: Methods and systems are disclosed for generating an extended reality (XR) try-on experience. The methods and systems store, in a multimodal memory, interaction data representing use of one or more interaction functions including data in different modalities. The methods and systems detect an object depicted in an image captured by an interaction client and generate, by a machine learning model, a prompt based on the object depicted in the image and the interaction data in the multimodal memory. The methods and systems generate an artificial texture based on the prompt and modify a texture of the object depicted in the image using the artificial texture that has been generated based on the prompt.

Abstract: Described is a system for dynamically applying model adaptations customized for individual users by detecting an image of a first real-world object from a camera feed, detecting landmarks on the first real-world object, and processing the landmarks on the first real-world object using a generative machine learning model to generate a first custom image template for the first real-world object where portions of the first custom image template are populated with visual content placed based on the first custom image template. The system then applies a content augmentation based on the first custom image template to the camera feed.

Abstract: Described is a system for overlaying visual content onto a real-world object by identifying a prompt of a user indicating a user's intent, accessing an image template, wherein the image template includes placement of features within the image template, and processing a combination of data associated with the image template and the prompt using a generative machine learning model to generate a first populated image template in which one or more portions of the image template are populated with visual content representing the user's intent. The system then proceeds to access an image depicting a real-world object and overlay the first populated image template that includes the visual content representative of the user's intent on at least a portion of the real-world object based on the placement of the features of the image template.

Abstract: A method for sharing a display of a mixed reality experience is described. In one aspect, the method includes pairing a head-wearable device with a handheld device to establish a shared coordinate system, and displaying, based on a pose of the handheld device, a virtual object of a first mixed reality application operating at the head-wearable device, with a second mixed reality application operating at the handheld device using the shared coordinate system.

Abstract: A case for a portable device like a smartphone includes light sources such as LEDs, which, when illuminated, can be detected and tracked by a head-worn augmented or virtual reality device. The light sources may be located at the corners of the case and may emit infrared light. A relative pose between the smartphone and the head-worn device can be determined based on computer vision techniques performed on images captured by the head-worn device that includes light from the light sources. Relative movement between the smartphone and the head-worn device can be used to provide user input to the head-worn device, as can touch input on the portable device. In some instances, the case is powered inductively from the portable device.

Abstract: The subject technology receives, by one or more hardware processors implementing a local wireless network, a request from a client device to mirror media content displayed on a screen of the client device on a wearable device. In response to the request, the subject technology causes a display of the media content in a mirroring lens of the wearable device. While the media content is being displayed in the mirroring lens of the wearable device, the subject technology tracks hand gestures of a user wearing the wearable device and viewing the media content displayed in the mirroring lens of the wearable device. The subject technology processes navigational or manipulation data based on the tracked hand gestures and sends a navigation or manipulation instruction to the client device or a mirroring lens processor of the wearable device based on the tracked hand gestures.

Abstract: Methods and systems are disclosed for generating an extended reality (XR) try-on experience. The methods and systems store, in a multimodal memory, interaction data representing use of one or more interaction functions including data in different modalities. The methods and systems detect an object depicted in an image captured by an interaction client and generate, by a machine learning model, a prompt based on the object depicted in the image and the interaction data in the multimodal memory. The methods and systems generate an artificial texture based on the prompt and modify a texture of the object depicted in the image using the artificial texture that has been generated based on the prompt.

Abstract: A method for sharing a display of a mixed reality experience is described. In one aspect, the method includes pairing a head-wearable device with a handheld device to establish a shared coordinate system, and displaying, based on a pose of the handheld device, a virtual object of a first mixed reality application operating at the head-wearable device, with a second mixed reality application operating at the handheld device using the shared coordinate system.

Abstract: Described is a system for dynamically applying model adaptations customized for individual users by detecting an image of a first real-world object from a camera feed, detecting landmarks on the first real-world object, and processing the landmarks on the first real-world object using a generative machine learning model to generate a first custom image template for the first real-world object where portions of the first custom image template are populated with visual content placed based on the first custom image template. The system then applies a content augmentation based on the first custom image template to the camera feed.

Abstract: Systems and methods are provided for using an external controller with an AR device. The system establishes, by one or more processors of the AR device, a communication with an external client device. The system overlays, by the AR device, a first AR object on a real-world environment being viewed using the AR device. The system receives interaction data from the external client device representing one or more inputs received by the external client device and, in response, modifies the first AR object by the AR device.

Abstract: A case for a portable device like a smartphone includes light sources such as LEDs, which, when illuminated, can be detected and tracked by a head-worn augmented or virtual reality device. The light sources may be located at the corners of the case and may emit infrared light. A relative pose between the smartphone and the head-worn device can be determined based on computer vision techniques performed on images captured by the head-worn device that includes light from the light sources. Relative movement between the smartphone and the head-worn device can be used to provide user input to the head-worn device, as can touch input on the portable device. In some instances, the case is powered inductively from the portable device.

Abstract: Systems and methods are provided for using an external controller with an AR device. The system establishes, by one or more processors of the AR device, a communication with an external client device. The system overlays, by the AR device, a first AR object on a real-world environment being viewed using the AR device. The system receives interaction data from the external client device representing one or more inputs received by the external client device and, in response, modifies the first AR object by the AR device.

Abstract: Example computer readable storage, extended reality (XR) wearable devices, and methods for remote presence are disclosed where example methods comprise: capturing, by an image capturing device of the XR wearable device, an image corresponding to a first user view of a real-world scene, sending the image to a computing device, receiving, from the computing device, an indication to pause sending next images to the computing device, and determining a plurality of three-dimensional (3D) coordinates corresponding to a plurality of positions within the image. The method may further include sending, to the computing device, the plurality of 3D coordinates corresponding to the plurality of positions within the image, and receiving, from the computing device, an indication of an augmentation and 3D coordinates associated with the augmentation.

Abstract: The present disclosure relates to systems and methods for enhancing the interaction between users and automated agents, such as digital assistants, by employing Large Language Models (LLMs) to infer the intent of spoken language. The invention involves continuously monitoring ambient audio, converting speech to text, and utilizing LLMs to determine whether spoken language is intended for the automated agent. A structured prompt, including the converted text and specific instructions, is sent to the LLM, which is fine-tuned to process domain-specific prompts. The LLM provides a structured output in a standardized format, indicating the user's intent. The system may involve multiple prompts to perform separate tasks, such as identifying intent and generating additional context-specific data. This approach facilitates a more natural and intuitive user experience by eliminating the need for wake words and allowing seamless conversational interaction with virtual assistants across various platforms and devices.

Abstract: The subject technology receives, by one or more hardware processors implementing a local wireless network, a request from a client device to mirror media content displayed on a screen of the client device on a wearable device. In response to the request, the subject technology causes a display of the media content in a mirroring lens of the wearable device. While the media content is being displayed in the mirroring lens of the wearable device, the subject technology tracks hand gestures of a user wearing the wearable device and viewing the media content displayed in the mirroring lens of the wearable device. The subject technology processes navigational or manipulation data based on the tracked hand gestures and sends a navigation or manipulation instruction to the client device or a mirroring lens processor of the wearable device based on the tracked hand gestures.

Abstract: A first mobile device scans an image including fiducial markings, displayed on a second mobile device, and determines a relative pose between the first mobile device and the second mobile device by generating a pose transformation of the fiducial markings displayed by the second mobile device. Information specifying updates to a pose of the second mobile device are received and processed device as user input to the first mobile device. Touch inputs received on the second mobile device may be processed as additional user input by the first mobile device. The touch inputs may be processed to select an augmented reality object displayed by the first mobile device, and the updates to the pose of the second mobile device may be processed to move the augmented reality object displayed by the first mobile device.

Abstract: Systems, methods, and computer readable media for remote annotations, drawings, and navigation instructions sent to an augmented reality (AR) wearable device from a computing device are disclosed. The AR wearable device captures images and sends them to the remote computing device to provide a real-time view of what the user of the AR wearable device sees. A user of the remote computing device can add navigation instructions and can select an image to annotate or draw on. The AR wearable device provides 3-dimensional (3D) coordinate information within a 3D world of the AR wearable device for the selected image. The user of the remote computing device then annotates or draws on the selected image. The remote computing device determines 3D coordinates for the annotations and drawings within the 3D world of the AR wearable device. The annotations and drawings are sent to the AR wearable device with associated 3D coordinates.

Abstract: A communication link is established between a first mobile device and a second mobile device using communication setup information in a machine-readable code that is displayed on a display of the second mobile device. The first mobile device captures and decodes an image of the machine-readable code to extract dynamically-generated communication setup information. A communication link is then established between the two devices using the communication setup information. The machine readable code may also be used as a fiducial marker to establish an initial relative pose between the two devices. Pose updates received from the second mobile device can then be used as user-interface inputs to the first mobile device.

Abstract: The subject technology receives, by one or more hardware processors implementing a local wireless network, a request from a client device to mirror media content displayed on a screen of the client device on a wearable device. In response to the request, the subject technology causes a display of the media content in a mirroring lens of the wearable device. While the media content is being displayed in the mirroring lens of the wearable device, the subject technology tracks hand gestures of a user wearing the wearable device and viewing the media content displayed in the mirroring lens of the wearable device. The subject technology processes navigational or manipulation data based on the tracked hand gestures and sends a navigation or manipulation instruction to the client device or a mirroring lens processor of the wearable device based on the tracked hand gestures.