Abstract: A device and system for visual reasoning in augmented reality environments employs adaptive multi-frame capture triggered by detection of user speech. Upon detecting speech, the device or system captures image frames at an initial frame capture rate, increasing capture frequency when a hand is detected in a captured image. Timestamped frames and transcribed speech form a prompt for a multimodal large language model, which extracts relevant details with constrained output. A separate language model then generates a final response. This two-stage approach optimizes processing efficiency and accuracy while preserving privacy by limiting continuous visual data collection. The system enables more natural and context-aware interactions in AR settings without complex gesture recognition algorithms.

Abstract: A system and method for generating and displaying three-dimensional (3D) content in an augmented reality (AR) environment based on voice input from multiple users. The system includes a server that receives text converted from speech detected at an AR device, generates prompts for language and image generation models, and processes the resulting 2D representation into a 3D model. The 3D model is refined and transmitted to the AR device for presentation. The system incorporates safety checks, supports multi-user interactions, and enables real-time synchronization of 3D content across multiple AR devices in a shared space. This invention integrates voice commands, advanced AI models, and multi-user AR interactions to create an immersive and collaborative 3D content generation experience.

Abstract: In an example, two different mechanisms for specifying a real-world object visible in a view are provided. Each of these different mechanisms has their own benefits, and indeed in another example a hybrid of the two mechanisms may be used, where a user can seamlessly switch between mechanisms based on their own desires or scenarios.

Abstract: In an example a unique user interface element and supporting data structure is introduced. More specifically, each content item (such as a search result) to be displayed is put in its own user interface element called a container. Each container displays the content item in its original format. Multiple containers are then put in another user interface element called an organizer. The organizer acts as a visual representation of the top search results in a single area of the view. A user is able to select on and interact with content items from within the organizer (such as selecting a paused video to play it from within the organizer), but is also able to drag the container from inside the organizer to outside the organizer, which creates a copy of the container to be viewed outside the organizer.

Abstract: A head-wearable device may provide an eXtended Reality (XR) experience that allows the user to control certain aspects of the experience with their hands. Specifically, one of the user's hands will be delineated as the dominant hand, while the other hand will be delineated as the non-dominant hand. Gestures made with the dominant hand will be used to interact with the head-wearable device to perform “primary” actions within the XR experience. These primary actions may vary from experience to experience but generally will involve actions taken to link the real-world environment to the head-wearable device, such as touching or gesturing towards real-world objects, and to perform main actions within a user interface displayed by the head-wearable device, such as selecting on search results, moving search results, etc. Enabling or disabling a microphone will be performed using the non-dominant hand, freeing up the dominant hand to perform the primary actions.

Abstract: An extended Reality (XR) system is provided that generates a dynamic XR user interface having a variety of user input modalities and types of XR user interfaces. The XR system provides a body-centric XR user interface on a hand of the user including a first interactive virtual object located on the hand. The XR system detects a first selection of the first interactive virtual object and provides a near-field XR user interface including a second interactive virtual object. The XR system detects a second selection of the second interactive virtual object and configures the near-field XR user interface to capture a user input. The XR user interface, captures the user input using the near-field XR user interface, generates content for a far-field XR user interface, provides the far-field XR user interface to the user, and displays the content to the user using the far-field XR user interface.

Abstract: An extended Reality (XR) system is provided that generates a dynamic XR user interface having a variety of user input modalities and types of XR user interfaces. The XR system provides a body-centric XR user interface on a hand of the user including a first interactive virtual object located on the hand. The XR system detects a first selection of the first interactive virtual object and provides a near-field XR user interface including a second interactive virtual object. The XR system detects a second selection of the second interactive virtual object and configures the near-field XR user interface to capture a user input. The XR user interface, captures the user input using the near-field XR user interface, generates content for a far-field XR user interface, provides the far-field XR user interface to the user, and displays the content to the user using the far-field XR user interface.

Abstract: A system and method for contextual three-dimensional messaging in augmented reality (AR) environments is disclosed. The system receives chat messages with specified real-world destinations and stores them associated with those locations. When a user wearing an AR device enters a destination location, the system detects their presence using techniques like GPS, Wi-Fi positioning, or computer vision. It then generates a 3D visual representation of the message and determines an appropriate spatial position within the physical environment based on environmental analysis and object detection. The 3D message is displayed at the determined position in the AR view. The system can analyze message content to identify topics and match them to detected real-world objects for contextual placement. Users can interact with displayed messages through gestures or voice commands to reply, forward, delete, or reposition messages.

Abstract: A wearable augmented reality device enables rapid, personalized messaging through a body-anchored communication interface. The device captures image data via cameras, detects and tracks a body part of the wearer, and displays a user interface anchored to the detected body part. The interface comprises user interface elements representing social connections. Upon detecting selection of an element, the device activates a microphone to record an audio message for delivery to the corresponding social connection(s). The interface may be dynamically updated to reflect changes in relationship attributes, messaging activity, or connection status. User interface elements can be ordered based on social connection scores or communication recency, with additional graphical elements indicating device types or custom avatars. This system facilitates seamless integration of digital communication with the physical world, enhancing social media and instant messaging experiences in mixed reality environments.

Abstract: A device and system for visual reasoning in augmented reality environments employs adaptive multi-frame capture triggered by detection of user speech. Upon detecting speech, the device or system captures image frames at an initial frame capture rate, increasing capture frequency when a hand is detected in a captured image. Timestamped frames and transcribed speech form a prompt for a multimodal large language model, which extracts relevant details with constrained output. A separate language model then generates a final response. This two-stage approach optimizes processing efficiency and accuracy while preserving privacy by limiting continuous visual data collection. The system enables more natural and context-aware interactions in AR settings without complex gesture recognition algorithms.

Abstract: The present disclosure relates to systems and methods for enhancing the interaction between users and automated agents, such as digital assistants, by employing Large Language Models (LLMs) to infer the intent of spoken language. The invention involves continuously monitoring ambient audio, converting speech to text, and utilizing LLMs to determine whether spoken language is intended for the automated agent. A structured prompt, including the converted text and specific instructions, is sent to the LLM, which is fine-tuned to process domain-specific prompts. The LLM provides a structured output in a standardized format, indicating the user's intent. The system may involve multiple prompts to perform separate tasks, such as identifying intent and generating additional context-specific data. This approach facilitates a more natural and intuitive user experience by eliminating the need for wake words and allowing seamless conversational interaction with virtual assistants across various platforms and devices.

Abstract: The disclosed methods and systems dynamically update a multimodal memory. The methods and systems generate a multimodal memory comprising interaction data including data in different modalities and add, at a first point in time, a first element to the multimodal memory representing a first attribute of a real-world object associated with a first set of data corresponding to a first modality. The methods and systems detect, at a second point in time, a second set of data corresponding to a second modality, the second set of data representing a second attribute of the real-world object and, in response, add a second element to the multimodal memory representing the second attribute of the real-world object.