Abstract: Eyewear providing an interactive augmented reality experience to users in a first physical environment viewing objects in a second physical environment (e.g., X-ray effect). The second environment may be a room positioned behind a barrier, such as a wall. The user views the second environment via a sensor system moveable on the wall using a track system. As the user in the first environment moves the eyewear to face the outside surface of the wall along a line-of-sight (LOS) at a location (x, y, z), the sensor system on the track system repositions to the same location (x, y, z) on the inside surface of wall. The image captured by the sensor system in the second environment is wirelessly transmitted to the eyewear for displayed on the eyewear displays, providing the user with an X-ray effect of looking through the wall to see the objects within the other environment.

Abstract: Methods and systems are disclosed for performing generating AR experiences on a messaging platform. The methods and systems perform operations including: receiving, from a client device, a request to access an augmented reality (AR) experience; adding one or more AR elements to a first image captured by the client device, the first image depicting a real-world object; storing data representing a position of the one or more AR elements relative to the real-world object, the data being maintained after the AR experience is terminated; receiving a request to resume the AR experience after the AR experience has been terminated; and in response to receiving the request to resume the AR experience, accessing the data that was stored prior to termination of the AR experience to generate a display of the AR experience that depicts the one or more AR elements at a particular position within a second image.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and method for providing permissions for accessing shared content collections. The program and method provide for receiving, from a first device associated with a first user, an indication of first user input to share a content collection between the first user and a second user selected by the first user, the content collection comprising at least one media content item, the second user corresponding to a contact of the first user within a messaging application; storing the content collection in association with the first user and the second user; providing the first user with a first set of permissions for accessing the content collection; and providing the second user with a second set of permissions for accessing the content collection, the second set of permissions being more restrictive than the first set of permissions.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, method, and user interfaces to facilitate communication between users via multiple modes of electronic communication. The system displays a conversation view that presents messages exchanged between a first user and at least one other user in an electronic chat conversation. The conversation view includes an interactive element to enable the first user to initiate a video call with the at least one other user. Based on receiving user input corresponding to an interaction with the interactive element, the system initiates the video call with the at least one other user. The system displays an active call view to facilitate the video call with the at least one other user. The system toggles from displaying the active call view to displaying the conversation view in response to further user input.

Abstract: A notification system is configured to perform operations that include: causing display of a graphical user interface at a client device associated with a user account, the graphical user interface including a menu element to display one or more notifications; accessing a list of user connections associated with the user account based on the causing display of the graphical user interface; determining that a user connection from among the list of user connections is engaged in a game session; and presenting a notification within the menu element in response to the determining that the user connection from among the list of user connections is engaged in a game session, the notification including a display of a user identifier associated with the user connection, and a game title associated with the game session.

Abstract: Methods and systems are disclosed for controlling connected devices. For example, a messaging application implemented on a client device detects a real-world object depicted in a captured image and determines a current location of the client device. The messaging application identifies a plurality of video cameras associated with the current location and selects a first video camera from the plurality of video cameras based on one or more attributes of the real-world object depicted in the image. The messaging application receives a video feed from the first video camera that is selected and causes the video feed received from the first video camera to be displayed on top of the real-world object depicted in the captured image.

Abstract: Eyewear including a voltage controller in the frame that generates dynamic analog control signals to control voltage regulators in the temple. The voltage regulators include a voltage rail for each electronic component in the temple. A separate analog control loop is coupled to each voltage regulator and receives the respective analog control signal. Each voltage regulator generates a rail voltage on the respective voltage rail that is controlled by the respective analog control signal. The analog control loop configures the respective voltage regulator as a voltage follower regulator such that the respective rail voltage follows a voltage of the analog control signal. A power source, such as a battery, is included in the temple and provides the operating power to each electronic component, and power is not communicated across a hinge to the temple electronic components.

Abstract: Systems and methods herein describe a multi-modal interaction system. The multi-modal interaction system, receives a selection of an augmented reality (AR) experience within an application on a computer device, displays a set of AR objects associated with the AR experience on a graphical user interface (GUI) of the computer device, display textual cues associated with the set of augmented reality objects on the GUI, receives a hand gesture and a voice command, modifies a subset of augmented reality objects of the set of augmented reality objects based on the hand gesture and the voice command, and displays the modified subset of augmented reality objects on the GUI.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and method for displaying augmented reality content. The program and method provide for displaying, within a messaging application running on a device, a first user interface for viewing a media content item; receiving user input via the first user interface, the user input corresponding to a predefined touch gesture; and in response to receiving the user input, switching, within the messaging application, from the first user interface to a second user interface in which a camera of the device is activated to capture an image, and displaying, via the second user interface, a resized version of the media content item with the image.

Abstract: Bending data is used to facilitate tracking operations of an extended reality (XR) device, such as hand tracking or other object tracking operations. The XR device obtains bending data indicative of bending of the XR device to accommodate a body part of a user wearing the XR device. The XR device determines, based on the bending data, whether to use previously identified biometric data in a tracking operation. A mode of the XR device is selected responsive to determining whether to use the previously identified biometric data. The selected mode is used to initialize the tracking operation. The selected mode may be a first mode in which the previously identified biometric data is used in the tracking operation or a second mode in which the previously identified biometric data is not used in the tracking operation.

Abstract: An architecture is provided for packaging visual overlay-based user interfaces (UIs) into mobile device applications to work as user interface extensions that allow certain flows and logic to be displayed on an eyewear device when connected to the mobile device application. The extension of the UIs of the mobile device applications to the display of the eyewear device allows for inexpensive experimentation with augmented reality (AR) UIs for eyewear devices and allows for reusing of business logic across mobile devices and associated eyewear devices. For example, a mobile device application for maps or navigation may be extended to show directions on an associated eyewear device once the destination is chosen in the navigation application on the mobile device. In this example, the business logic would still live in the navigation application on the mobile device but the user would see AR directions overlaid on a display of the eyewear device.

Abstract: Context-sensitive remote controls for use with electronic devices (e.g., eyewear device). The electronic device is configured to perform activities (e.g., email, painting, navigation, gaming). The context-sensitive remote control includes a display having a display area, a display driver coupled to the display, and a transceiver. The remote control additionally includes memory that stores controller layout configurations for display in the display area of the display by the display driver. A processor in the context-sensitive remote control is configured to establish, via the transceiver, communication with an electronic device, detect an activity currently being performed by the electronic device, select one of the controller layout configurations responsive to the detected activity, and present, via the display driver, the selected controller layout configuration in the display area of the display.

Abstract: Systems and methods are provided for performing operations on an augmented reality (AR) device. The system accesses, by the AR device, movement data comprising inertial measurement data and camera data. The system determines three-dimensional (3D) movement of the AR device based on the movement data. The system presents, by the AR device, an AR object on a real-world environment being viewed using the AR device. The system, in response to determining the 3D movement of the AR device, modifies the AR object by the AR device.

Abstract: Systems and methods are provided for performing operations including: receiving, by a messaging application, input that selects an option to generate a message using an avatar with a caption; presenting, by the messaging application, the avatar and a caption entry region proximate to the avatar; populating, by the messaging application, the caption entry region with a text string comprising one or more words; determining, by the messaging application, context based on the one or more words in the text string; and modifying, by the messaging application, an expression of the avatar based on the determined context.

Abstract: The subject technology receives, by a client device from a storage device, first image data captured by the client device at a previous time. The subject technology receives first metadata corresponding to at least a first image processing operation and a second image processing operation. The subject technology generates, in a first render pass, second image data based on the first metadata and the image processing operation performed on the first image data. The subject technology generates, in a second render pass, third image data based on the first metadata and the second image processing operation performed on the second image data. The subject technology generates second metadata comprising information corresponding to the third image data. The subject technology generates a composite AR content item comprising the second metadata, the third image data, and the first image data.

Abstract: A messaging system performs neural network hair rendering for images provided by users of the messaging system. A method of neural network hair rendering includes processing a three-dimensional (3D) model of fake hair and a first real hair image depicting a first person to generate a fake hair structure, and encoding, using a fake hair encoder neural subnetwork, the fake hair structure to generate a coded fake hair structure. The method further includes processing, using a cross-domain structure embedding neural subnetwork, the coded fake hair structure to generate a fake and real hair structure, and encoding, using an appearance encoder neural subnetwork, a second real hair image depicting a second person having a second head to generate an appearance map. The method further includes processing, using a real appearance renderer neural subnetwork, the appearance map and the fake and real hair structure to generate a synthesized real image.

Abstract: In one or more implementations, user content items generated using a client application may be shared with users that are not contacts of the user within the client application. A user interface that indicates a number of recipients of the user content item may be generated that also includes a first section that displays the user content item and a second section to add text content to the user content item. In various examples, one or more classifications may be associated with the user content item.

Abstract: Visual-inertial tracking of an eyewear device using sensors. The eyewear device monitors the sensors of a visual inertial odometry system (VIOS) that provide input for determining a position of the device within its environment. The eyewear device determines the status of the VIOS based information from the sensors and adjusts the plurality of sensors (e.g., by turning on/off sensors, changing the sampling rate, of a combination thereof) based on the determined status. The eyewear device then determines the position of the eyewear device within the environment using the adjusted plurality of sensors.

Abstract: System, method, and non-transitory computer readable medium for presenting images on a mobile device. Images are presented by monitoring the location and the orientation of the mobile device, requesting previously captured images corresponding to where the previously captured images were captured in relation to the location from a server, receiving the requested previously captured images from the server, selecting images from the requested previously captured images responsive to the location and the orientation of the mobile device, generating overlay images from the selected images including image icons associated with the selected images, presenting the overlay images on an optical assembly, receiving an image selection identifying one of the image icons in the presented overlay images, and display the selected image associated with the identified image icon on the viewing area of the optical assembly.

Abstract: Systems and methods for automated local story generation and curation are described. In one example embodiment, a server computer receives content from client devices, and processes the content to identify content characteristics. Stories are then generated based on the characteristics of the received content, and the stories are communicated to client devices. In certain embodiments, selection at a client device of an individual piece of content within a story may further be used by the system to provide the client device with a sub-story that includes pieces of content sharing content characteristics with the characteristics of the selected image or video.