Abstract: A model generation system captures, using cameras, hand-tracking data of a gesture made by a user demonstrating the gesture. The model generation system generates a three-dimensional model of the gesture using the hand-tracking data and provides a display of the three-dimensional model to the user. The model generation system receives, from the user, model refining data refining the three-dimensional model. The model generation system generates a refined three-dimensional model of the gesture using the model refining data and the three-dimensional model. The refined three-dimensional model is used for detecting the gesture.

Abstract: Systems and methods are presented for capturing a video in real-time by an image capture device using a skeletal pose system. The skeletal pose system identifies first pose information in the video, applies a first virtual effect to the video in response to identifying the first pose information, identifies second pose information in the video, and applies a second virtual effect to the video in response to identifying the first pose information.

Abstract: The subject matter describe herein relates to a system and method for creating augmented reality (AR) applications using artificial intelligence. The system comprises a multi-agent architecture including a lens or AR content creator agent, AR engineer agent, and designer agent that collaborate to generate AR application designs based on user input. A content management system stores reusable components and asset generators provide customized visual elements. The user interface allows natural language interactions to iteratively refine the AR application. The system leverages large language models and retrieval-augmented generation to construct appropriate prompts and select relevant components. Generated designs are assembled into executable AR applications using a plugin that interfaces with an AR development environment. This AI-assisted approach enables rapid creation of diverse, engaging AR experiences with minimal technical expertise required from users.

Abstract: Systems, methods, and computer readable media for augmented reality beauty product tutorials. Methods disclose determining from live images of an augmented reality (AR) tutorial effects, the effects indicating changes to the live images of a presenter of the AR tutorial from a beauty product being applied to a body part of the presenter. The methods further comprising determining from the live images motion, the motion indicating motion of the beauty product from the beauty product being applied to the body part of the presenter and storing the effects and the motion.

Abstract: Systems and methods herein describe a method for capturing a video in real-time by an image capture device. The system provides a plurality of visual pose hints, identifies first pose information in the video while capturing the video, applies a first series of virtual effects to the video, identifies second pose information, and applies a second series of virtual effects to the video, the second series of virtual effects based on the first series of virtual effects.

Abstract: Systems and methods are presented for capturing a video in real-time by an image capture device using a skeletal pose system. The skeletal pose system identifies first pose information in the video, applies a first virtual effect to the video in response to identifying the first pose information, identifies second pose information in the video, and applies a second virtual effect to the video in response to identifying the first pose information.

Abstract: Systems, methods, and computer readable media for augmented reality beauty product tutorials. Methods disclose determining from live images of an augmented reality (AR) tutorial effects, the effects indicating changes to the live images of a presenter of the AR tutorial from a beauty product being applied to a body part of the presenter. The methods further comprising determining from the live images motion, the motion indicating motion of the beauty product from the beauty product being applied to the body part of the presenter and storing the effects and the motion.

Abstract: Aspects of the present disclosure involve a system for navigating menus and selecting options using hand gestures. The system detects a first hand gesture depicted in a video, the first hand gesture performed by a first hand of a person. The system, in response to detecting the first hand gesture, accesses a first menu comprising a plurality of options. The system detects, within a threshold period of time from when the first hand gesture is detected, a second hand gesture depicted in the video. The system, in response to detecting, within the threshold period of time from when the first hand gesture is detected, the second hand gesture, performs an operation associated with an individual option of the plurality of options of the first menu.

Abstract: Systems and methods herein describe a method for capturing a video in real-time by an image capture device. The system provides a plurality of visual pose hints, identifies first pose information in the video while capturing the video, applies a first series of virtual effects to the video, identifies second pose information, and applies a second series of virtual effects to the video, the second series of virtual effects based on the first series of virtual effects.

Abstract: Systems, methods, and computer-readable media for adding beauty products to tutorials are presented. Methods include accessing video data comprising images of a presenter creating a tutorial, the tutorial depicting the presenter applying a beauty product to a body part of the presenter. Methods further include processing the video data to identify changes to the body part of the presenter from an application of the beauty product, and responding to identifying changes to the body part of the presenter from the application of the beauty product by processing the video data to identify the beauty product. Methods further include retrieving information regarding the beauty product and causing presentation of information regarding the beauty product on a display device.

Abstract: Systems and methods describe an augmented reality (AR) system for generating AR experiences in real-time. The AR experience system receives a selection of an AR experience from an application running on a computer device, displays a first set of textual cues associated with the AR experience on the computer device, receives a first set of image data corresponding to the first set of textual cues, generates a first AR object associated with the AR experience, displays the first AR object on the computer device and a second set of textual cues associated with the AR experience, receives a second set of image data corresponding to the second set of textual cues, and generates a second AR object associated with the AR experience.

Abstract: Systems, methods, and computer-readable media for adding beauty products to tutorials are presented. Methods include accessing video data comprising images of a presenter creating a tutorial, the tutorial depicting the presenter applying a beauty product to a body part of the presenter. Methods further include processing the video data to identify changes to the body part of the presenter from an application of the beauty product, and responding to identifying changes to the body part of the presenter from the application of the beauty product by processing the video data to identify the beauty product. Methods further include retrieving information regarding the beauty product and causing presentation of information regarding the beauty product on a display device.

Abstract: A model generation system captures, using cameras, hand-tracking data of a gesture made by a user demonstrating the gesture. The model generation system generates a three-dimensional model of the gesture using the hand-tracking data and provides a display of the three-dimensional model to the user. The model generation system receives, from the user, model refining data refining the three-dimensional model. The model generation system generates a refined three-dimensional model of the gesture using the model refining data and the three-dimensional model. The refined three-dimensional model is used for detecting the gesture.

Abstract: Systems and methods herein describe a multi-modal interaction system. The multi-modal interaction system, receives a selection of an augmented reality (AR) experience within an application on a computer device, displays a set of AR objects associated with the AR experience on a graphical user interface (GUI) of the computer device, display textual cues associated with the set of augmented reality objects on the GUI, receives a hand gesture and a voice command, modifies a subset of augmented reality objects of the set of augmented reality objects based on the hand gesture and the voice command, and displays the modified subset of augmented reality objects on the GUI.

Abstract: Systems and methods describe an augmented reality (AR) system for generating AR experiences in real-time. The AR experience system receives a selection of an AR experience from an application running on a computer device, displays a first set of textual cues associated with the AR experience on the computer device, receives a first set of image data corresponding to the first set of textual cues, generates a first AR object associated with the AR experience, displays the first AR object on the computer device and a second set of textual cues associated with the AR experience, receives a second set of image data corresponding to the second set of textual cues, and generates a second AR object associated with the AR experience.

Abstract: Systems, methods, and computer readable media for augmented reality beauty product tutorials are disclosed. Methods disclose displaying on a first portion of a screen a beauty product tutorial including pre-recorded images of a presenter applying a beauty product to a body part of the presenter and determining first effects indicating changes to make to live images of a user to simulate applying the beauty product to the body part of the user, where the first effects are determined based on second effects indicating changes to the pre-recorded images from applying the beauty product to the body part of the presenter. Methods further disclose processing the live images of the user to add the first effects and indications of motion of the beauty product and displaying the processed live images of the user on a second portion of the screen.

Abstract: Systems and methods herein describe a multi-modal interaction system. The multi-modal interaction system, receives a selection of an augmented reality (AR) experience within an application on a computer device, displays a set of AR objects associated with the AR experience on a graphical user interface (GUI) of the computer device, display textual cues associated with the set of augmented reality objects on the GUI, receives a hand gesture and a voice command, modifies a subset of augmented reality objects of the set of augmented reality objects based on the hand gesture and the voice command, and displays the modified subset of augmented reality objects on the GUI.

Abstract: Systems, methods, and computer readable media for augmented reality beauty product tutorials are disclosed. Methods disclose displaying on a first portion of a screen a beauty product tutorial including pre-recorded images of a presenter applying a beauty product to a body part of the presenter and determining first effects indicating changes to make to live images of a user to simulate applying the beauty product to the body part of the user, where the first effects are determined based on second effects indicating changes to the pre-recorded images from applying the beauty product to the body part of the presenter. Methods further disclose processing the live images of the user to add the first effects and indications of motion of the beauty product and displaying the processed live images of the user on a second portion of the screen.

Abstract: Systems and methods herein describe receiving an image via an image capture device, using a machine learning model, generating an image augmentation decision, accessing an augmented reality content item, associating the generated image augmentation decision with the augmented reality content item, modifying the received image using the augmented reality content item and the associated image augmentation decision, and causing presentation of the modified image on a graphical user interface of a computing device.

Abstract: Methods and systems are disclosed for operating an extended reality (XR) experience using one or more machine learning models. The methods and systems accessing, by an interaction application, an XR application and receive, by the interaction application, a query that defines one or more attributes of the XR application. The methods and systems generate a prompt for a generative machine learning model using the query and process the prompt using the generative machine learning model to generate one or more data objects that match the one or more attributes defined by the query. The methods and systems generate, using the XR application, one or more XR objects based on the one or more data objects generated by the generative machine learning model responsive to the prompt.