Abstract: Methods and systems are disclosed for performing operations comprising: receiving an image that includes a depiction of a face of a user; generating a plurality of landmarks of the face based on the received image; removing a set of interfering landmarks from the plurality of landmarks resulting in a remaining set of landmarks of the plurality of landmarks; obtaining a depth map for the face of the user; and computing a real-world scale of the face of the user based on the depth map and the remaining set of landmarks.

Abstract: Aspects of the present disclosure involve a system for providing virtual experiences. The system accesses, by a messaging application, an image depicting a person. The system generates, by the messaging application, a three-dimensional (3D) avatar based on the person depicted in the image. The system receives input that selects a pose for the 3D avatar and one or more fashion items to be worn by the 3D avatar and places, by the messaging application, the 3D avatar in the selected pose and wearing the one or more fashion items in an augmented reality (AR) experience.

Abstract: Methods and systems are disclosed for performing operations comprising: receiving an image that includes a depiction of a person wearing a fashion item; generating a segmentation of the fashion item by the person depicted in the image; receiving voice input associated with the person depicted in the image; in response to receiving the voice input, generating one or more augmented reality elements representing the voice input; and applying the one or more augmented reality elements to the fashion item worn by the person based on the segmentation of the fashion item worn by the person.

Abstract: Methods and systems are disclosed for performing operations comprising: receiving an image that includes a depiction of a person wearing a fashion item; generating a segmentation of the fashion item worn by the person depicted in the image; identifying a facial expression of the user depicted in the image; and in response to identifying the facial expression, applying one or more augmented reality elements to the fashion item worn by the person based on the segmentation of the fashion item worn by the person.

Abstract: Methods and systems are disclosed for generating 3D assets, such as for an extended reality (XR) experience. The system receives a plurality of textures associated with an object, each texture of the plurality of textures corresponding to a different view of the object and automatically generates an initial three-dimensional (3D) model of the object based on an initial alignment of the plurality of textures to respective portions of the 3D model. The system receives input that adjusts the initial alignment of the plurality of textures to the respective portions of the 3D model and combines the plurality of textures into a single texture based on the input, the single texture defining visual attributes of the object from multiple views. The system stores the 3D model in association with the single texture.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, and a method for performing operations comprising: receiving a video that depicts a person; tracking three-dimensional (3D) movement of the person within the video using a 3D reference point; computing a 3D position for placement of an augmented reality item relative to the 3D reference point; causing to be displayed the augmented reality item within the video at the 3D position; and updating the 3D position of the augmented reality item in the video as the 3D reference point changes based on the 3D movement of the person.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, and a method for performing operations comprising receiving an image that depicts a person, identifying a set of skeletal joints of the person and identifying a pose of the person depicted in the image based on positioning of the set of skeletal joints. The operations also include receiving speech input comprising a request to perform an AR operation and an ambiguous intent, discerning the ambiguous intent of the speech input based on the pose of the person depicted in the image and in response to receiving the speech input, performing the AR operation based on discerning the ambiguous intent of the speech input based on the pose of the person depicted in the image.

Abstract: Aspects of the present disclosure involve a system for presenting AR items. The system performs operations including receiving a video that includes a depiction of one or more real-world objects in a real-world environment and obtaining depth data related to the real-world environment. The operations include generating a three-dimensional (3D) model of the real-world environment based on the video and the depth data and adding an augmented reality (AR) item to the video based on the 3D model of the real-world environment. The operations include determining that the AR item has been placed on a vertical plane of the real-world environment and modifying an orientation of the AR item to correspond to an orientation of the vertical plane.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, and a method for performing operations comprising: receiving, by a client device associated with a first user, a communication from a second user; retrieving, from the communication, a movement vector representing three-dimensional (3D) movement of a set of skeletal joints of the second user; receiving, by the client device associated with the first user, input that selects a 3D avatar; and animating, based on the movement vector, the 3D avatar to mimic the 3D movement of the set of skeletal joints of the second user.

Abstract: Methods and systems are disclosed for performing operations for transferring garments in a video from one real-world object to another in real time. The operations comprise receiving a first video that includes a depiction of a first person wearing a first garment in a first pose and obtaining a second video that includes a depiction of a second person wearing a second garment in a second pose. The operations comprise modifying a pose of the second person to match the first pose of the first person depicted in the first video. The operations comprise generating a whole-body segmentation of the second garment which the second person is wearing in the second video and changing an appearance of the first person from wearing the first garment to wearing the second garment based on the whole-body segmentation of the second garment which the second person is wearing in the second video.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, and a method for performing operations comprising: receiving a video that depicts a person. The operations further include identifying a set of skeletal joints of the person. The operations further include identifying a pose of the person depicted in the video based on positioning of the set of skeletal joints (or detecting a hand pose, detecting a mirror frame, or detecting a mobile device). The operations further include determining, based on the pose of the person (or detecting a hand pose, detecting a mirror frame, or detecting a mobile device), that the video comprises a mirror reflection of the person. The operations further include, in response to determining that the video comprises the mirror reflection of the person, causing display of a 3D virtual object in the video.

Abstract: Methods and systems are disclosed for performing operations comprising: accessing content received from a first client device associated with a first user; processing the content to identify a first image that depicts the first user wearing a first fashion item; determining a first pose of the first user depicted in the first image; searching a plurality of products to identify a first product that corresponds to the first pose of the first user depicted in the first image and that includes one or more attributes associated with the first fashion item; modifying the first image to generate an advertisement that depicts the first user wearing the first product; and during a content browsing session being accessed by the first client device, causing the advertisement that depicts the first user wearing the first product to be displayed automatically on the first client device.

Abstract: Methods and systems are disclosed for performing operations comprising: receiving a monocular image that includes a depiction of a person wearing an article of clothing; generating a segmentation of the article of clothing worn by the person in the monocular image; obtaining one or more audio-track related augmented reality elements; and applying the one or more audio-track related augmented reality elements to the article of clothing worn by the person based on the segmentation of the article of clothing worn by the person.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and a method for detecting a pose of a user. The program and method include operations comprising receiving a monocular image that includes a depiction of a body of a user; detecting a plurality of skeletal joints of the body based on the monocular image; accessing a video feed comprising a plurality of monocular images received prior to the monocular image; filtering, using the video feed, the plurality of skeletal joints of the body detected based on the monocular image; and determining a pose represented by the body depicted in the monocular image based on the filtered plurality of skeletal joints of the body.

Abstract: Methods and systems are disclosed for using machine learning models to recommend fashion item fit styles based on body surface landmarks. The methods and systems access one or more images depicting a person wearing one or more fashion items and process, using one or more machine learning models, the one or more images to estimate a data set comprising a set of body landmarks of the person, a set of garment classifications associated with the one or more fashion items, and a set of garment segmentations for the one or more fashion items. The methods and systems identify one or more fit styles associated with the person based on the estimated data set and cause presentation of one or more real-world fashion items matching the identified one or more fit styles.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, and a method for performing operations comprising: receiving a video that depicts a person; tracking three-dimensional (3D) movement of the person within the video using a 3D reference point; computing a 3D position for placement of an augmented reality item relative to the 3D reference point; causing to be displayed the augmented reality item within the video at the 3D position; and updating the 3D position of the augmented reality item in the video as the 3D reference point changes based on the 3D movement of the person.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, and a method for performing operations comprising: capturing a video that depicts a person; identifying a set of skeletal joints of the person depicted in the video; storing a movement vector representing previously captured three-dimensional (3D) movement of the set of skeletal joints of the person depicted in the video; receiving input that selects a 3D avatar; and animating, based on the movement vector, the 3D avatar to mimic the previously captured 3D movement of the set of skeletal joints of the person depicted in the video.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, and a method for performing operations comprising: receiving, by a client device associated with a first user, a communication from a second user; retrieving, from the communication, a movement vector representing three-dimensional (3D) movement of a set of skeletal joints of the second user; receiving, by the client device associated with the first user, input that selects a 3D avatar; and animating, based on the movement vector, the 3D avatar to mimic the 3D movement of the set of skeletal joints of the second user.

Abstract: Methods and systems are disclosed for performing operations for transferring garments in a video from one real-world object to another in real time. The operations comprise receiving a first video that includes a depiction of a first person wearing a first garment in a first pose and obtaining a second video that includes a depiction of a second person wearing a second garment in a second pose. The operations comprise modifying a pose of the second person to match the first pose of the first person depicted in the first video. The operations comprise generating a whole-body segmentation of the second garment which the second person is wearing in the second video and changing an appearance of the first person from wearing the first garment to wearing the second garment based on the whole-body segmentation of the second garment which the second person is wearing in the second video.

Abstract: Methods and systems are disclosed for generating a physical garment using a machine learning model. The methods and systems receive a plurality of parameters of an optimization problem, the plurality of parameters describing a fashion item, and form a prompt based on values of the plurality of parameters. The prompt is processed by a generative machine learning model to output an image comprising an artificial fashion item corresponding to the values of the plurality of parameters. An augmented reality experience is generated in which a real-world object is overlaid with a virtual object that depicts the artificial fashion item. Feedback associated with the augmented reality experience is used to condition fabrication of a real-world fashion item that resembles the artificial fashion item.