Abstract: Methods and systems are disclosed for performing operations comprising: receiving an image that includes a depiction of a person wearing a fashion item; generating a segmentation of the fashion item by the person depicted in the image; receiving voice input associated with the person depicted in the image; in response to receiving the voice input, generating one or more augmented reality elements representing the voice input; and applying the one or more augmented reality elements to the fashion item worn by the person based on the segmentation of the fashion item worn by the person.

Abstract: Methods and systems are disclosed for performing operations comprising: receiving an image that includes a depiction of a person wearing a fashion item; generating a segmentation of the fashion item worn by the person depicted in the image; generating one or more game-based augmented reality elements; and applying the one or more game-based augmented reality elements to the fashion item worn by the person based on the segmentation of the fashion item worn by the person.

Abstract: Methods and systems are disclosed for performing operations comprising: receiving an image that includes a depiction of a person wearing a fashion item; generating a segmentation of the fashion item worn by the person depicted in the image; identifying a facial expression of the user depicted in the image; and in response to identifying the facial expression, applying one or more augmented reality elements to the fashion item worn by the person based on the segmentation of the fashion item worn by the person.

Abstract: Methods and systems are disclosed for performing operations comprising: receiving a monocular image that includes a depiction of a person wearing an article of clothing; generating a segmentation of the article of clothing worn by the person in the monocular image; obtaining one or more audio-track related augmented reality elements; and applying the one or more audio-track related augmented reality elements to the article of clothing worn by the person based on the segmentation of the article of clothing worn by the person.

Abstract: Methods and systems are disclosed for performing operations comprising: receiving a video that includes a depiction of a person wearing a fashion item; generating a segmentation of the fashion item worn by the person depicted in the video; applying one or more augmented reality elements to the fashion item worn by the person based on the segmentation of the fashion item worn by the person; detecting a gesture performed by the person in the video; and modifying the one or more augmented reality elements that have been applied to the fashion item worn by the person based on the gesture performed by the person.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and a method for detecting a pose of a user. The program and method include operations comprising receiving a monocular image that includes a depiction of a body of a user; detecting a plurality of skeletal joints of the body based on the monocular image; accessing a video feed comprising a plurality of monocular images received prior to the monocular image; filtering, using the video feed, the plurality of skeletal joints of the body detected based on the monocular image; and determining a pose represented by the body depicted in the monocular image based on the filtered plurality of skeletal joints of the body.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and a method for performing operations comprising: receiving, from a client device of a first user, a request from the first user to engage in an AR shopping experience curated by a store; identifying a first real-world product available for purchase from the store; receiving an image of a real-world environment of the first user; generating a first AR item that represents the first real-world product; comparing visual attributes of the first AR item to physical layouts of a plurality of real-world objects depicted in the image of the real-world environment; and overlaying the first AR item on a first real-world object of the plurality of real-world objects in the image responsive to comparing the visual attributes of the first AR item to the physical layouts of the plurality of real-world objects.

Abstract: Methods and systems are disclosed for performing operations comprising: receiving an image that includes a depiction of a face of a user; generating a plurality of landmarks of the face based on the received image; removing a set of interfering landmarks from the plurality of landmarks resulting in a remaining set of landmarks of the plurality of landmarks; obtaining a depth map for the face of the user; and computing a real-world scale of the face of the user based on the depth map and the remaining set of landmarks.

Abstract: Methods and systems are disclosed for performing operations comprising: accessing content received from a first client device associated with a first user; processing the content to identify a first image that depicts the first user wearing a first fashion item; determining a first pose of the first user depicted in the first image; searching a plurality of products to identify a first product that corresponds to the first pose of the first user depicted in the first image and that includes one or more attributes associated with the first fashion item; modifying the first image to generate an advertisement that depicts the first user wearing the first product; and during a content browsing session being accessed by the first client device, causing the advertisement that depicts the first user wearing the first product to be displayed automatically on the first client device.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and a method for detecting a pose of a user. The program and method include receiving a monocular image that includes a depiction of a body of a user; detecting a plurality of skeletal joints of the body depicted in the monocular image; and determining a pose represented by the body depicted in the monocular image based on the detected plurality of skeletal joints of the body. A pose of an avatar is modified to match the pose represented by the body depicted in the monocular image by adjusting a set of skeletal joints of a rig of an avatar based on the detected plurality of skeletal joints of the body; and the avatar having the modified pose that matches the pose represented by the body depicted in the monocular image is generated for display.

Abstract: A messaging system performs virtual clothing try-on. A method of virtual clothing try-on may include accessing a target garment image and a person image of a person wearing a source garment and processing the person image to generate a source garment mask and a person mask. The method may further include processing the source garment mask, the person mask, the target garment image, and a target garment mask to generate a warping, the warping indicating a warping to apply to the target garment image. The method may further include processing the target garment to warp the target garment in accordance with the warping to generate a warped target garment image, processing the warped target garment image to blend with the person image to generate a person with a blended target garment image, and processing the person with blended target garment image to fill in holes to generate an output image.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, and a method for performing operations comprising: capturing a video that depicts a person; identifying a set of skeletal joints of the person depicted in the video; storing a movement vector representing previously captured three-dimensional (3D) movement of the set of skeletal joints of the person depicted in the video; receiving input that selects a 3D avatar; and animating, based on the movement vector, the 3D avatar to mimic the previously captured 3D movement of the set of skeletal joints of the person depicted in the video.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and a method for performing operations comprising: receiving, from a client device of a first user, a request from the first user to engage in an AR shopping experience curated by a store; identifying a first real-world product available for purchase from the store; receiving an image of a real-world environment of the first user; generating a first AR item that represents the first real-world product; comparing visual attributes of the first AR item to physical layouts of a plurality of real-world objects depicted in the image of the real-world environment; and overlaying the first AR item on a first real-world object of the plurality of real-world objects in the image responsive to comparing the visual attributes of the first AR item to the physical layouts of the plurality of real-world objects.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing programs and methods for performing operations comprising: receiving a request from a client device of a first user to engage in a shared virtual reality shopping experience with a second user; generating, for display on respective client devices of the first and second users, the shared virtual reality shopping experience comprising a plurality of virtual reality items that represent real-world products; receiving, from the client device of the second user, data indicating a selection of a first virtual reality item of the plurality of virtual reality items made by the second user; and modifying a display attribute of the first virtual item in the display of the shared virtual reality shopping experience on the client device of the first user to indicate the selection of the first virtual reality item made by the second user.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and a method for performing operations comprising: receiving a request from a client device of a first user to engage in a shared augmented reality (AR) shopping experience with a second user; generating, for display on respective client devices of the first and second users, the shared AR shopping experience comprising a plurality of AR items that represent real-world products overlaid on a shared real-world environment of the first and second users; determining that the second user is interacting with a first AR item of the plurality of AR items; and in response to detecting that the second user is depicted in an image captured by the client device of the first user, overlaying the first AR item on the second user in the image.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and a method for detecting a pose of a user. The program and method include operations comprising: receiving a monocular image that includes a depiction of a body of a user in a given pose; detecting a plurality of skeletal joints of the body depicted in the monocular image; generating an outline of the body of the user based on the plurality of skeletal joints; retrieving a mask that represents a target pose; comparing the outline of the body depicted in the monocular image with the mask that represents the target pose; and determining that the given pose of the body of the user matches the target pose in response to comparing the outline of the body with the mask.

Abstract: Embodiments of the present disclosure relate generally to systems for enhancing a first media item through the addition of a supplemental second media item. A user may provide a request to enhance a selected media item, and in response, an enhancement system retrieves and presents a curated collection of supplemental content to be added to the media, to the user. The user may review the curated collection of supplemental content, for example by providing a tactile input to scroll through the curated collection of content.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and a method for performing operations comprising: receiving a video that includes a depiction of one or more objects in a room within a home; determining a room classification for the room by processing the one or more objects depicted in the video; selecting one or more augmented reality items available for purchase based on the room classification and the one or more objects depicted in the video; and generating, for display within the video, the one or more augmented reality items that have been selected at a display position within the video corresponding to the one or more objects depicted in the video.

Abstract: Methods and systems are disclosed for performing operations comprising: receiving an image that includes a depiction of a face of a user; generating a plurality of landmarks of the face based on the received image; removing a set of interfering landmarks from the plurality of landmarks resulting in a remaining set of landmarks of the plurality of landmarks; obtaining a depth map for the face of the user; and computing a real-world scale of the face of the user based on the depth map and the remaining set of landmarks.

Abstract: Methods and systems are disclosed for performing operations comprising: receiving, by one or more processors, an image that includes a depiction of a face of a user; computing a real-world scale of the face of the user based on a selected subset of landmarks of the face of the user; obtaining an augmented reality graphical element comprising augmented reality eyewear; scaling the augmented reality graphical element based on the computed real-world scale of the face; and positioning the scaled augmented reality graphical element within the image on the face of the user.