Abstract: Aspects of the present disclosure involve a system for presenting AR items. The system performs operations including receiving a video that includes a depiction of one or more real-world objects in a real-world environment and obtaining depth data related to the real-world environment. The operations include generating a three-dimensional (3D) model of the real-world environment based on the video and the depth data and adding an augmented reality (AR) item to the video based on the 3D model of the real-world environment. The operations include determining that the AR item has been placed on a vertical plane of the real-world environment and modifying an orientation of the AR item to correspond to an orientation of the vertical plane.

Abstract: Aspects of the present disclosure involve a system for presenting AR items. The system receives a video that includes a depiction of a real-world object in a real-world environment. The system generates a three-dimensional (3D) bounding box for the real-world object and stabilizes the 3D bounding box based on one or more sensors of the device. The system determines a position, orientation, and dimensions of the real-world object based on the stabilized 3D bounding box and renders a display of an augmented reality (AR) item within the video based on the position, orientation, and dimensions of the real-world object.

Abstract: Aspects of the present disclosure involve a system for performing real-time in-painting using machine learning techniques. The system receives a video that includes a depiction of a real-world object in a real-world environment. The system accesses a segmentation associated with the real-world object and removes a depiction of the real-world object from a region of a first frame of the video. The system processes, by a machine learning model, the first frame and one or more previous frames of the video that precede the first frame to generate a new frame in which portions of the first frame have been blended into the region from which the depiction of the real-world object has been removed.

Abstract: Aspects of the present disclosure involve a system for presenting AR items. The system performs operations including receiving a video that includes a depiction of one or more real-world objects in a real-world environment and obtaining depth data related to the real-world environment. The operations include generating a three-dimensional (3D) model of the real-world environment based on the video and the depth data and adding an augmented reality (AR) item to the video based on the 3D model of the real-world environment. The operations include determining that the AR item has been placed on a vertical plane of the real-world environment and modifying an orientation of the AR item to correspond to an orientation of the vertical plane.

Abstract: Aspects of the present disclosure involve a system for providing AR experiences. The system accesses, by a messaging application, an image depicting a real-world fashion item of a user and generates a three-dimensional (3D) virtual fashion item based on the real-world fashion item depicted in the image. The system stores the 3D virtual fashion item in a database that includes a virtual wardrobe comprising a plurality of 3D virtual fashion items associated with the user. The system generates, by the messaging application, an augmented reality (AR) experience that allows the user to interact with the virtual wardrobe.

Abstract: Aspects of the present disclosure involve a system for presenting augmented reality (AR) items. The system performs operations including receiving a video that includes a depiction of a real-world environment and generating a 3D model of the real-world environment based on the video. The operations include determining, based on the 3D model of the real-world environment, that an AR item has been placed in the video at a particular 3D position and identifying a portion of the 3D model corresponding to the real-world environment currently being displayed on a screen. The operations include determining that the 3D position of the AR item is excluded from the portion of the 3D model currently being displayed on the screen and in response, displaying an indicator that identifies the 3D position of the AR item in the 3D model relative to the portion of the 3D model currently being displayed on a screen.

Abstract: Methods and systems are disclosed for performing operations comprising: receiving a monocular image that includes a depiction of a whole body of a user; generating a segmentation of the whole body of the user based on the monocular image; accessing a video feed comprising a plurality of monocular images received prior to the monocular image; smoothing, using the video feed, the segmentation of the whole body generated based on the monocular image to provide a smoothed segmentation; and applying one or more visual effects to the monocular image based on the smoothed segmentation.

Abstract: Aspects of the present disclosure involve a system for providing virtual experiences. The system performs operations including selecting, by a messaging application, a virtual experience that represents a previously captured real-world environment at a first location; accessing an image representing a new real-world environment at a second location, the image depicting a plurality of real-world objects; receiving input that selects a first real-world object from the plurality of real-world objects depicted in the image; and modifying the image, accessed at the second location, based on the virtual experience to depict the previously captured real-world environment with the first real-world object.

Abstract: Methods and systems are disclosed for performing operations comprising: receiving a monocular image that includes a depiction of a whole body of a user; generating a segmentation of the whole body of the user based on the monocular image; accessing a video feed comprising a plurality of monocular images received prior to the monocular image; smoothing, using the video feed, the segmentation of the whole body generated based on the monocular image to provide a smoothed segmentation; and applying one or more visual effects to the monocular image based on the smoothed segmentation.

Abstract: Aspects of the present disclosure involve a system for presenting AR items. The system performs operations including receiving a video that includes a depiction of one or more real-world objects in a real-world environment and obtaining depth information related to the real-world environment; and generating a 3D model of the real-world environment. The operations further include determining 3D placement and orientation for an AR item based on data associated with the AR item and the 3D model of the real-world environment and causing display of a marker in the video that specifies the 3D placement and orientation of the AR item. The operations further include rendering a display of the AR item within the video according to the 3D placement and orientation in response to movement of the marker within the video.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and a method for performing operations comprising: receiving a video that includes a depiction of a real-world object in a real-world environment; determining a classification for the real-world environment by processing the real-world object depicted in the video; selecting an augmented reality (AR) item based on the classification of the real-world environment and the real-world object depicted in the video; modifying pixels corresponding to the real-world object depicted in the video to generate a modified video that excludes the depiction of the real-world object; and adding the AR item to the modified video at a display position corresponding to the modified pixels.

Abstract: Aspects of the present disclosure involve a system for presenting augmented reality (AR) items. The system performs operations including receiving a video that includes a depiction of a real-world environment and generating a 3D model of the real-world environment based on the video. The operations include determining, based on the 3D model of the real-world environment, that an AR item has been placed in the video at a particular 3D position and identifying a portion of the 3D model corresponding to the real-world environment currently being displayed on a screen. The operations include determining that the 3D position of the AR item is excluded from the portion of the 3D model currently being displayed on the screen and in response, displaying an indicator that identifies the 3D position of the AR item in the 3D model relative to the portion of the 3D model currently being displayed on a screen.

Abstract: Aspects of the present disclosure involve a system for presenting AR items. The system performs operations including receiving a video that includes a depiction of one or more real-world objects in a real-world environment and obtaining depth data related to the real-world environment. The operations include generating a three-dimensional (3D) model of the real-world environment based on the video and the depth data and adding an augmented reality (AR) item to the video based on the 3D model of the real-world environment. The operations include determining that the AR item has been placed on a vertical plane of the real-world environment and modifying an orientation of the AR item to correspond to an orientation of the vertical plane.

Abstract: Methods and systems are disclosed for performing operations comprising: receiving a monocular image that includes a depiction of a user wearing a garment; generating a segmentation of the garment worn by the user in the monocular image; accessing a video feed comprising a plurality of monocular images received prior to the monocular image; smoothing, using the video feed, the segmentation of the garment worn by the user to provide a smoothed segmentation of the garment worn by the user; and applying one or more visual effects to the monocular image based on the smoothed segmentation of the garment worn by the user.

Abstract: Methods and systems are disclosed for performing operations comprising: receiving a monocular image that includes a depiction of a whole body of a user; generating a segmentation of the whole body of the user based on the monocular image; accessing a video feed comprising a plurality of monocular images received prior to the monocular image; smoothing, using the video feed, the segmentation of the whole body generated based on the monocular image to provide a smoothed segmentation; and applying one or more visual effects to the monocular image based on the smoothed segmentation.

Abstract: Methods and systems are disclosed for performing operations comprising: receiving a monocular image that includes a depiction of a whole body of a user; generating a segmentation of the whole body of the user based on the monocular image by applying one or more machine learning techniques; receiving input that selects a visualization mode; and applying one or more visual effects corresponding to the visualization mode to the monocular image based on the segmentation.