Abstract: Methods and systems are disclosed for performing operations for transferring garments from one real-world object to another in real time. The operations comprise receiving a first video that includes a depiction of a first person wearing a first upper-body garment in a first pose and obtaining a second video that includes a depiction of a second person wearing a second upper-body garment in a second pose. A pose of the second person depicted in the second video is modified to match the first pose of the first person depicted in the first video. The operations comprise generating an upper-body segmentation of the second upper-body garment which the second person is wearing in the second video in the modified pose and replacing the first upper-body garment worn by the first person in the first video with the second upper-body garment based on the upper-body segmentation.

Abstract: Examples relate to systems and methods for generating an avatar animation. The systems and methods access an audio file comprising speech, spoken by a user, captured by a microphone of a user system, and receive input that selects an avatar associated with the user. The systems and methods process the audio file and the avatar, selected by the received input, by a generative machine learning model to generate an animation of the avatar having lips moving to represent the avatar speaking the speech of the audio file. The systems and methods generate a video comprising a depiction of the generated animation of the avatar speaking the speech of the audio file.

Abstract: Methods and systems are disclosed for using machine learning models to perform pixel-based deformation of fashion items. The methods and systems receive one or more images depicting a person in an individual pose and receive a first source image depicting a first view of a target fashion item and a second source image depicting a second view of the target fashion item. The methods and systems process, using one or more machine learning models, the one or more images that depict the person in the individual pose together with the first and second source images to generate a flow field, the flow field indicating a likelihood of existence and location of each pixel of the one or more images relative to the first and second source images. The methods and systems modify a portion of the one or more images to overlay the target fashion item on the person.

Abstract: Methods and systems are disclosed for generating an extended reality (XR) try-on experience based on an image produced by a diffusion model. The system receives a first image depicting a real-world object and receives a second image depicting a target fashion item. The system generates a warped image in which pixels of the target fashion item depicted in the second image replace pixels of a portion of the real-world object in the first image and generates one or more segmentation maps corresponding to incomplete portions of the warped image. The system analyzes the warped image and the one or more segmentation maps using a generative machine learning model to generate an artificial image that populates the incomplete portions of the warped image to depict the real-world object wearing the target fashion item.

Abstract: Methods and systems are disclosed for applying machine learning models to compressed videos. The system receives a video, depicting an object, that has previously been compressed using one or more video compression processes. The system analyzes, using one or more machine learning models, the video that has previously been compressed to generate a prediction corresponding to the object depicted in the video, with one or more artifacts resulting from application of the one or more machine learning models to the video that has been previously compressed being absent from the prediction. The system generates a visual output based on the prediction in which the one or more artifacts are absent.

Abstract: Methods and systems are disclosed for using machine learning models to perform pixel-based deformation of fashion items. The methods and systems receive one or more images depicting a person in an individual pose and receive a first source image depicting a first view of a target fashion item and a second source image depicting a second view of the target fashion item. The methods and systems process, using one or more machine learning models, the one or more images that depict the person in the individual pose together with the first and second source images to generate a flow field, the flow field indicating a likelihood of existence and location of each pixel of the one or more images relative to the first and second source images. The methods and systems modify a portion of the one or more images to overlay the target fashion item on the person.

Abstract: Methods and systems are disclosed for using machine learning models to perform pixel-based deformation of fashion items. The methods and systems receive one or more images depicting a first person in a first pose and receive a source image depicting a target fashion item worn on a portion of a body of a second person in a second pose. The methods and systems process, using one or more machine learning models, the one or more images together with the source image to generate a flow field indicating existence and location of each pixel of the one or more images in the source image and modify, based on the flow field, a portion of the one or more images to overlay the target fashion item on the first person including one or more portions of the target fashion item that extend beyond a body of the first person.

Abstract: Methods and systems are disclosed for using machine learning models to perform smoothing in latent space using optical flow information. The methods and systems access a first frame of a video depicting an object and a second frame of the video, the second frame corresponding to a later time period in the video than the first frame. The methods and systems generate optical flow information based on the first frame and the second frame, the optical flow information describing movement of the object from the first frame to the second frame. The methods and systems smooth a latent space generated by one or more neural network encoders of a machine learning model using the optical flow information and process the smoothed latent space by one or more neural network decoders to generate a result of the machine learning model.

Abstract: Methods and systems are disclosed for applying machine learning models to compressed videos. The system receives a video, depicting an object, that has previously been compressed using one or more video compression processes. The system analyzes, using one or more machine learning models, the video that has previously been compressed to generate a prediction corresponding to the object depicted in the video, with one or more artifacts resulting from application of the one or more machine learning models to the video that has been previously compressed being absent from the prediction. The system generates a visual output based on the prediction in which the one or more artifacts are absent.

Abstract: Methods and systems are disclosed for generating an extended reality (XR) try-on experience based on an image produced by a diffusion model. The system receives a first image depicting a real-world object and receives a second image depicting a target fashion item. The system generates a warped image in which pixels of the target fashion item depicted in the second image replace pixels of a portion of the real-world object in the first image and generates one or more segmentation maps corresponding to incomplete portions of the warped image. The system analyzes the warped image and the one or more segmentation maps using a generative machine learning model to generate an artificial image that populates the incomplete portions of the warped image to depict the real-world object wearing the target fashion item.

Abstract: Methods and systems are disclosed for transferring garments from one real-world object to another in real time using body landmarks. The system receives a first image that includes a depiction of a first person wearing a fashion item in a first pose. The system obtains a second image that includes a depiction of a second person in a second pose and generates a first set of body landmarks corresponding the first person in the first pose and a second set of body landmarks corresponding the second person wearing in the first pose. The system computes a deviation between the first set of body landmarks and the second set of body landmarks. The system generates a new image that depicts the second person wearing the fashion item worn by the first person based on the deviation between the first set of body landmarks and the second set of body landmarks.

Abstract: Methods and systems are disclosed for using machine learning models to perform pixel-based deformation of fashion items. The methods and systems receive one or more images depicting a first person in a first pose and receive a source image depicting a target fashion item worn on a portion of a body of a second person in a second pose. The methods and systems process, using one or more machine learning models, the one or more images together with the source image to generate a flow field indicating existence and location of each pixel of the one or more images in the source image and modify, based on the flow field, a portion of the one or more images to overlay the target fashion item on the first person including one or more portions of the target fashion item that extend beyond a body of the first person.

Abstract: Systems and methods are provided for generating an augmented reality (AR) experience. The systems and methods receive a video depicting movement of a humanoid and a target image depicting an object. The systems and methods process, by a generative machine learning (ML) model, the video and the target image to generate a new video depicting the object performing the movement. The systems and methods generate the AR experience using the new video to overlay a face of a user on a portion of the new video.

Abstract: Methods and systems are disclosed for transferring garments from one real-world object to another in real time using body landmarks. The system receives a first image that includes a depiction of a first person wearing a fashion item in a first pose. The system obtains a second image that includes a depiction of a second person in a second pose and generates a first set of body landmarks corresponding the first person in the first pose and a second set of body landmarks corresponding the second person wearing in the first pose. The system computes a deviation between the first set of body landmarks and the second set of body landmarks. The system generates a new image that depicts the second person wearing the fashion item worn by the first person based on the deviation between the first set of body landmarks and the second set of body landmarks.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and a method for performing operations comprising: receiving a video that includes a depiction of one or more objects in a room within a home; determining a room classification for the room by processing the one or more objects depicted in the video; selecting one or more augmented reality items available for purchase based on the room classification and the one or more objects depicted in the video; and generating, for display within the video, the one or more augmented reality items that have been selected at a display position within the video corresponding to the one or more objects depicted in the video.

Abstract: Methods and systems are disclosed for using machine learning models to perform pixel-based deformation of fashion items using N-flows estimation. The methods and systems receive one or more images depicting a first person in a first pose and receive a source image depicting a target fashion item worn on a portion of a body of a second person in a second pose, the target fashion item comprising a plurality of parts. The methods and systems process, using one or more machine learning models, the one or more images together with the source image to generate a set of data comprising a plurality of flow fields each associated with a different one of the plurality of parts of the target fashion item. The methods and systems modify a portion of the one or more images to overlay the target fashion item on the first person in the first pose.

Abstract: Aspects of the present disclosure involve a system for an augmented reality (AR) try-on experience with lighting adjustment. The system accesses an image that depicts a real-world person. The system retrieves an AR fashion item. The system applies the image and the AR fashion item to a machine learning model to estimate a lighting adjustment for the AR fashion item based on lighting properties of the real-world person depicted in the image, the machine learning model trained to match lighting properties of images depicting real-world objects to lighting properties of AR objects and vice versa. The system combines the AR fashion item with the image that depicts the real-world person based on the estimated lighting adjustment to generate a modified image.

Abstract: Methods and systems are disclosed for using machine learning models to perform pixel-based deformation of fashion items. The methods and systems receive one or more images depicting a person in an individual pose and receive a first source image depicting a first view of a target fashion item and a second source image depicting a second view of the target fashion item. The methods and systems process, using one or more machine learning models, the one or more images that depict the person in the individual pose together with the first and second source images to generate a flow field, the flow field indicating a likelihood of existence and location of each pixel of the one or more images relative to the first and second source images. The methods and systems modify a portion of the one or more images to overlay the target fashion item on the person.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and a method for performing operations comprising: receiving a video that includes a depiction of one or more objects in a room within a home; determining a room classification for the room by processing the one or more objects depicted in the video; selecting one or more augmented reality items available for purchase based on the room classification and the one or more objects depicted in the video; and generating, for display within the video, the one or more augmented reality items that have been selected at a display position within the video corresponding to the one or more objects depicted in the video.

Abstract: Aspects of the present disclosure involve a system for an augmented reality (AR) try-on experience with lighting adjustment. The system accesses an image that depicts a real-world person. The system retrieves an AR fashion item. The system applies the image and the AR fashion item to a machine learning model to estimate a lighting adjustment for the AR fashion item based on lighting properties of the real-world person depicted in the image, the machine learning model trained to match lighting properties of images depicting real-world objects to lighting properties of AR objects and vice versa. The system combines the AR fashion item with the image that depicts the real-world person based on the estimated lighting adjustment to generate a modified image.