Abstract: Methods and systems are disclosed for performing operations comprising: receiving a video that includes a depiction of a person wearing a fashion item; generating a segmentation of the fashion item worn by the person depicted in the video; applying one or more augmented reality elements to the fashion item worn by the person based on the segmentation of the fashion item worn by the person; detecting a gesture performed by the person in the video; and modifying the one or more augmented reality elements that have been applied to the fashion item worn by the person based on the gesture performed by the person.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing programs and methods for performing operations comprising: receiving a request from a client device of a first user to engage in a shared virtual reality shopping experience with a second user; generating, for display on respective client devices of the first and second users, the shared virtual reality shopping experience comprising a plurality of virtual reality items that represent real-world products; receiving, from the client device of the second user, data indicating a selection of a first virtual reality item of the plurality of virtual reality items made by the second user; and modifying a display attribute of the first virtual item in the display of the shared virtual reality shopping experience on the client device of the first user to indicate the selection of the first virtual reality item made by the second user.

Abstract: Methods and systems are disclosed for generating 3D assets, such as for an extended reality (XR) experience. The system accesses a three-dimensional (3D) model of an object that has been generated based on at least one texture image of the object. The system processes the at least one texture image of the object by a machine learning model to generate a depth map and selects a portion of the 3D model. The system modifies the selected portion of the 3D model based on the depth map that has been generated by the machine learning model.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and a method for performing operations comprising: receiving, from a client device of a first user, a request from the first user to engage in an AR shopping experience curated by a store; identifying a first real-world product available for purchase from the store; receiving an image of a real-world environment of the first user; generating a first AR item that represents the first real-world product; comparing visual attributes of the first AR item to physical layouts of a plurality of real-world objects depicted in the image of the real-world environment; and overlaying the first AR item on a first real-world object of the plurality of real-world objects in the image responsive to comparing the visual attributes of the first AR item to the physical layouts of the plurality of real-world objects.

Abstract: Methods and systems are disclosed for performing operations for transferring garments from one real-world object to another in real time. The operations comprise receiving a first video that includes a depiction of a first person wearing a first upper-body garment in a first pose and obtaining a second video that includes a depiction of a second person wearing a second upper-body garment in a second pose. A pose of the second person depicted in the second video is modified to match the first pose of the first person depicted in the first video. The operations comprise generating an upper-body segmentation of the second upper-body garment which the second person is wearing in the second video in the modified pose and replacing the first upper-body garment worn by the first person in the first video with the second upper-body garment based on the upper-body segmentation.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, and a method for performing operations comprising receiving an image that depicts a person, identifying a set of skeletal joints of the person and identifying a pose of the person depicted in the image based on positioning of the set of skeletal joints. The operations also include receiving speech input comprising a request to perform an AR operation and an ambiguous intent, discerning the ambiguous intent of the speech input based on the pose of the person depicted in the image and in response to receiving the speech input, performing the AR operation based on discerning the ambiguous intent of the speech input based on the pose of the person depicted in the image.

Abstract: Examples relate to systems and methods for generating an avatar animation. The systems and methods access an audio file comprising speech, spoken by a user, captured by a microphone of a user system, and receive input that selects an avatar associated with the user. The systems and methods process the audio file and the avatar, selected by the received input, by a generative machine learning model to generate an animation of the avatar having lips moving to represent the avatar speaking the speech of the audio file. The systems and methods generate a video comprising a depiction of the generated animation of the avatar speaking the speech of the audio file.

Abstract: Methods and systems are disclosed for using machine learning models to recommend fashion item fit styles based on body surface landmarks. The methods and systems access one or more images depicting a person wearing one or more fashion items and process, using one or more machine learning models, the one or more images to estimate a data set comprising a set of body landmarks of the person, a set of garment classifications associated with the one or more fashion items, and a set of garment segmentations for the one or more fashion items. The methods and systems identify one or more fit styles associated with the person based on the estimated data set and cause presentation of one or more real-world fashion items matching the identified one or more fit styles.

Abstract: Methods and systems are disclosed for applying machine learning models to compressed videos. The system receives a video, depicting an object, that has previously been compressed using one or more video compression processes. The system analyzes, using one or more machine learning models, the video that has previously been compressed to generate a prediction corresponding to the object depicted in the video, with one or more artifacts resulting from application of the one or more machine learning models to the video that has been previously compressed being absent from the prediction. The system generates a visual output based on the prediction in which the one or more artifacts are absent.

Abstract: The subject matter describe herein relates to a system and method for creating augmented reality (AR) applications using artificial intelligence. The system comprises a multi-agent architecture including a lens or AR content creator agent, AR engineer agent, and designer agent that collaborate to generate AR application designs based on user input. A content management system stores reusable components and asset generators provide customized visual elements. The user interface allows natural language interactions to iteratively refine the AR application. The system leverages large language models and retrieval-augmented generation to construct appropriate prompts and select relevant components. Generated designs are assembled into executable AR applications using a plugin that interfaces with an AR development environment. This AI-assisted approach enables rapid creation of diverse, engaging AR experiences with minimal technical expertise required from users.

Abstract: Methods and systems are disclosed for generating a physical garment using a machine learning model. The methods and systems receive a plurality of parameters of an optimization problem, the plurality of parameters describing a fashion item, and form a prompt based on values of the plurality of parameters. The prompt is processed by a generative machine learning model to output an image comprising an artificial fashion item corresponding to the values of the plurality of parameters. An augmented reality experience is generated in which a real-world object is overlaid with a virtual object that depicts the artificial fashion item. Feedback associated with the augmented reality experience is used to condition fabrication of a real-world fashion item that resembles the artificial fashion item.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and a method for performing operations comprising: receiving, from a client device of a first user, a request from the first user to engage in an AR shopping experience curated by a store; identifying a first real-world product available for purchase from the store; receiving an image of a real-world environment of the first user; generating a first AR item that represents the first real-world product; comparing visual attributes of the first AR item to physical layouts of a plurality of real-world objects depicted in the image of the real-world environment; and overlaying the first AR item on a first real-world object of the plurality of real-world objects in the image responsive to comparing the visual attributes of the first AR item to the physical layouts of the plurality of real-world objects.

Abstract: Methods and systems are disclosed for generating mirrored 3D assets for an extended reality (XR) experience. The system receives a three-dimensional (3D) object comprising a target and analyzes the 3D object using one or more machine learning models to generate data associated with a mirrored version of the target of the 3D object. The system applies the mirrored version of the target to a mirrored version of the 3D object using the generated data and generats a new 3D object comprising the mirrored version of the 3D object and the mirrored version of the target.

Abstract: Methods and systems are disclosed for generating an extended reality (XR) try-on experience based on an image produced by a diffusion model. The system receives an image depicting a real-world object and generates a prompt comprising a textual description of a fashion item. The system analyzes the image and the textual description of the fashion item using a generative machine learning model to generate an artificial image that depicts an artificial object that resembles the real-world object wearing an artificial fashion item matching the textual description of the fashion item. The system identifies an object comprising a real-world product image that matches visual attributes of the artificial fashion item and replaces the artificial fashion item in the artificial image with the object to generate an output image.

Abstract: Aspects of the present disclosure involve a system for generating images that depict a real-world object in a scene. The system receives image content comprising a depth map of a scene and a three-dimensional (3D) model of a real-world object. The system receives a textual description for a background. The system applies the image content and the textual description to a machine learning model to generate a scene image that depicts the real-world object on the background corresponding to the textual description.

Abstract: Methods and systems are disclosed for using machine learning models to perform smoothing in latent space using optical flow information. The methods and systems access a first frame of a video depicting an object and a second frame of the video, the second frame corresponding to a later time period in the video than the first frame. The methods and systems generate optical flow information based on the first frame and the second frame, the optical flow information describing movement of the object from the first frame to the second frame. The methods and systems smooth a latent space generated by one or more neural network encoders of a machine learning model using the optical flow information and process the smoothed latent space by one or more neural network decoders to generate a result of the machine learning model.

Abstract: Methods and systems are disclosed for applying machine learning models to compressed videos. The system receives a video, depicting an object, that has previously been compressed using one or more video compression processes. The system analyzes, using one or more machine learning models, the video that has previously been compressed to generate a prediction corresponding to the object depicted in the video, with one or more artifacts resulting from application of the one or more machine learning models to the video that has been previously compressed being absent from the prediction. The system generates a visual output based on the prediction in which the one or more artifacts are absent.

Abstract: Aspects of the present disclosure involve a system for synthetically generating a view of an object to present one or more AR fashion items in an image. The system accesses a first image depicting a first portion of a body of a person from a first camera view. The system detects a fashion item in the first image that is being worn by the person in the first image, the fashion item being depicted in the first image from the first camera view. The system generates an augmented reality (AR) item that visually resembles the fashion item from the second camera view and modifies the second image with the AR item to present a synthetic view of the second portion of the body of the person wearing the AR item corresponding to the fashion item.

Abstract: Aspects of the present disclosure involve a system for presenting AR items. The system performs operations including receiving a video that includes a depiction of one or more real-world objects in a real-world environment and obtaining depth information related to the real-world environment; and generating a 3D model of the real-world environment. The operations further include determining 3D placement and orientation for an AR item based on data associated with the AR item and the 3D model of the real-world environment and causing display of a marker in the video that specifies the 3D placement and orientation of the AR item. The operations further include rendering a display of the AR item within the video according to the 3D placement and orientation in response to movement of the marker within the video.

Abstract: Aspects of the present disclosure involve a system for automatically zooming in on objects in an image. The system accesses a video comprising a frame depicting a person and detects a fashion item being worn by the person in the frame of the video. The system, in response to detecting the fashion item being worn by the person, extracts, from the frame, a portion comprising a depiction of the fashion item being worn by the person. The system generates a focus image of the fashion item using the portion extracted from the frame and presents the focus image of the fashion item together with the video comprising the frame depicting the person.