Abstract: A system and method for determining a correspondence map between a first and second image by determining a set of correspondence vectors for each pixel in the first image and selecting a correspondence vector from the set of correspondence vectors based on a cost value.

Abstract: Methods and systems are disclosed for generating an animatable garment from a single image. The system accesses a monocular image depicting a person wearing a fashion item and generates a three-dimensional (3D) mesh representing the person in a canonical space. The system determines a pose of the person depicted in the monocular image and modifies a pose of the 3D mesh to match the determined pose of the person. The system extracts a portion of the 3D mesh corresponding to the fashion item and generates an extended reality (XR) item using the extracted portion of the 3D mesh.

Abstract: Methods and systems are disclosed for generating a body mesh from a single image. The system predicts both a volumetric reconstruction tensor of the monocular image and a pose of an object by applying a first machine learning model to a monocular image. The system identifies a portion of the pose of the object that corresponds to a point in a canonical space associated with a set of position encoding information. The system obtains a point of the volumetric reconstruction tensor corresponding to the identified portion of the pose. The system classifies the obtained point as being inside or outside of a canonical volume by applying a second machine learning model to the obtained point of the volumetric reconstruction tensor together with the set of position encoding information. The system generates a three-dimensional (3D) mesh representing the object in the canonical space.

Abstract: A system and/or method for imaging system calibration, including: determining a dense correspondence map matching features in a first image to features in a second image, and determining updated calibration parameters associated with the image acquisition system based on the matching features.

Abstract: An system for augmenting images using hand surface normal estimation is provided. In a model training phase, 3D models of hands are generated using 3D data of hands in a variety of positions. Target normal training data is generated that includes normals of surfaces of the 3D models and synthetic 2D image training data corresponding to the 3D models and the normals. The target normal training data and the synthetic image training data are used to train a normal estimation model. The normal estimation is used by an interactive application to generate augmentations that are applied to hand image data.

Abstract: A system for generating extended reality effects using image data of hands and a depth estimation model. The depth estimation model is trained using pairings of synthetic 2D image data with sets of depths and segmentation masks. An extended reality system captures image data of hands in a real-world scene and uses the image data and the depth estimation model to generate the extended reality effects. The extended reality effects are provided to a user during an extended reality experience.

Abstract: Methods and systems are disclosed for performing operations comprising: receiving a video that includes a depiction of a real-world object; generating a three-dimensional (3D) body mesh associated with the real-world object that tracks movement of the real-world object across frames of the video; obtaining an external mesh associated with an augmented reality element; automatically establishing a correspondence between the 3D body mesh associated with the real-world object and the external mesh; deforming the external mesh based on movement of the real-world object and the established correspondence with the 3D body mesh; and modifying the video to include a display of the augmented reality element based on the deformed external mesh.

Abstract: A system and/or method for imaging system calibration, including: determining a dense correspondence map matching features in a first image to features in a second image, and determining updated calibration parameters associated with the image acquisition system based on the matching features.

Abstract: Methods and systems are disclosed for performing real-time deforming operations. The system receives an image that includes a depiction of a real-world object. The system applies a machine learning model to the image to generate a warping field and segmentation mask, the machine learning model trained to establish a relationship between a plurality of training images depicting real-world objects and corresponding ground-truth warping fields and segmentation masks associated with a target shape. The system applies the generated warping field and segmentation mask to the image to warp the real-world object depicted in the image to the target shape.

Abstract: Aspects of the present disclosure involve a system and a method for performing operations comprising: receiving a plurality of bone scale coefficients each corresponding to respective bones of a skeleton model; receiving a plurality of joint angle coefficients that collectively define a pose for the skeleton model; generating the skeleton model based on the received bone scale coefficients and the received joint angle coefficients; generating a base surface based on the plurality of bone scale coefficients; generating an identity surface by deformation of the base surface; and generating the 3D body model by mapping the identity surface onto the posed skeleton model.

Abstract: Methods and systems are disclosed for performing operations comprising: receiving a video that includes a depiction of a real-world object; generating a three-dimensional (3D) body mesh associated with the real-world object that tracks movement of the real-world object across frames of the video; obtaining an external mesh associated with an augmented reality element; automatically establishing a correspondence between the 3D body mesh associated with the real-world object and the external mesh; deforming the external mesh based on movement of the real-world object and the established correspondence with the 3D body mesh; and modifying the video to include a display of the augmented reality element based on the deformed external mesh.

Abstract: Aspects of the present disclosure involve a system for providing virtual experiences. The system accesses, by a messaging application, an image depicting a person. The system generates, by the messaging application, a three-dimensional (3D) avatar based on the person depicted in the image. The system receives input that selects a pose for the 3D avatar and one or more fashion items to be worn by the 3D avatar and places, by the messaging application, the 3D avatar in the selected pose and wearing the one or more fashion items in an augmented reality (AR) experience.

Abstract: A system and/or method for imaging system calibration, including: determining a dense correspondence map matching features in a first image to features in a second image, and determining updated calibration parameters associated with the image acquisition system based on the matching features.

Abstract: A system and method for determining a correspondence map between a first and second image by determining a set of correspondence vectors for each pixel in the first image and selecting a correspondence vector from the set of correspondence vectors based on a cost value.

Abstract: A system and method for determining a correspondence map between a first and second image by determining a set of correspondence vectors for each pixel in the first image and selecting a correspondence vector from the set of correspondence vectors based on a cost value.

Abstract: Methods and systems are disclosed for performing operations comprising: receiving a video that includes a depiction of a real-world object; generating a three-dimensional (3D) body mesh associated with the real-world object that tracks movement of the real-world object across frames of the video; obtaining an external mesh associated with an augmented reality element; automatically establishing a correspondence between the 3D body mesh associated with the real-world object and the external mesh; deforming the external mesh based on movement of the real-world object and the established correspondence with the 3D body mesh; and modifying the video to include a display of the augmented reality element based on the deformed external mesh.

Abstract: A system and/or method for imaging system calibration, including: determining a dense correspondence map matching features in a first image to features in a second image, and determining updated calibration parameters associated with the image acquisition system based on the matching features.

Abstract: A system and/or method for imaging system calibration, including: determining a dense correspondence map matching features in a first image to features in a second image, and determining updated calibration parameters associated with the image acquisition system based on the matching features.

Abstract: A system and method for determining a correspondence map between a first and second image by determining a set of correspondence vectors for each pixel in the first image and selecting a correspondence vector from the set of correspondence vectors based on a cost value.

Abstract: A system and/or method for imaging system calibration, including: determining a dense correspondence map matching features in a first image to features in a second image, and determining updated calibration parameters associated with the image acquisition system based on the matching features.