Abstract: Systems and embodiments herein describe an augmented reality (AR) object rendering system. The AR object rendering system receives an image, generates a set of noise parameters and a set of blur parameters for the image using a neural network trained on a paired dataset of images, identifies an AR object associated with the image, modifies the AR object using the set of noise parameters and the set of blur parameters, displays the modified augmented reality object within the image.

Abstract: A three-dimensional asset (3D) reconstruction technique for generating a 3D asset representing an object from images of the object. The images are captured from different viewpoints in a darkroom using one or more light sources having known locations. The system estimates camera poses for each of the captured images and then constructs a 3D surface mesh made up of surfaces using the captured images and their respective estimated camera poses. Texture properties for each of the surfaces of the 3D surface mesh are then refined to generate the 3D asset.

Abstract: A data item is identified on a device. A neural network that includes an adversarial transformation subnetwork is applied to the data item to generate a modified data item. Output indicative of the modified data item is caused to be presented on the device. The neural network further comprises an encoder and a decoder. The neural network is trained in at least two stages. At least one of the encoder and the decoder is trained in a first stage and the adversarial transformation subnetwork is trained in a second stage.

Abstract: Methods and systems are disclosed for performing operations for generating a photorealistic rendering of an object. The operations include: accessing a set of albedo textures and a machine learning model associated with a real-world object, the set of albedo textures and a machine learning model having been trained based on a plurality of viewpoints of the real-world object; obtaining a three-dimensional (3D) mesh of the real-world object; receiving input that selects a new viewpoint that differs from the plurality of viewpoints of the real-world object; and generating a photorealistic rendering of the real-world object from the new viewpoint based on the 3D mesh of the real-world object, the set of albedo textures, and the machine learning model associated with the real-world object.

Abstract: A method for applying lighting conditions to a virtual object in an augmented reality (AR) device is described. In one aspect, the method includes generating, using a camera of a mobile device, an image, accessing a virtual object corresponding to an object in the image, identifying lighting parameters of the virtual object based on a machine learning model that is pre-trained with a paired dataset, the paired dataset includes synthetic source data and synthetic target data, the synthetic source data includes environment maps and 3D scans of items depicted in the environment map, the synthetic target data includes a synthetic sphere image rendered in the same environment map, applying the lighting parameters to the virtual object, and displaying, in a display of the mobile device, the shaded virtual object as a layer to the image.

Abstract: Systems and embodiments herein describe an augmented reality (AR) object rendering system. The AR object rendering system receives an image, generates a set of noise parameters and a set of blur parameters for the image using a neural network trained on a paired dataset of images, identifies an AR object associated with the image, modifies the AR object using the set of noise parameters and the set of blur parameters, displays the modified augmented reality object within the image.

Abstract: An image manipulation system for generating modified images using a generative adversarial network (GAN) trains GANs using domain changes, aligns input images with generated images, classifies and associates target images based on a symmetry, and uses a modified discriminator structure. A method for domain changes includes generating, using a pre-trained GAN trained on a plurality of first target images, a plurality of images, and determining a feature for each of the plurality of images. The method further includes determining the feature for each of a plurality of second target images and matching, based on the feature, second target images of the plurality of second target images with the plurality of images. The method further includes training a discriminator of the pre-trained GAN with the second target images and the plurality of images.

Abstract: Methods and systems are disclosed for performing automatically creating AR experiences on a messaging platform. The methods and systems perform operations that include: receiving, via a graphical user interface (GUI), input that specifies a plurality of image transformation parameters; accessing a set of sample source images; modifying the set of sample source images based on the plurality of image transformation parameters to generate a set of sample target images; training a machine learning model to generate a given target image from a given source image by establishing a relationship between the set of sample source images and the set of sample target images; and automatically generating an augmented reality experience comprising the trained machine learning model.

Abstract: Systems and methods herein describe a video compression system. The described systems and methods accesses a sequence of image frames from a first computing device, the sequence of image frames comprising a first image frame and a second image frame, detects a first set of keypoints for the first image frame, transmits the first image frame and the first set of keypoints to a second computing device, detects a second set of keypoints for the second image frame, transmits the second set of keypoints to the second computing device, causes an animated image to be displayed on the second computing device.

Abstract: Systems and methods herein describe a video compression system. The described systems and methods accesses a sequence of image frames from a first computing device, the sequence of image frames comprising a first image frame and a second image frame, detects a first set of keypoints for the first image frame, transmits the first image frame and the first set of keypoints to a second computing device, detects a second set of keypoints for the second image frame, transmits the second set of keypoints to the second computing device, causes an animated image to be displayed on the second computing device.

Abstract: Systems and methods are disclosed for generating, a source image sequence using an image sensor of the computing device, the source image sequence comprising a plurality of source images depicting a head and face, identifying driving image sequence data to modify face image feature data in the source image sequence, generating, using an image transformation neural network, a modified source image sequence comprising a plurality of modified source images depicting modified versions of the head and face, and storing the modified source image sequence on the computing device.

Abstract: Systems and methods herein describe an image stylization system. The image stylization system accesses a set of images corresponding to a target domain style, generates a set of paired images using a first machine learning model, analyze the generated set of paired images using a second machine learning model trained to analyze the generated set of paired images based on a plurality of protected feature criteria, determines a set of image transformations for the generated set of pairs, generates a transformed set of paired images by performing the set of image transformations on the set of paired images, and generates stylized images corresponding to the target domain style using a supervised image translation model trained on the transformed set of paired images.

Abstract: A data item is identified on a device. A neural network that includes an adversarial transformation subnetwork is applied to the data item to generate a modified data item. Output indicative of the modified data item is caused to be presented on the device. The neural network further comprises an encoder and a decoder. The neural network is trained in at least two stages. At least one of the encoder and the decoder is trained in a first stage and the adversarial transformation subnetwork is trained in a second stage.

Abstract: A mobile device can implement a neural network-based domain transfer scheme to modify an image in a first domain appearance to a second domain appearance. The domain transfer scheme can be configured to detect an object in the image, apply an effect to the image, and blend the image using color space adjustments and blending schemes to generate a realistic result image. The domain transfer scheme can further be configured to efficiently execute on the constrained device by removing operational layers based on resources available on the mobile device.

Abstract: Systems and methods are disclosed for generating, a source image sequence using an image sensor of the computing device, the source image sequence comprising a plurality of source images depicting a head and face, identifying driving image sequence data to modify face image feature data in the source image sequence, generating, using an image transformation neural network, a modified source image sequence comprising a plurality of modified source images depicting modified versions of the head and face, and storing the modified source image sequence on the computing device.

Abstract: Systems and methods herein describe an image manipulation system for generating modified images using a generative adversarial network. The image manipulation system accesses a pre-trained generative adversarial network (GAN), fine-tunes the pre-trained GAN by training a portion of existing neural network layers of the pre-trained GAN and newly added layers of the pre-trained GAN on a secondary image domain, adjusts the weights of the fine-tuned GAN using the weights of the pre-trained GAN, and stores the fine-tuned GAN. An image transformation system uses the generated modified images to train a subsequent neural network, which can access a face from a client device and transform it to a domain of images used for GAN fine-tuning.

Abstract: Systems and methods herein describe a video compression system. The described systems and methods acceses a sequence of image frames from a first computing device, the sequence of image frames comprising a first image frame and a second image frame, detects a first set of keypoints for the first image frame, transmits the first image frame and the first set of keypoints to a second computing device, detects a second set of keypoints for the second image frame, transmits the second set of keypoints to the second computing device, causes an animated image to be displayed on the second computing device.

Abstract: A mobile device can implement a neural network-based domain transfer scheme to modify an image in a first domain appearance to a second domain appearance. The domain transfer scheme can be configured to detect an object in the image, apply an effect to the image, and blend the image using color space adjustments and blending schemes to generate a realistic result image. The domain transfer scheme can further be configured to efficiently execute on the constrained device by removing operational layers based on resources available on the mobile device.

Abstract: A mobile device can implement a neural network-based domain transfer scheme to modify an image in a first domain appearance to a second domain appearance. The domain transfer scheme can be configured to detect an object in the image, apply an effect to the image, and blend the image using color space adjustments and blending schemes to generate a realistic result image. The domain transfer scheme can further be configured to efficiently execute on the constrained device by removing operational layers based on resources available on the mobile device.

Abstract: A method for image analysis comprises receiving one or more images of a plurality of lesions captured from a body of a person, extracting one or more features of the plurality of lesions from the one or more images, analyzing the extracted one or more features, wherein the analyzing comprises determining a distance between at least two lesions with respect to the extracted one or more features, and determining whether any of the plurality of lesions is an outlier based on the analyzing.