Abstract: An improved system architecture uses a pipeline including a Generative Adversarial Network (GAN) including a generator neural network and a discriminator neural network to generate an image. An input image in a first domain and information about a target domain are obtained. The domains correspond to image styles. An initial latent space representation of the input image is produced by encoding the input image. An initial output image is generated by processing the initial latent space representation with the generator neural network. Using the discriminator neural network, a score is computed indicating whether the initial output image is in the target domain. A loss is computed based on the computed score. The loss is minimized to compute an updated latent space representation. The updated latent space representation is processed with the generator neural network to generate an output image in the target domain.

Abstract: Systems and methods generate a filtering function for editing an image with reduced attribute correlation. An image editing system groups training data into bins according to a distribution of a target attribute. For each bin, the system samples a subset of the training data based on a pre-determined target distribution of a set of additional attributes in the training data. The system identifies a direction in the sampled training data corresponding to the distribution of the target attribute to generate a filtering vector for modifying the target attribute in an input image, obtains a latent space representation of an input image, applies the filtering vector to the latent space representation of the input image to generate a filtered latent space representation of the input image, and provides the filtered latent space representation as input to a neural network to generate an output image with a modification to the target attribute.

Abstract: Embodiments provide systems, methods, and computer storage media for fitting 3D primitives to a 3D point cloud. In an example embodiment, 3D primitives are fit to a 3D point cloud using a global primitive fitting network that evaluates the entire 3D point cloud and a local primitive fitting network that evaluates local patches of the 3D point cloud. The global primitive fitting network regresses a representation of larger (global) primitives that fit the global structure. To identify smaller 3D primitives for regions with fine detail, local patches are constructed by sampling from a pool of points likely to contain fine detail, and the local primitive fitting network regresses a representation of smaller (local) primitives that fit the local structure of each of the local patches. The global and local primitives are merged into a combined, multi-scale set of fitted primitives, and representative primitive parameters are computed for each fitted primitive.

Abstract: A method, apparatus, and non-transitory computer readable medium for image processing are described. Embodiments of the method, apparatus, and non-transitory computer readable medium include identifying an original image including a plurality of semantic attributes, wherein each of the semantic attributes represents a complex set of features of the original image; identifying a target attribute value that indicates a change to a target attribute of the semantic attributes; computing a modified feature vector based on the target attribute value, wherein the modified feature vector incorporates the change to the target attribute while holding at least one preserved attribute of the semantic attributes substantially unchanged; and generating a modified image based on the modified feature vector, wherein the modified image includes the change to the target attribute and retains the at least one preserved attribute from the original image.

Abstract: Embodiments provide systems, methods, and computer storage media for fitting 3D primitives to a 3D point cloud. In an example embodiment, 3D primitives are fit to a 3D point cloud using a global primitive fitting network that evaluates the entire 3D point cloud and a local primitive fitting network that evaluates local patches of the 3D point cloud. The global primitive fitting network regresses a representation of larger (global) primitives that fit the global structure. To identify smaller 3D primitives for regions with fine detail, local patches are constructed by sampling from a pool of points likely to contain fine detail, and the local primitive fitting network regresses a representation of smaller (local) primitives that fit the local structure of each of the local patches. The global and local primitives are merged into a combined, multi-scale set of fitted primitives, and representative primitive parameters are computed for each fitted primitive.

Abstract: An improved system architecture uses a Generative Adversarial Network (GAN) including a specialized generator neural network to generate multiple resolution output images. The system produces a latent space representation of an input image. The system generates a first output image at a first resolution by providing the latent space representation of the input image as input to a generator neural network comprising an input layer, an output layer, and a plurality of intermediate layers and taking the first output image from an intermediate layer, of the plurality of intermediate layers of the generator neural network. The system generates a second output image at a second resolution different from the first resolution by providing the latent space representation of the input image as input to the generator neural network and taking the second output image from the output layer of the generator neural network.

Abstract: Systems and methods generate a filtering function for editing an image with reduced attribute correlation. An image editing system groups training data into bins according to a distribution of a target attribute. For each bin, the system samples a subset of the training data based on a pre-determined target distribution of a set of additional attributes in the training data. The system identifies a direction in the sampled training data corresponding to the distribution of the target attribute to generate a filtering vector for modifying the target attribute in an input image, obtains a latent space representation of an input image, applies the filtering vector to the latent space representation of the input image to generate a filtered latent space representation of the input image, and provides the filtered latent space representation as input to a neural network to generate an output image with a modification to the target attribute.

Abstract: An improved system architecture uses a pipeline including a Generative Adversarial Network (GAN) including a generator neural network and a discriminator neural network to generate an image. An input image in a first domain and information about a target domain are obtained. The domains correspond to image styles. An initial latent space representation of the input image is produced by encoding the input image. An initial output image is generated by processing the initial latent space representation with the generator neural network. Using the discriminator neural network, a score is computed indicating whether the initial output image is in the target domain. A loss is computed based on the computed score. The loss is minimized to compute an updated latent space representation. The updated latent space representation is processed with the generator neural network to generate an output image in the target domain.

Abstract: An improved system architecture uses a pipeline including an encoder and a Generative Adversarial Network (GAN) including a generator neural network to generate edited images with improved speed, realism, and identity preservation. The encoder produces an initial latent space representation of an input image by encoding the input image. The generator neural network generates an initial output image by processing the initial latent space representation of the input image. The system generates an optimized latent space representation of the input image using a loss minimization technique that minimizes a loss between the input image and the initial output image. The loss is based on target perceptual features extracted from the input image and initial perceptual features extracted from the initial output image. The system outputs the optimized latent space representation of the input image for downstream use.

Abstract: Systems and methods dynamically adjust an available range for editing an attribute in an image. An image editing system computes a metric for an attribute in an input image as a function of a latent space representation of the input image and a filtering vector for editing the input image. The image editing system compares the metric to a threshold. If the metric exceeds the threshold, then the image editing system selects a first range for editing the attribute in the input image. If the metric does not exceed the threshold, a second range is selected. The image editing system causes display of a user interface for editing the input image comprising an interface element for editing the attribute within the selected range.

Abstract: Systems and methods train and apply a specialized encoder neural network for fast and accurate projection into the latent space of a Generative Adversarial Network (GAN). The specialized encoder neural network includes an input layer, a feature extraction layer, and a bottleneck layer positioned after the feature extraction layer. The projection process includes providing an input image to the encoder and producing, by the encoder, a latent space representation of the input image. Producing the latent space representation includes extracting a feature vector from the feature extraction layer, providing the feature vector to the bottleneck layer as input, and producing the latent space representation as output. The latent space representation produced by the encoder is provided as input to the GAN, which generates an output image based upon the latent space representation. The encoder is trained using specialized loss functions including a segmentation loss and a mean latent loss.

Abstract: Systems and methods train an encoder neural network for fast and accurate projection into the latent space of a Generative Adversarial Network (GAN). The encoder is trained by providing an input training image to the encoder and producing, by the encoder, a latent space representation of the input training image. The latent space representation is provided as input to the GAN to generate a generated training image. A latent code is sampled from a latent space associated with the GAN and the sampled latent code is provided as input to the GAN. The GAN generates a synthetic training image based on the sampled latent code. The sampled latent code is provided as input to the encoder to produce a synthetic training code. The encoder is updated by minimizing a loss between the generated training image and the input training image, and the synthetic training code and the sampled latent code.

Abstract: A method, apparatus, and non-transitory computer readable medium for image processing are described. Embodiments of the method, apparatus, and non-transitory computer readable medium include identifying an original image including a plurality of semantic attributes, wherein each of the semantic attributes represents a complex set of features of the original image; identifying a target attribute value that indicates a change to a target attribute of the semantic attributes; computing a modified feature vector based on the target attribute value, wherein the modified feature vector incorporates the change to the target attribute while holding at least one preserved attribute of the semantic attributes substantially unchanged; and generating a modified image based on the modified feature vector, wherein the modified image includes the change to the target attribute and retains the at least one preserved attribute from the original image.

Abstract: This application relates generally to augmenting images and videos with dynamic object compositing, and more specifically, to generating synthetic training data to train a machine learning model to automatically augment an image or video with a dynamic object. The synthetic training data may contain multiple data points from thousands of simulated dynamic object movements within a virtual environment. Based on the synthetic training data, the machine learning model may determine the movement of a new dynamic object within new virtual environment.

Abstract: This application relates generally to augmenting images and videos with dynamic object compositing, and more specifically, to generating synthetic training data to train a machine learning model to automatically augment an image or video with a dynamic object. The synthetic training data may contain multiple data points from thousands of simulated dynamic object movements within a virtual environment. Based on the synthetic training data, the machine learning model may determine the movement of a new dynamic object within new virtual environment.

Abstract: Methods and systems for generating digital models from objects. In particular, one or more embodiments determine a plurality of correspondences for first and second components of an object. One or more embodiments estimate a joint connecting the first and second components based on the correspondences. One or more embodiments jointly determine a global transformation and one or more joint parameters that map the plurality of components of the object from the first digital scan to the second digital scan. One or more embodiments also updating the correspondences based on the determined global transformation and parameter(s). One or more embodiments re-estimate the joint based on the updated correspondences. One or more embodiments select a candidate joint with a lowest error estimate from a plurality of candidate joints according to determined global transformations and joint parameter(s) for the candidate joints.

Abstract: The systems and techniques disclosed herein provide tutorials for drawing three dimensional objects with accurate proportions and perspective. A user is able to select an object and a viewpoint to automatically generate a tutorial. Regardless of the object and viewpoint, an easy-to-use tutorial is produced that guides the user to draw the object with accurate proportions and perspective. Given a segmented 3D model of the object and a camera viewpoint, a sequence of steps for constructing the scaffold is determined. The sequence of steps is based on an intelligent selection of primitives and inter-primitive anchorings that provides an order for drawing the primitives and makes the scaffold easy to construct. The primitives and inter-primitive anchorings are selected from a rich set of possibilities that allow for some inaccuracies to reduce the difficulty of the tutorial. The primitives and inter-primitive anchoring are selected to balance the difficulty and the potential inaccuracy.

Abstract: The systems and techniques disclosed herein provide tutorials for drawing three dimensional objects with accurate proportions and perspective. A user is able to select an object and a viewpoint to automatically generate a tutorial. Regardless of the object and viewpoint, an easy-to-use tutorial is produced that guides the user to draw the object with accurate proportions and perspective. Given a segmented 3D model of the object and a camera viewpoint, a sequence of steps for constructing the scaffold is determined. The sequence of steps is based on an intelligent selection of primitives and inter-primitive anchorings that provides an order for drawing the primitives and makes the scaffold easy to construct. The primitives and inter-primitive anchorings are selected from a rich set of possibilities that allow for some inaccuracies to reduce the difficulty of the tutorial. The primitives and inter-primitive anchoring are selected to balance the difficulty and the potential inaccuracy.

Abstract: Methods and systems for generating digital models from objects. In particular, one or more embodiments determine a plurality of correspondences for first and second components of an object. One or more embodiments estimate a joint connecting the first and second components based on the correspondences. One or more embodiments jointly determine a global transformation and one or more joint parameters that map the plurality of components of the object from the first digital scan to the second digital scan. One or more embodiments also updating the correspondences based on the determined global transformation and parameter(s). One or more embodiments re-estimate the joint based on the updated correspondences. One or more embodiments select a candidate joint with a lowest error estimate from a plurality of candidate joints according to determined global transformations and joint parameter(s) for the candidate joints.