Abstract: The present disclosure relates to systems, methods, and non-transitory computer-readable media that generates a refined digital image that corrects inpainting artifacts in a modified digital image with inpainted pixels. Furthermore, the disclosed systems receive an inpainting request to inpaint a region in a digital image (e.g., a region indicated by a mask). Moreover, the disclosed systems generate a modified digital image from the inpainting request and the digital image, where the modified digital image includes one or more inpainted portions in place of the region. Further, the disclosed systems use a neural network-based refiner model to generate a refined digital image from the modified digital image that corrects artifacts around borders of the one or more inpainted portions in the modified digital image.

Abstract: In implementation of techniques for generating a latent space for a three-dimensional scene, a computing device implements a latent space system to receive a three-dimensional representation of a scene. Using a machine learning model, the latent space system encodes a tokenized three-dimensional representation based on the three-dimensional representation of the scene into a latent space. The latent space system then decodes three-dimensional information related to the scene from the latent space using the machine learning model.

Abstract: In implementations of systems for human reposing based on multiple input views, a computing device implements a reposing system to receive input data describing: input digital images; pluralities of keypoints corresponding to the input digital images, the pluralities of keypoints representing poses of a person depicted in the input digital images; and a plurality of keypoints representing a target pose. The reposing system generates selection masks corresponding to the input digital images by processing the input data using a machine learning model. The selection masks represent likelihoods of spatial correspondence between pixels of an output digital image and portions of the input digital images. The reposing system generates the output digital image depicting the person in the target pose for display in a user interface based on the selection masks and the input data.

Abstract: A method, non-transitory computer readable medium, apparatus, and system for image generation are described. An embodiment of the present disclosure includes obtaining an input image, an inpainting mask, and a plurality of content preservation values corresponding to different regions of the inpainting mask, and identifying a plurality of mask bands of the inpainting mask based on the plurality of content preservation values. An image generation model generates an output image based on the input image and the inpainting mask. The output image is generated in a plurality of phases. Each of the plurality of phases uses a corresponding mask band of the plurality of mask bands as an input.

Abstract: The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via scene-based editing using image understanding facilitated by artificial intelligence. For example, in one or more embodiments the disclosed systems utilize generative machine learning models to create modified digital images portraying human subjects. In particular, the disclosed systems generate modified digital images by performing infill modifications to complete a digital image or human inpainting for portions of a digital image that portrays a human. Moreover, in some embodiments, the disclosed systems perform reposing of subjects portrayed within a digital image to generate modified digital images. In addition, the disclosed systems in some embodiments perform facial expression transfer and facial expression animations to generate modified digital images or animations.

Abstract: Systems and methods for training a Generative Adversarial Network (GAN) using feature regularization are described herein. Embodiments are configured to generate a candidate image using a generator network of a GAN, classify the candidate image as real or generated using a discriminator network of the GAN, and train the GAN to generate realistic images based on the classifying of the candidate image. The training process includes regularizing a gradient with respect to features extracted using a discriminator network of the GAN.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for image generation includes obtaining a source image and a lighting input that indicates a lighting condition for the source image. An image generation model generates a relighted foreground image based on the source image. A relighted background image is also generated based on the lighting input. The relighted foreground image depicts a foreground element with the lighting condition and the relighted background image depicts a background element with the lighting condition. The relighted foreground image and the relighted background image are combined to obtain a relighted image, wherein the relighted image depicts the foreground element and the background element with the lighting condition.

Abstract: A modeling system accesses a two-dimensional (2D) input image displayed via a user interface, the 2D input image depicting, at a first view, a first object. At least one region of the first object is not represented by pixel values of the 2D input image. The modeling system generates, by applying a 3D representation generation model to the 2D input image, a three-dimensional (3D) representation of the first object that depicts an entirety of the first object including the first region. The modeling system displays, via the user interface, the 3D representation, wherein the 3D representation is viewable via the user interface from a plurality of views including the first view.

Abstract: Systems and methods for image processing (e.g., image extension or image uncropping) using neural networks are described. One or more aspects include obtaining an image (e.g., a source image, a user provided image, etc.) having an initial aspect ratio, and identifying a target aspect ratio (e.g., via user input) that is different from the initial aspect ratio. The image may be positioned in an image frame having the target aspect ratio, where the image frame includes an image region containing the image and one or more extended regions outside the boundaries of the image. An extended image may be generated (e.g., using a generative neural network), where the extended image includes the image in the image region as well as generated image portions in the extended regions and the one or more generated image portions comprise an extension of a scene element depicted in the image.

Abstract: The present disclosure relates to systems, non-transitory computer-readable media, and methods for utilizing machine learning models to generate modified digital images. In particular, in some embodiments, the disclosed systems generate image editing directions between textual identifiers of two visual features utilizing a language prediction machine learning model and a text encoder. In some embodiments, the disclosed systems generated an inversion of a digital image utilizing a regularized inversion model to guide forward diffusion of the digital image. In some embodiments, the disclosed systems utilize cross-attention guidance to preserve structural details of a source digital image when generating a modified digital image with a diffusion neural network.

Abstract: The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via scene-based editing using image understanding facilitated by artificial intelligence. For example, in one or more embodiments the disclosed systems utilize generative machine learning models to create modified digital images portraying human subjects. In particular, the disclosed systems generate modified digital images by performing infill modifications to complete a digital image or human inpainting for portions of a digital image that portrays a human. Moreover, in some embodiments, the disclosed systems perform reposing of subjects portrayed within a digital image to generate modified digital images. In addition, the disclosed systems in some embodiments perform facial expression transfer and facial expression animations to generate modified digital images or animations.

Abstract: An image processing system obtains an input image (e.g., a user provided image, etc.) and a mask indicating an edit region of the image. A user selects an image editing mode for an image generation network from a plurality of image editing modes. The image generation network generates an output image using the input image, the mask, and the image editing mode.

Abstract: The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify two-dimensional images via scene-based editing using three-dimensional representations of the two-dimensional images. For instance, in one or more embodiments, the disclosed systems utilize three-dimensional representations of two-dimensional images to generate and modify shadows in the two-dimensional images according to various shadow maps. Additionally, the disclosed systems utilize three-dimensional representations of two-dimensional images to modify humans in the two-dimensional images. The disclosed systems also utilize three-dimensional representations of two-dimensional images to provide scene scale estimation via scale fields of the two-dimensional images. In some embodiments, the disclosed systems utilizes three-dimensional representations of two-dimensional images to generate and visualize 3D planar surfaces for modifying objects in two-dimensional images.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for image processing include obtaining an input image and a modification prompt, where the input image depicts an object with a first attribute and the modification prompt describes a modification from the first attribute to a second attribute different from the first attribute, encoding the modification prompt to obtain a text embedding, where the text embedding represents the modification in an embedding space, and generating a modified image based on the input image and the text embedding, where the modified image depicts the object with the second attribute.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for text generation includes obtaining an input image including a first element having a first value of an attribute and a second element having a second value of the attribute. An image encoder of a multi-modal machine learning model then encodes the input image to obtain an image embedding and a text decoder of the multi-modal machine learning model generates an output text based on the image embedding. The output text indicates that the first element has the first value of the attribute and that the second element has the second value of the attribute.

Abstract: An image generation system implements a multi-branch GAN to generate images that each express visually similar content in a different modality. A generator portion of the multi-branch GAN includes multiple branches that are each tasked with generating one of the different modalities. A discriminator portion of the multi-branch GAN includes multiple fidelity discriminators, one for each of the generator branches, and a consistency discriminator, which constrains the outputs generated by the different generator branches to appear visually similar to one another. During training, outputs from each of the fidelity discriminators and the consistency discriminator are used to compute a non-saturating GAN loss. The non-saturating GAN loss is used to refine parameters of the multi-branch GAN during training until model convergence. The trained multi-branch GAN generates multiple images from a single input, where each of the multiple images depicts visually similar content expressed in a different modality.

Abstract: The present disclosure relates to systems, methods, and non-transitory computer-readable media that performs shadow removal and harmonizes lighting properties of a foreground with a background. Furthermore, the disclosed systems receive a shadow removal request for an input digital image that includes a foreground object with a shadow occluding at least part of the foreground object. Moreover, the disclosed systems generate a combined embedding from a mask of the foreground object and the input digital image. Further, the disclosed systems generate a modified digital image without the shadow occluding at least part of the foreground object and lighting properties of the foreground object harmonized with lighting properties of a background.

Abstract: Embodiments include systems and methods for generative image filling based on text and a reference image. In one aspect, the system obtains an input image, a reference image, and a text prompt. Then, the system encodes the reference image to obtain an image embedding and encodes the text prompt to obtain a text embedding. Subsequently, a composite image is generated based on the input image, the image embedding, and the text embedding.

Abstract: Systems and methods for inserting an object into a background are described. Examples of the systems and methods include obtaining a background image including a region for inserting the object, and encoding the background image to obtain an encoded background. A modified image is then generated based on the encoded background using a diffusion model. The modified image depicts the object within the region.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for image generation includes obtaining an input image and an input prompt. In some cases, the input image depicts a scene and the input prompt indicates a target element to be added to the scene. The image generation model generates a normalized output based on the input image and the input prompt by performing a channel shift on a preliminary output of the image generation model. A synthetic image is generated including the scene of the input image and the target element of the input prompt that is harmonized with the scene.