Abstract: Systems and methods for training a Generative Adversarial Network (GAN) using feature regularization are described herein. Embodiments are configured to generate a candidate image using a generator network of a GAN, classify the candidate image as real or generated using a discriminator network of the GAN, and train the GAN to generate realistic images based on the classifying of the candidate image. The training process includes regularizing a gradient with respect to features extracted using a discriminator network of the GAN.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for image generation includes obtaining a source image and a lighting input that indicates a lighting condition for the source image. An image generation model generates a relighted foreground image based on the source image. A relighted background image is also generated based on the lighting input. The relighted foreground image depicts a foreground element with the lighting condition and the relighted background image depicts a background element with the lighting condition. The relighted foreground image and the relighted background image are combined to obtain a relighted image, wherein the relighted image depicts the foreground element and the background element with the lighting condition.

Abstract: A modeling system accesses a two-dimensional (2D) input image displayed via a user interface, the 2D input image depicting, at a first view, a first object. At least one region of the first object is not represented by pixel values of the 2D input image. The modeling system generates, by applying a 3D representation generation model to the 2D input image, a three-dimensional (3D) representation of the first object that depicts an entirety of the first object including the first region. The modeling system displays, via the user interface, the 3D representation, wherein the 3D representation is viewable via the user interface from a plurality of views including the first view.

Abstract: Systems and methods for image processing (e.g., image extension or image uncropping) using neural networks are described. One or more aspects include obtaining an image (e.g., a source image, a user provided image, etc.) having an initial aspect ratio, and identifying a target aspect ratio (e.g., via user input) that is different from the initial aspect ratio. The image may be positioned in an image frame having the target aspect ratio, where the image frame includes an image region containing the image and one or more extended regions outside the boundaries of the image. An extended image may be generated (e.g., using a generative neural network), where the extended image includes the image in the image region as well as generated image portions in the extended regions and the one or more generated image portions comprise an extension of a scene element depicted in the image.

Abstract: The present disclosure relates to systems, non-transitory computer-readable media, and methods for utilizing machine learning models to generate modified digital images. In particular, in some embodiments, the disclosed systems generate image editing directions between textual identifiers of two visual features utilizing a language prediction machine learning model and a text encoder. In some embodiments, the disclosed systems generated an inversion of a digital image utilizing a regularized inversion model to guide forward diffusion of the digital image. In some embodiments, the disclosed systems utilize cross-attention guidance to preserve structural details of a source digital image when generating a modified digital image with a diffusion neural network.

Abstract: The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via scene-based editing using image understanding facilitated by artificial intelligence. For example, in one or more embodiments the disclosed systems utilize generative machine learning models to create modified digital images portraying human subjects. In particular, the disclosed systems generate modified digital images by performing infill modifications to complete a digital image or human inpainting for portions of a digital image that portrays a human. Moreover, in some embodiments, the disclosed systems perform reposing of subjects portrayed within a digital image to generate modified digital images. In addition, the disclosed systems in some embodiments perform facial expression transfer and facial expression animations to generate modified digital images or animations.

Abstract: An image processing system obtains an input image (e.g., a user provided image, etc.) and a mask indicating an edit region of the image. A user selects an image editing mode for an image generation network from a plurality of image editing modes. The image generation network generates an output image using the input image, the mask, and the image editing mode.

Abstract: The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify two-dimensional images via scene-based editing using three-dimensional representations of the two-dimensional images. For instance, in one or more embodiments, the disclosed systems utilize three-dimensional representations of two-dimensional images to generate and modify shadows in the two-dimensional images according to various shadow maps. Additionally, the disclosed systems utilize three-dimensional representations of two-dimensional images to modify humans in the two-dimensional images. The disclosed systems also utilize three-dimensional representations of two-dimensional images to provide scene scale estimation via scale fields of the two-dimensional images. In some embodiments, the disclosed systems utilizes three-dimensional representations of two-dimensional images to generate and visualize 3D planar surfaces for modifying objects in two-dimensional images.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for image processing include obtaining an input image and a modification prompt, where the input image depicts an object with a first attribute and the modification prompt describes a modification from the first attribute to a second attribute different from the first attribute, encoding the modification prompt to obtain a text embedding, where the text embedding represents the modification in an embedding space, and generating a modified image based on the input image and the text embedding, where the modified image depicts the object with the second attribute.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for text generation includes obtaining an input image including a first element having a first value of an attribute and a second element having a second value of the attribute. An image encoder of a multi-modal machine learning model then encodes the input image to obtain an image embedding and a text decoder of the multi-modal machine learning model generates an output text based on the image embedding. The output text indicates that the first element has the first value of the attribute and that the second element has the second value of the attribute.

Abstract: An image generation system implements a multi-branch GAN to generate images that each express visually similar content in a different modality. A generator portion of the multi-branch GAN includes multiple branches that are each tasked with generating one of the different modalities. A discriminator portion of the multi-branch GAN includes multiple fidelity discriminators, one for each of the generator branches, and a consistency discriminator, which constrains the outputs generated by the different generator branches to appear visually similar to one another. During training, outputs from each of the fidelity discriminators and the consistency discriminator are used to compute a non-saturating GAN loss. The non-saturating GAN loss is used to refine parameters of the multi-branch GAN during training until model convergence. The trained multi-branch GAN generates multiple images from a single input, where each of the multiple images depicts visually similar content expressed in a different modality.

Abstract: The present disclosure relates to systems, methods, and non-transitory computer-readable media that performs shadow removal and harmonizes lighting properties of a foreground with a background. Furthermore, the disclosed systems receive a shadow removal request for an input digital image that includes a foreground object with a shadow occluding at least part of the foreground object. Moreover, the disclosed systems generate a combined embedding from a mask of the foreground object and the input digital image. Further, the disclosed systems generate a modified digital image without the shadow occluding at least part of the foreground object and lighting properties of the foreground object harmonized with lighting properties of a background.

Abstract: Embodiments include systems and methods for generative image filling based on text and a reference image. In one aspect, the system obtains an input image, a reference image, and a text prompt. Then, the system encodes the reference image to obtain an image embedding and encodes the text prompt to obtain a text embedding. Subsequently, a composite image is generated based on the input image, the image embedding, and the text embedding.

Abstract: Systems and methods for inserting an object into a background are described. Examples of the systems and methods include obtaining a background image including a region for inserting the object, and encoding the background image to obtain an encoded background. A modified image is then generated based on the encoded background using a diffusion model. The modified image depicts the object within the region.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for image generation includes obtaining an input image and an input prompt. In some cases, the input image depicts a scene and the input prompt indicates a target element to be added to the scene. The image generation model generates a normalized output based on the input image and the input prompt by performing a channel shift on a preliminary output of the image generation model. A synthetic image is generated including the scene of the input image and the target element of the input prompt that is harmonized with the scene.

Abstract: A method, apparatus, and non-transitory computer readable medium for image generation are described. Embodiments of the present disclosure obtain a content input and a style input via a user interface or from a database. The content input includes a target spatial layout and the style input includes a target style. A content encoder of an image processing apparatus encodes the content input to obtain a spatial layout mask representing the target spatial layout. A style encoder of the image processing apparatus encodes the style input to obtain a style embedding representing the target style. An image generation model of the image processing apparatus generates an image based on the spatial layout mask and the style embedding, where the image includes the target spatial layout and the target style.

Abstract: In one implementation of subject-aware background video generation, a processing device generates mask data and foreground feature data from frames of a subject video. The mask data separates a subject depicted in the subject video from an environment therein. The foreground feature data describes the features of the subject. The processing device receives a condition frame that depicts a different environment. A machine-learning model generates a composite video by aligning the subject's movement with the different environment from inputs of the foreground feature data, the mask data, and the condition frame, which conditions the generation of the different environment for the composite video. The processing device then presents the composite video via a user interface.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for image generation includes obtaining an object image and a target lighting indicator, generating a shading map based on the object image and the target lighting indicator, and generating a relighted image based on the object image and the shading map. The relighted image depicts an object from the object image with lighting based on the target lighting indicator.

Abstract: Embodiments include systems and methods for generative image filling based on text and a reference image. In one aspect, the system obtains an input image, a reference image, and a text prompt. Then, the system encodes the reference image to obtain an image embedding and encodes the text prompt to obtain a text embedding. Subsequently, a composite image is generated based on the input image, the image embedding, and the text embedding.

Abstract: The present disclosure relates to systems, non-transitory computer-readable media, and methods for utilizing machine learning models to generate modified digital images. In particular, in some embodiments, the disclosed systems generate image editing directions between textual identifiers of two visual features utilizing a language prediction machine learning model and a text encoder. In some embodiments, the disclosed systems generated an inversion of a digital image utilizing a regularized inversion model to guide forward diffusion of the digital image. In some embodiments, the disclosed systems utilize cross-attention guidance to preserve structural details of a source digital image when generating a modified digital image with a diffusion neural network.