Abstract: A method, non-transitory computer readable medium, apparatus, and system for image generation include obtaining an input image having a first resolution, where the input image includes random noise, and generating a low-resolution image based on the input image, where the low-resolution image has the first resolution. The method, non-transitory computer readable medium, apparatus, and system further include generating a high-resolution image based on the low-resolution image, where the high-resolution image has a second resolution that is greater than the first resolution.

Abstract: The present disclosure relates to systems, non-transitory computer-readable media, and methods for inpainting digital images utilizing mask-robust machine-learning models. In particular, in one or more embodiments, the disclosed systems obtain an initial mask for an object depicted in a digital image. Additionally, in some embodiments, the disclosed systems generate, utilizing a mask-robust inpainting machine-learning model, an inpainted image from the digital image and the initial mask. Moreover, in some implementations, the disclosed systems generate a relaxed mask that expands the initial mask. Furthermore, in some embodiments, the disclosed systems generate a modified image by compositing the inpainted image and the digital image utilizing the relaxed mask.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for controllable image synthesis using image elements include obtaining an image depicting a scene and encoding, using an encoder of an image generation model, a first region of the image to obtain a first encoded image element. A transformation is applied to the first encoded image element to obtain a transformed image element, where the transformation modifies an object in the scene located withing the first region of the image. A decoder of the image generation model generates an edited image depicting the scene with the modified object based on the transformed image element.

Abstract: Systems and methods for fine-tuning diffusion models are described. Embodiments of the present disclosure obtain an input text indicating an element to be included in an image; generate a synthetic image depicting the element based on the input text using a diffusion model trained by comparing synthetic images depicting the element to training images depicting elements similar to the element and updating selected parameters corresponding to an attention layer of the diffusion model based on the comparison.

Abstract: Systems and methods for image processing (e.g., image extension or image uncropping) using neural networks are described. One or more aspects include obtaining an image (e.g., a source image, a user provided image, etc.) having an initial aspect ratio, and identifying a target aspect ratio (e.g., via user input) that is different from the initial aspect ratio. The image may be positioned in an image frame having the target aspect ratio, where the image frame includes an image region containing the image and one or more extended regions outside the boundaries of the image. An extended image may be generated (e.g., using a generative neural network), where the extended image includes the image in the image region as well as generated image portions in the extended regions and the one or more generated image portions comprise an extension of a scene element depicted in the image.

Abstract: In some embodiments, a computing system accesses multiple training images of an object for customizing a text-to-image generative model, comprising one or more transformer models and a three-dimensional (3D) feature prediction model. The computing system extracts a training target feature representation based on a training target image using a transformer model. The computing system predicts a training 3D feature representation in a training target camera viewpoint based on a set of training reference images using the 3D feature prediction model. The computing system reconstructs the training target image of the object based on the training 3D feature representation and the training target feature representation. The computing system adjusts one or more parameters of the 3D feature prediction model by optimizing a loss function based on the training target image and the reconstructed training target image to obtain a trained 3D feature prediction model, thereby customizing the text-to-image generative model.

Abstract: An image processing system obtains an input image (e.g., a user provided image, etc.) and a mask indicating an edit region of the image. A user selects an image editing mode for an image generation network from a plurality of image editing modes. The image generation network generates an output image using the input image, the mask, and the image editing mode.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for image generation include obtaining an object prompt and a background prompt, wherein the object prompt describes an object with a target effect and the background prompt describes a scene. A noise input is generated based on the object prompt and the background prompt, where the noise input indicates a location of the object within the scene. An image generation model generates a synthetic image based on the object prompt, the background prompt, and the noise input. The synthetic image depicts the object at the location within the scene with the target effect applied to the object.

Abstract: Various disclosed embodiments are directed to resizing, via down-sampling and up-sampling, a high-resolution input image in order to meet machine learning model low-resolution processing requirements, while also producing a high-resolution output image for image inpainting via a machine learning model. Some embodiments use a refinement model to refine the low-resolution inpainting result from the machine learning model such that there will be clear content with high resolution both inside and outside of the mask region in the output. Some embodiments employ new model architecture for the machine learning model that produces the inpainting result—an advanced Cascaded Modulated Generative Adversarial Network (CM-GAN) that includes Fast Fourier Convolution (FCC) layers at the skip connections between the encoder and decoder.

Abstract: The present disclosure relates to systems, non-transitory computer-readable media, and methods for utilizing machine learning to generate a mask for an object portrayed in a digital image. For example, in some embodiments, the disclosed systems utilize a neural network to generate an image feature representation from the digital image. The disclosed systems can receive a selection input identifying one or more pixels corresponding to the object. In addition, in some implementations, the disclosed systems generate a modified feature representation by integrating the selection input into the image feature representation. Moreover, in one or more embodiments, the disclosed systems utilize an additional neural network to generate a plurality of masking proposals for the object from the modified feature representation. Furthermore, in some embodiments, the disclosed systems utilize a further neural network to generate the mask for the object from the modified feature representation and/or the masking proposals.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for text generation includes obtaining an input image including a first element having a first value of an attribute and a second element having a second value of the attribute. An image encoder of a multi-modal machine learning model then encodes the input image to obtain an image embedding and a text decoder of the multi-modal machine learning model generates an output text based on the image embedding. The output text indicates that the first element has the first value of the attribute and that the second element has the second value of the attribute.

Abstract: The present disclosure relates to systems, non-transitory computer-readable media, and methods for latent-based editing of digital images using a generative neural network. In particular, in one or more embodiments, the disclosed systems perform latent-based editing of a digital image by mapping a feature tensor and a set of style vectors for the digital image into a joint feature style space. In one or more implementations, the disclosed systems apply a joint feature style perturbation and/or modification vectors within the joint feature style space to determine modified style vectors and a modified feature tensor. Moreover, in one or more embodiments the disclosed systems generate a modified digital image utilizing a generative neural network from the modified style vectors and the modified feature tensor.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for assessing perceptual similarity include training an image generation model based on a latent code perceptual similarity by obtaining training data including a first latent code representing a first image and a second latent code representing a second image and encoding, using a perceptual similarity model, the first latent code and the second latent code to obtain a first feature stack and a second feature stack, respectively. The perceptual similarity model generates the latent code perceptual similarity based on the first feature stack and the second feature stack, wherein the latent code perceptual similarity represents a perceptual similarity between the first image and the second image. Then parameters of the image generation model are updated based on the latent code perceptual similarity.

Abstract: Methods, systems, and non-transitory computer readable storage media are disclosed for generating neural network based perceptual artifact segmentations in synthetic digital image content. The disclosed system utilizing neural networks to detect perceptual artifacts in digital images in connection with generating or modifying digital images. The disclosed system determines a digital image including one or more synthetically modified portions. The disclosed system utilizes an artifact segmentation machine-learning model to detect perceptual artifacts in the synthetically modified portion(s). The artifact segmentation machine-learning model is trained to detect perceptual artifacts based on labeled artifact regions of synthetic training digital images. Additionally, the disclosed system utilizes the artifact segmentation machine-learning model in an iterative inpainting process. The disclosed system utilizes one or more digital image inpainting models to inpaint in a digital image.

Abstract: Embodiments include systems and methods for generative image filling based on text and a reference image. In one aspect, the system obtains an input image, a reference image, and a text prompt. Then, the system encodes the reference image to obtain an image embedding and encodes the text prompt to obtain a text embedding. Subsequently, a composite image is generated based on the input image, the image embedding, and the text embedding.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for generating suggested inpainting prompts include first obtaining an image depicting a first element. Embodiments then generate, using an embedding generation model, a text embedding based on the image and a noise input, where the text embedding represents the first element from the first image and a second element generated by the embedding generation model based on the noise input. Subsequently, embodiments generate a text prompt that includes the first element and the second element based on the text embedding.

Abstract: Systems and methods include generating synthetic videos based on a custom motion. A video generation system obtains a text prompt including an object and a custom motion token. The custom motion token represents a custom motion. The system encodes the text prompt to obtain a text embedding. Subsequently, a video generation model generates a synthetic video depicting the object performing the custom motion based on the text embedding using a video generation model.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for generating a seamless version of a coarse edit image includes obtaining a reference image, the coarse edit image, and an occlusion mask. The coarse edit image depicts an object from the reference image at a target position, and the occlusion mask indicates an occluded region of the coarse edit image. Embodiments then extract, using a detail extraction model of an image generation model, detail features from the reference image based on the occlusion mask. Subsequently, embodiments generate, using the image generation model, a synthetic image depicting the object at the target position based on the coarse edit image by performing cross-frame attention between the detail features and the coarse edit image.

Abstract: The present disclosure relates to systems, non-transitory computer-readable media, and methods for inpainting digital images utilizing mask-robust machine-learning models. In particular, in one or more embodiments, the disclosed systems obtain an initial mask for an object depicted in a digital image. Additionally, in some embodiments, the disclosed systems generate, utilizing a mask-robust inpainting machine-learning model, an inpainted image from the digital image and the initial mask. Moreover, in some implementations, the disclosed systems generate a relaxed mask that expands the initial mask. Furthermore, in some embodiments, the disclosed systems generate a modified image by compositing the inpainted image and the digital image utilizing the relaxed mask.

Abstract: Systems and methods for image generation are provided. An aspect of the systems and methods includes obtaining a text prompt, generating a style vector based on the text prompt, generating an adaptive convolution filter based on the style vector, and generating an image corresponding to the text prompt based on the adaptive convolution filter.