Abstract: A method, apparatus, non-transitory computer readable medium, and system for generating images based on a target prompt and an anchor prompt include obtaining the target prompt the anchor prompt. The target prompt describes a first element, and the anchor prompt describes a second element. A first attention block of an image generation model generates a first attention output based on the target prompt and a second attention block of the image generation model generates a second attention output based on the anchor prompt.

Abstract: The present disclosure relates to systems, non-transitory computer-readable media, and methods for generating synthesized digital images through a conditioned diffusion neural network utilizing an image prompt and a color conditioning input. In some embodiments, the disclosed systems receive an image prompt containing a text description of a digital image and a color conditioning input defining the position of a certain color value from a client device. In some embodiments, the disclosed systems condition a diffusion neural network using the color conditioning input and use the conditioned diffusion neural network to process the image prompt to generate a synthesized digital image correlating with the image prompt and the color conditioning input. In some embodiments, the disclosed systems provide the synthesized digital image for display on a client device.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for image generation include obtaining an object prompt and a background prompt, wherein the object prompt describes an object with a target effect and the background prompt describes a scene. A noise input is generated based on the object prompt and the background prompt, where the noise input indicates a location of the object within the scene. An image generation model generates a synthetic image based on the object prompt, the background prompt, and the noise input. The synthetic image depicts the object at the location within the scene with the target effect applied to the object.

Abstract: Systems and methods for image processing are described. Embodiments of the present disclosure, via a multi-modal encoder of an image processing apparatus, encodes a text prompt to obtain a text embedding. A color encoder of the image processing apparatus encodes a color prompt to obtain a color embedding. A diffusion prior model of the image processing apparatus generates an image embedding based on the text embedding and the color embedding. A latent diffusion model of the image processing apparatus generates an image based on the image embedding, where the image includes an element from the text prompt and a color from the color prompt.

Abstract: An image generation method comprises obtaining a content prompt, a style prompt, and a visual intensity parameter, where the content prompt indicates an object, the style prompt indicates a style, and the visual intensity parameter indicates a level of the style. A content latent code and a style latent code are generated based on the content prompt and the style prompt, respectively, and the content latent code and the style latent code are combined based on the visual intensity parameter to obtain a combined latent code. An image generation model generates a synthetic image based on the combined latent code, where the synthetic image includes the object from the content prompt and the style from the style prompt at the level indicated by the visual intensity parameter.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for generating style-matched images include obtaining a content prompt and a style prompt. The content prompt includes an object and the style prompt includes a style element. Embodiments then encode the content prompt and the style prompt to obtain a content embedding and a style embedding, respectively. Subsequently, embodiments apply a content mask to the content embedding and a style mask to the style embedding to obtain a weighted content embedding and a weighted style embedding, respectively. Embodiments then generate, using an image generation model, a synthetic image based on the weighted content embedding and the weighted style embedding. The synthetic image depicts the object from the content prompt and the style element from the style prompt.

Abstract: A method, apparatus, and non-transitory computer readable medium for image generation are described. Embodiments of the present disclosure obtain a content input and a style input via a user interface or from a database. The content input includes a target spatial layout and the style input includes a target style. A content encoder of an image processing apparatus encodes the content input to obtain a spatial layout mask representing the target spatial layout. A style encoder of the image processing apparatus encodes the style input to obtain a style embedding representing the target style. An image generation model of the image processing apparatus generates an image based on the spatial layout mask and the style embedding, where the image includes the target spatial layout and the target style.

Abstract: The present disclosure relates to systems, methods, and non-transitory computer readable media for utilizing a diffusion neural network for mask aware image and typography editing. For example, in one or more embodiments the disclosed systems utilize a text-image encoder to generate a base image embedding from a base digital image. Moreover, the disclosed systems generate a mask-segmented image by combining a shape mask with the base digital image. In one or more implementations, the disclosed systems utilize noising steps of a diffusion noising model to generate a mask-segmented image noise map from the mask-segmented image. Furthermore, the disclosed systems utilize a diffusion neural network to create a stylized image corresponding to the shape mask from the base image embedding and the mask-segmented image noise map.

Abstract: The present disclosure relates to systems, methods, and non-transitory computer readable media for utilizing a diffusion prior neural network for text guided digital image editing. For example, in one or more embodiments the disclosed systems utilize a text-image encoder to generate a base image embedding from the base digital image and an edit text embedding from edit text. Moreover, the disclosed systems utilize a diffusion prior neural network to generate a text-image embedding. In particular, the disclosed systems inject the base image embedding at a conceptual editing step of the diffusion prior neural network and condition a set of steps of the diffusion prior neural network after the conceptual editing step utilizing the edit text embedding. Furthermore, the disclosed systems utilize a diffusion neural network to create a modified digital image from the text-edited image embedding and the base image embedding.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for image processing include obtaining a structural input indicating a target spatial structure, encoding, using a condition encoder, the structural input to obtain a structural encoding representing the target spatial structure, and generating, using an image generation model, a synthetic image based on the structural encoding, where the synthetic image depicts an object having the target spatial structure.

Abstract: Techniques for generation of images based on a variety of input conditions or modalities are described. In one embodiment, one or more processing devices receive a plurality of input modalities comprising multiple images and a text input in a natural language. The processing devices generate image embeddings for the multiple images and a text embedding for the text input. The processing devices, using a machine learning model, generate an output image based on the image embeddings and the text embedding. The output image includes portions of the multiple images.

Abstract: Systems and methods for image processing are described. Embodiments of the present disclosure include an image generation network configured to encode a plurality of abstract images using a style encoder to obtain a plurality of abstract style encodings, wherein the style encoder is trained to represent image style separately from image content. A clustering component clusters the plurality of abstract style encodings to obtain an abstract style cluster comprising a subset of the plurality of abstract style encodings. A preset component generates an abstract style transfer preset representing the abstract style cluster.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for image generation include obtaining a text prompt and a conditioning attribute. The text prompt is encoded to obtain a text embedding. The conditioning attribute is encoded to obtain an attribute embedding. Then a synthesized image is generated using an image generation model based on the text embedding and the attribute embedding. The synthesized image has the conditioning attribute and depicts an element of the text prompt.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for media processing includes obtaining a text prompt and a style input, where the text prompt describes image content and the style input describes an image style, generating a text embedding based on the text prompt, where the text embedding represents the image content, generating a style embedding based on the style input, where the style embedding represents the image style, and generating a synthetic image based on the text embedding and the style embedding, where the text embedding is provided to the image generation model at a first step and the style embedding is provided to the image generation model at a second step after the first step.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for image generation include encoding a text prompt to obtain a text embedding. An image prompt is encoded to obtain an image embedding. Cross-attention is performed on the text embedding and then on the image embedding to obtain a text attention output and an image attention output, respectively. A synthesized image is generated based on the text attention output and the image attention output.

Abstract: The present disclosure relates to systems, non-transitory computer-readable media, and methods for generating digital images by conditioning a diffusion neural network with input prompts. In particular, in one or more embodiments, the disclosed systems generate, utilizing a reverse diffusion model, an image noise representation from a first image prompt. Additionally, in some embodiments, the disclosed systems generate, utilizing a diffusion neural network conditioned with a first vector representation of the first image prompt, a first denoised image representation from the image noise representation. Moreover, in some embodiments, the disclosed systems generate, utilizing the diffusion neural network conditioned with a second vector representation of a second image prompt, a second denoised image representation from the image noise representation.

Abstract: A method, apparatus, and non-transitory computer readable medium for image generation are described. Embodiments of the present disclosure obtain a content input and a style input via a user interface or from a database. The content input includes a target spatial layout and the style input includes a target style. A content encoder of an image processing apparatus encodes the content input to obtain a spatial layout mask representing the target spatial layout. A style encoder of the image processing apparatus encodes the style input to obtain a style embedding representing the target style. An image generation model of the image processing apparatus generates an image based on the spatial layout mask and the style embedding, where the image includes the target spatial layout and the target style.

Abstract: The present disclosure relates to systems, non-transitory computer-readable media, and methods for selectively conditioning layers of a neural network and utilizing the neural network to generate a digital image. In particular, in some embodiments, the disclosed systems condition an upsampling layer of a neural network with an image vector representation of an image prompt. Additionally, in some embodiments, the disclosed systems condition an additional upsampling layer of the neural network with a text vector representation of a text prompt without the image vector representation of the image prompt. Moreover, in some embodiments, the disclosed systems generate, utilizing the neural network, a digital image from the image vector representation and the text vector representation.

Abstract: Systems and methods for image processing are described. Embodiments of the present disclosure, via a multi-modal encoder of an image processing apparatus, encodes a text prompt to obtain a text embedding. A color encoder of the image processing apparatus encodes a color prompt to obtain a color embedding. A diffusion prior model of the image processing apparatus generates an image embedding based on the text embedding and the color embedding. A latent diffusion model of the image processing apparatus generates an image based on the image embedding, where the image includes an element from the text prompt and a color from the color prompt.

Abstract: Systems and methods for generating abstract backgrounds are described. Embodiments are configured to obtain an input prompt, encode the input prompt to obtain a prompt embedding, and generate a latent vector based on the prompt embedding and a noise vector. Embodiments include a multimodal encoder configured to generate the prompt embedding, which is an intermediate representation the prompt. In some cases, the prompt includes or indicates an “abstract background” type image. The latent vector is generated using a mapping network of a generative adversarial network (GAN). Embodiments are further configured to generate an image based on the latent vector using the GAN.