Abstract: A method, apparatus, non-transitory computer readable medium, and system for image processing include obtaining an input image and a text prompt, wherein the input image depicts a first image element and the text prompt describes a second image element, generating a multimodal embedding based on the input image and the text prompt, wherein the multimodal embedding represents the first image element and the second image element in a multimodal embedding space, generating a guidance embedding based on the multimodal embedding, wherein the guidance embedding represents the first image element and the second image element in a guidance embedding space different from the multimodal embedding space, and generating a synthetic image based on the guidance embedding, wherein the synthetic image depicts the first image element and the second image element.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for image processing include obtaining a text prompt describing an image element, generating an image embedding based on the text prompt, where the image embedding represents visual features of the image element, and generating a synthetic image depicting the image element based on the image embedding.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for image processing include obtaining a text prompt describing a story, generating a first scene prompt and a second scene prompt based on the text prompt, where the first scene prompt describes a first scene of the story and the second scene prompt describes a second scene of the story, and generating a first synthetic image and a second synthetic image based on the first scene prompt and the second scene prompt, respectively, where the first synthetic image depicts the first scene and the second synthetic image depicts the second scene.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for image generation include obtaining an object prompt and a background prompt, wherein the object prompt describes an object with a target effect and the background prompt describes a scene. A noise input is generated based on the object prompt and the background prompt, where the noise input indicates a location of the object within the scene. An image generation model generates a synthetic image based on the object prompt, the background prompt, and the noise input. The synthetic image depicts the object at the location within the scene with the target effect applied to the object.

Abstract: Methods, systems, and non-transitory computer readable media are disclosed for accurately and efficiently generating groups of images portraying semantically similar objects for utilization in building machine learning models. In particular, the disclosed system utilizes metadata and spatial statistics to extract semantically similar objects from a repository of digital images. In some embodiments, the disclosed system generates color embeddings and content embeddings for the identified objects. The disclosed system can further group similar objects together within a query space by utilizing a clustering algorithm to create object clusters and then refining and combining the object clusters within the query space. In some embodiments, the disclosed system utilizes one or more of the object clusters to build a machine learning model.

Abstract: Certain aspects and features of the present disclosure relate to providing interactive diffusion-based texture editing. For example, one or more textual prompts corresponding to an appearance of a texture can be provided. For example, a method involves accessing a texture image and a textual prompt corresponding to the texture image. The method further involves computing, using an image-conditioned diffusion model, image embeddings corresponding to the textual prompt. The method also involves defining, using the image embeddings, a varying appearance of the texture image. The varying appearance corresponds to the textual prompt. The method additionally involves presenting the varying appearance of the texture image for display in an interactive texture editing element.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for image generation include obtaining a text embedding of a text prompt and an image embedding of an image prompt. Some embodiments map the text embedding into a joint embedding space to obtain a joint text embedding and map the image embedding into the joint embedding space to obtain a joint image embedding. Some embodiments generate a synthetic image based on the joint text embedding and the joint image embedding.

Abstract: Systems and methods for image processing are described. Embodiments of the present disclosure, via a multi-modal encoder of an image processing apparatus, encodes a text prompt to obtain a text embedding. A color encoder of the image processing apparatus encodes a color prompt to obtain a color embedding. A diffusion prior model of the image processing apparatus generates an image embedding based on the text embedding and the color embedding. A latent diffusion model of the image processing apparatus generates an image based on the image embedding, where the image includes an element from the text prompt and a color from the color prompt.

Abstract: Systems and methods for generating tile-able patterns from text include obtaining a text prompt and generating, by a generation prior model, a latent vector based on the text prompt, where the generation prior model is trained to output vectors within a distribution of tile-able patterns. An image generation model then generates an output image based on the latent vector. The output image comprises a tile-able pattern including an element from the text prompt.

Abstract: A method, apparatus, and non-transitory computer readable medium for image generation are described. Embodiments of the present disclosure obtain a content input and a style input via a user interface or from a database. The content input includes a target spatial layout and the style input includes a target style. A content encoder of an image processing apparatus encodes the content input to obtain a spatial layout mask representing the target spatial layout. A style encoder of the image processing apparatus encodes the style input to obtain a style embedding representing the target style. An image generation model of the image processing apparatus generates an image based on the spatial layout mask and the style embedding, where the image includes the target spatial layout and the target style.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for image generation include obtaining an input prompt, a reference image, and a transform input. The input prompt describes a scene, the reference image depicts an object, and the transform input indicates a target level of a transformation for the object. An object embedding is generated, using an object encoder of an image generation model, based on the reference image and the transform input. The object embedding represents the object and the target level of the transformation. A synthetic image is generated, using the image generation model, based on the input prompt and the object embedding. The synthetic image depicts the object in the scene from the input prompt with the target level of the transformation.

Abstract: The present disclosure relates to systems, methods, and non-transitory computer readable media for utilizing a diffusion neural network for mask aware image and typography editing. For example, in one or more embodiments the disclosed systems utilize a text-image encoder to generate a base image embedding from a base digital image. Moreover, the disclosed systems generate a mask-segmented image by combining a shape mask with the base digital image. In one or more implementations, the disclosed systems utilize noising steps of a diffusion noising model to generate a mask-segmented image noise map from the mask-segmented image. Furthermore, the disclosed systems utilize a diffusion neural network to create a stylized image corresponding to the shape mask from the base image embedding and the mask-segmented image noise map.

Abstract: The present disclosure relates to systems, methods, and non-transitory computer readable media for utilizing a diffusion prior neural network for text guided digital image editing. For example, in one or more embodiments the disclosed systems utilize a text-image encoder to generate a base image embedding from the base digital image and an edit text embedding from edit text. Moreover, the disclosed systems utilize a diffusion prior neural network to generate a text-image embedding. In particular, the disclosed systems inject the base image embedding at a conceptual editing step of the diffusion prior neural network and condition a set of steps of the diffusion prior neural network after the conceptual editing step utilizing the edit text embedding. Furthermore, the disclosed systems utilize a diffusion neural network to create a modified digital image from the text-edited image embedding and the base image embedding.

Abstract: Systems and methods for image processing, and specifically for generating object-agnostic image representations, are described. Embodiments of the present disclosure receive a training image including a foreground object and a background, remove the foreground object from the training image to obtain a modified training image, inpaint a portion of the modified training image corresponding to the foreground object to obtain an inpainted training image, encode the training image and the inpainted training image using a machine learning model to obtain an encoded training image and an encoded inpainted training image, and update parameters of the machine learning model based on the encoded training image and the encoded inpainted training image.

Abstract: A method, apparatus, non-transitory computer readable medium, apparatus, and system for image processing include obtaining a plurality of images and a plurality of tags, wherein each of the plurality of tags represents a corresponding element of at least one of the plurality of images, computing a plurality of image-tag similarity scores, wherein each of the plurality of image-tag similarity scores indicate a similarity between one of the plurality of images and one of the plurality of tags, computing a plurality of classification scores corresponding to the plurality of tags, respectively, by averaging a subset of the plurality of image-tag similarity scores corresponding to each of the plurality of tags, and selecting a representative tag for the plurality of images based on the representative tag having a highest classification score among the plurality of classification scores.

Abstract: Systems and methods for image processing are described. Embodiments of the present disclosure obtain a text prompt for text guided image generation. A multi-modal encoder of an image processing apparatus encodes the text prompt to obtain a text embedding. A diffusion prior model of the image processing apparatus converts the text embedding to an image embedding. A latent diffusion model of the image processing apparatus generates an image based on the image embedding, wherein the image includes an element described by the text prompt.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for image generation includes obtaining an input image and an input prompt, where the input image depicts an object and the input prompt describes a lighting condition for the object, generating relighted image features based on the input image and the input prompt, where the relighted image features represent the object with the lighting condition, and generating a synthetic image based on the relighted image features, where the synthetic image depicts the object with the lighting condition.

Abstract: Techniques for generation of images based on a variety of input conditions or modalities are described. In one embodiment, one or more processing devices receive a plurality of input modalities comprising multiple images and a text input in a natural language. The processing devices generate image embeddings for the multiple images and a text embedding for the text input. The processing devices, using a machine learning model, generate an output image based on the image embeddings and the text embedding. The output image includes portions of the multiple images.

Abstract: Systems and methods for image processing are described. Embodiments of the present disclosure include an image generation network configured to encode a plurality of abstract images using a style encoder to obtain a plurality of abstract style encodings, wherein the style encoder is trained to represent image style separately from image content. A clustering component clusters the plurality of abstract style encodings to obtain an abstract style cluster comprising a subset of the plurality of abstract style encodings. A preset component generates an abstract style transfer preset representing the abstract style cluster.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for image processing include obtaining a reference image an input prompt describing an image element, identifying an object from the reference image; generating, using an image generation model, image features representing the object based on the reference image, and generating, using the image generation model, a synthetic image depicting the image element and the object based on the input prompt and the image features from the reference image.