Abstract: A fractional training process can be performed training images to an instance of a machine-learned generative image model to obtain a partially trained instance of the model. A fractional optimization process can be performed with the partially trained instance to an instance of a machine-learned three-dimensional (3D) implicit representation model obtain a partially optimized instance of the model. Based on the plurality of training images, pseudo multi-view subject images can be generated with the partially optimized instance of the 3D implicit representation model and a fully trained instance of the generative image model; The partially trained instance of the model can be trained with a set of training data. The partially optimized instance of the machine-learned 3D implicit representation model can be trained with the machine-learned multi-view image model.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a text-to-image model so that the text-to-image model generates images that each depict a variable instance of an object class when the object class without the unique identifier is provided as a text input, and that generates images that each depict a same subject instance of the object class when the unique identifier is provided as the text input.

Abstract: Methods, systems, mobile devices, and non-transitory computer-readable mediums for easily aesthetically enhancing images such as selfies. An example algorithm's input has three parts: image, manipulation magnitude, and text guidance. The algorithm includes two parts: (1) guidance generation based on public and personal aesthetic preferences, and (2) selfie generation. The first part outputs an image to maximize an aesthetic enhancement score (e.g., a beauty score) while following the manipulation input where the output image contains a manipulation direction. The second part is a conditional diffusion model that accepts the rendered output image from the first part and is conditioned on the input image and outputs the final image. The second part is personalized by the user's images.

Abstract: Methods and techniques for manipulating the color of an image based on a text-based description are presented herein. A system can access an input image and an input text. The system can process, using a machine-learned recolorizing model, the input image to generate a recolorized image. A system can determine the similarity between the recolorized image and the input text description using a loss function and pre-trained encoder(s) which have been trained on a large dataset of text and images to convert the text and image inputs into the same embedding space. The system can then modify the one or more parameter values of the machine-learned recolorizing model to minimize the value of the loss function. Thus, after a plurality of iterations, the machine-learned recolorizing model will generate a recolorized photo that matches the description given in the input text.

Abstract: Described herein are techniques for personalizing vision-language models (VLMs) to understand user-specific concepts, without modifying original model weights. Pre-trained VLMs are augmented with external concept heads that identify user-specific concepts in input images. A concept embedding vector is computed to represent the user-specific concept within an intermediate feature space of the VLM through iterative optimization. Then, when processing an input image and the concept is detected, the concept embedding is appended to image features extracted by a vision encoder of the VLM. Personalized textual outputs incorporating the user-specific concept are generated in response to input images and language instructions. Regularization techniques balance attention between the appended concept embedding and original image features, maintaining alignment between outputs and inputs.

Abstract: A fractional training process can be performed training images to an instance of a machine-learned generative image model to obtain a partially trained instance of the model. A fractional optimization process can be performed with the partially trained instance to an instance of a machine-learned three-dimensional (3D) implicit representation model obtain a partially optimized instance of the model. Based on the plurality of training images, pseudo multi-view subject images can be generated with the partially optimized instance of the 3D implicit representation model and a fully trained instance of the generative image model; The partially trained instance of the model can be trained with a set of training data. The partially optimized instance of the machine-learned 3D implicit representation model can be trained with the machine-learned multi-view image model.

Abstract: A media application provides, as input to a diffusion model, an initial image and a request to change a lighting in the initial image, wherein the initial image includes a subject and a sky. The media application outputs, with the diffusion model, an output image that satisfies the request. The media application determines, from the initial image, a sky segment and a subject segment. The media application generates a sky mask that corresponds to the sky segment and a subject mask that corresponds to the subject segment. The media application modifies a coloring of the initial image to match a coloring of the output image. The media application blends the modified initial image with the output image to form a blended image while using the subject mask to prevent modification to the subject from the modified initial image and the sky mask to prevent modification to the sky from the output image during the blending.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a text-to-image model so that the text-to-image model generates images that each depict a variable instance of an object class when the object class without the unique identifier is provided as a text input, and that generates images that each depict a same subject instance of the object class when the unique identifier is provided as the text input.

Abstract: Methods and systems for modifying a digital image are described herein. The method can include performing vividness scoring for a plurality of pixels of the digital image, determining one or more candidate pixels based on the vividness scoring for the plurality of pixels, and agglomerating the one or more candidate pixels into one or more suggested agglomerates. The method can also include determining at least one subject of the digital image, removing at least one agglomerate from the one or more suggested agglomerates based on at least one of the at least one subject of the digital image or one or more characteristics of the at least one agglomerate, generating a modified digital image with the one or more suggested agglomerates modified, and outputting the modified digital image.

Abstract: Some implementations are directed to editing a source image, where the source image is one generated based on processing a source natural language (NL) prompt using a Large-scale language-image (LLI) model. Those implementations edit the source image based on user interface input that indicates an edit to the source NL prompt, and optionally independent of any user interface input that specifies a mask in the source image and/or independent of any other user interface input. Some implementations of the present disclosure are additionally or alternatively directed to applying prompt-to-prompt editing techniques to editing a source image that is one generated based on a real image, and that approximates the real image.

Abstract: A method includes receiving an input image. The input image corresponds to one or more masked regions to be inpainted. The method includes providing the input image to a first neural network. The first neural network outputs a first inpainted image at a first resolution, and the one or more masked regions are inpainted in the first inpainted image. The method includes creating a second inpainted image by increasing a resolution of the first inpainted image from the first resolution to a second resolution. The second resolution is greater than the first resolution such that the one or more inpainted masked regions have an increased resolution. The method includes providing the second inpainted image to a second neural network. The second neural network outputs a first refined inpainted image at the second resolution, and the first refined inpainted image is a refined version of the second inpainted image.

Abstract: Techniques for tuning an image editing operator for reducing a distractor in raw image data are presented herein. The image editing operator can access the raw image data and a mask. The mask can indicate a region of interest associated with the raw image data. The image editing operator can process the raw image data and the mask to generate processed image data. Additionally, a trained saliency model can process at least the processed image data within the region of interest to generate a saliency map that provides saliency values. Moreover, a saliency loss function can compare the saliency values provided by the saliency map for the processed image data within the region of interest to one or more target saliency values. Subsequently, the one or more parameter values of the image editing operator can be modified based at least in part on the saliency loss function.

Abstract: Examples described herein relate to personalized image generation using combined image features. A plurality of input images is provided by a user of an interaction application. Each of the plurality of input images depicts at least part of a subject. Each input image is encoded to obtain an identity representation. The identity representations obtained from the plurality of input images are combined to obtain a combined identity representation associated with the subject. A personalized output image is generated via a generative machine learning model. The generative machine learning model processes the combined identity representation and at least one additional image generation control to generate the personalized output image. At a user device, the personalized output image is presented in a user interface of the interaction application.

Abstract: Some implementations are directed to editing a source image, where the source image is one generated based on processing a source natural language (NL) prompt using a Large-scale language-image (LLI) model. Those implementations edit the source image based on user interface input that indicates an edit to the source NL prompt, and optionally independent of any user interface input that specifies a mask in the source image and/or independent of any other user interface input. Some implementations of the present disclosure are additionally or alternatively directed to applying prompt-to-prompt editing techniques to editing a source image that is one generated based on a real image, and that approximates the real image.

Abstract: Techniques for tuning an image editing operator for reducing a distractor in raw image data are presented herein. The image editing operator can access the raw image data and a mask. The mask can indicate a region of interest associated with the raw image data. The image editing operator can process the raw image data and the mask to generate processed image data. Additionally, a trained saliency model can process at least the processed image data within the region of interest to generate a saliency map that provides saliency values. Moreover, a saliency loss function can compare the saliency values provided by the saliency map for the processed image data within the region of interest to one or more target saliency values. Subsequently, the one or more parameter values of the image editing operator can be modified based at least in part on the saliency loss function.

Abstract: Systems and methods for augmenting data can leverage one or more machine-learned models and contextual attention data to provide more realistic and efficient data augmentation. For example, systems and methods for inpainting can leverage a machine-learned model to generate predicted contextual attention data and blend the predicted contextual attention data with obtained contextual attention data to determine replacement data for augmenting an image to replace one or more occlusions. The obtained contextual attention data can include user-guided contextual attention.

Abstract: Systems and methods for identifying a personalized prior within a generative model's latent vector space based on a set of images of a given subject. In some examples, the present technology may further include using the personalized prior to confine the inputs of a generative model to a latent vector space associated with the given subject, such that when the model is tasked with editing an image of the subject (e.g., to perform inpainting to fill in masked areas, improve resolution, or deblur the image), the subject's identifying features will be reflected in the images the model produces.

Abstract: A method includes obtaining an input image having a region to be inpainted, an indication of the region to be inpainted, and a guide image. The method also includes determining, by an encoder model, a first latent representation of the input image and a second latent representation of the guide image, and generating a combined latent representation based on the first latent representation and the second latent representation. The method additionally includes generating, by a style generative adversarial network model and based on the combined latent representation, an intermediate output image that includes inpainted image content for the region to be inpainted in the input image. The method further includes generating, based on the input image, the indication of the region, and the intermediate output image, an output image representing the input image with the region to be inpainted including the inpainted image content from the intermediate output image.

Abstract: Systems and methods for identifying a personalized prior within a generative model's latent vector space based on a set of images of a given subject. In some examples, the present technology may further include using the personalized prior to confine the inputs of a generative model to a latent vector space associated with the given subject, such that when the model is tasked with editing an image of the subject (e.g., to perform inpainting to fill in masked areas, improve resolution, or deblur the image), the subject's identifying features will be reflected in the images the model produces.

Abstract: A fractional training process can be performed training images to an instance of a machine-learned generative image model to obtain a partially trained instance of the model. A fractional optimization process can be performed with the partially trained instance to an instance of a machine-learned three-dimensional (3D) implicit representation model obtain a partially optimized instance of the model. Based on the plurality of training images, pseudo multi-view subject images can be generated with the partially optimized instance of the 3D implicit representation model and a fully trained instance of the generative image model; The partially trained instance of the model can be trained with a set of training data. The partially optimized instance of the machine-learned 3D implicit representation model can be trained with the machine-learned multi-view image model.