Abstract: A method, apparatus, non-transitory computer readable medium, and system for image generation includes obtaining an input image and an input prompt. In some cases, the input image depicts a scene and the input prompt indicates a target element to be added to the scene. The image generation model generates a normalized output based on the input image and the input prompt by performing a channel shift on a preliminary output of the image generation model. A synthetic image is generated including the scene of the input image and the target element of the input prompt that is harmonized with the scene.

Abstract: The present disclosure relates to systems, methods, and non-transitory computer-readable media that performs shadow removal and harmonizes lighting properties of a foreground with a background. Furthermore, the disclosed systems receive a shadow removal request for an input digital image that includes a foreground object with a shadow occluding at least part of the foreground object. Moreover, the disclosed systems generate a combined embedding from a mask of the foreground object and the input digital image. Further, the disclosed systems generate a modified digital image without the shadow occluding at least part of the foreground object and lighting properties of the foreground object harmonized with lighting properties of a background.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for multimodal generation include obtaining an input image and an image generation prompt, where the input image depicts a view of an object and the image generation prompt describes a different view of the object, generating a set of custom tokens based on the input image, where the set of custom tokens includes an image generation token and an image understanding token and generating a synthetic image based on the image generation prompt and the image generation token, where the synthetic image depicts the different view of the object described by the image generation prompt and preserves an identity of the object from the input image.

Abstract: In implementation of techniques for generating a latent space for a three-dimensional scene, a computing device implements a latent space system to receive a three-dimensional representation of a scene. Using a machine learning model, the latent space system encodes a tokenized three-dimensional representation based on the three-dimensional representation of the scene into a latent space. The latent space system then decodes three-dimensional information related to the scene from the latent space using the machine learning model.

Abstract: In implementations of techniques and systems for bounding box motion control for image-to-video generation, a processing device receives an input defining an initial location in an initial video frame and a target location in a target video frame. The processing device determines, for a set of intermediary video frames, a set of intermediary locations as exhibiting movement between the initial location and the target location. The processing device generates a digital video that includes an object as disposed at the initial location in the initial video frame, the set of intermediary locations in the set of intermediary video frames, and the target location in the target video frame using a transformer-based diffusion model.

Abstract: Systems and methods for training a Generative Adversarial Network (GAN) using feature regularization are described herein. Embodiments are configured to generate a candidate image using a generator network of a generative adversarial network (GAN), and classify the candidate image as real or generated using a discriminator network of the GAN. Embodiments then train the GAN to generate realistic images based on the classifying of the candidate image. The training includes regularizing a gradient with respect to features generated by the GAN.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for generating a response includes obtaining an input prompt that includes an input media description. A language generation model generates a text token based on the input prompt. Subsequently, a media generation model generates a synthetic media based on the input prompt, wherein the synthetic media depicts a scene corresponding to the input description. A response to the input prompt generated based on the text token and the synthetic media is presented to the user.

Abstract: An image processing system obtains an input image (e.g., a user provided image, etc.) and a mask indicating an edit region of the image. A user selects an image editing mode for an image generation network from a plurality of image editing modes. The image generation network generates an output image using the input image, the mask, and the image editing mode.

Abstract: A method, apparatus, non-transitory computer readable medium, apparatus, and system for image processing include obtaining a text prompt and an input image depicting a person, generating a latent code based on the text prompt and the input image, wherein the latent code is optimized by an identity preserving loss, and generating, using an image generator of a machine learning model, a synthetic image based on the latent code, wherein the synthetic image includes an element of the text prompt and preserves an identity of the person in the input image.

Abstract: The present disclosure relates to systems, methods, and non-transitory computer-readable media that generates a refined digital image that corrects inpainting artifacts in a modified digital image with inpainted pixels. Furthermore, the disclosed systems receive an inpainting request to inpaint a region in a digital image (e.g., a region indicated by a mask). Moreover, the disclosed systems generate a modified digital image from the inpainting request and the digital image, where the modified digital image includes one or more inpainted portions in place of the region. Further, the disclosed systems use a neural network-based refiner model to generate a refined digital image from the modified digital image that corrects artifacts around borders of the one or more inpainted portions in the modified digital image.

Abstract: In implementation of techniques for generating a latent space for a three-dimensional scene, a computing device implements a latent space system to receive a three-dimensional representation of a scene. Using a machine learning model, the latent space system encodes a tokenized three-dimensional representation based on the three-dimensional representation of the scene into a latent space. The latent space system then decodes three-dimensional information related to the scene from the latent space using the machine learning model.

Abstract: In implementations of systems for human reposing based on multiple input views, a computing device implements a reposing system to receive input data describing: input digital images; pluralities of keypoints corresponding to the input digital images, the pluralities of keypoints representing poses of a person depicted in the input digital images; and a plurality of keypoints representing a target pose. The reposing system generates selection masks corresponding to the input digital images by processing the input data using a machine learning model. The selection masks represent likelihoods of spatial correspondence between pixels of an output digital image and portions of the input digital images. The reposing system generates the output digital image depicting the person in the target pose for display in a user interface based on the selection masks and the input data.

Abstract: A method, non-transitory computer readable medium, apparatus, and system for image generation are described. An embodiment of the present disclosure includes obtaining an input image, an inpainting mask, and a plurality of content preservation values corresponding to different regions of the inpainting mask, and identifying a plurality of mask bands of the inpainting mask based on the plurality of content preservation values. An image generation model generates an output image based on the input image and the inpainting mask. The output image is generated in a plurality of phases. Each of the plurality of phases uses a corresponding mask band of the plurality of mask bands as an input.

Abstract: The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via scene-based editing using image understanding facilitated by artificial intelligence. For example, in one or more embodiments the disclosed systems utilize generative machine learning models to create modified digital images portraying human subjects. In particular, the disclosed systems generate modified digital images by performing infill modifications to complete a digital image or human inpainting for portions of a digital image that portrays a human. Moreover, in some embodiments, the disclosed systems perform reposing of subjects portrayed within a digital image to generate modified digital images. In addition, the disclosed systems in some embodiments perform facial expression transfer and facial expression animations to generate modified digital images or animations.

Abstract: Systems and methods for training a Generative Adversarial Network (GAN) using feature regularization are described herein. Embodiments are configured to generate a candidate image using a generator network of a GAN, classify the candidate image as real or generated using a discriminator network of the GAN, and train the GAN to generate realistic images based on the classifying of the candidate image. The training process includes regularizing a gradient with respect to features extracted using a discriminator network of the GAN.

Abstract: A method, apparatus, non-transitory computer readable medium, and system for image generation includes obtaining a source image and a lighting input that indicates a lighting condition for the source image. An image generation model generates a relighted foreground image based on the source image. A relighted background image is also generated based on the lighting input. The relighted foreground image depicts a foreground element with the lighting condition and the relighted background image depicts a background element with the lighting condition. The relighted foreground image and the relighted background image are combined to obtain a relighted image, wherein the relighted image depicts the foreground element and the background element with the lighting condition.

Abstract: A modeling system accesses a two-dimensional (2D) input image displayed via a user interface, the 2D input image depicting, at a first view, a first object. At least one region of the first object is not represented by pixel values of the 2D input image. The modeling system generates, by applying a 3D representation generation model to the 2D input image, a three-dimensional (3D) representation of the first object that depicts an entirety of the first object including the first region. The modeling system displays, via the user interface, the 3D representation, wherein the 3D representation is viewable via the user interface from a plurality of views including the first view.

Abstract: Systems and methods for image processing (e.g., image extension or image uncropping) using neural networks are described. One or more aspects include obtaining an image (e.g., a source image, a user provided image, etc.) having an initial aspect ratio, and identifying a target aspect ratio (e.g., via user input) that is different from the initial aspect ratio. The image may be positioned in an image frame having the target aspect ratio, where the image frame includes an image region containing the image and one or more extended regions outside the boundaries of the image. An extended image may be generated (e.g., using a generative neural network), where the extended image includes the image in the image region as well as generated image portions in the extended regions and the one or more generated image portions comprise an extension of a scene element depicted in the image.

Abstract: The present disclosure relates to systems, non-transitory computer-readable media, and methods for utilizing machine learning models to generate modified digital images. In particular, in some embodiments, the disclosed systems generate image editing directions between textual identifiers of two visual features utilizing a language prediction machine learning model and a text encoder. In some embodiments, the disclosed systems generated an inversion of a digital image utilizing a regularized inversion model to guide forward diffusion of the digital image. In some embodiments, the disclosed systems utilize cross-attention guidance to preserve structural details of a source digital image when generating a modified digital image with a diffusion neural network.

Abstract: The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via scene-based editing using image understanding facilitated by artificial intelligence. For example, in one or more embodiments the disclosed systems utilize generative machine learning models to create modified digital images portraying human subjects. In particular, the disclosed systems generate modified digital images by performing infill modifications to complete a digital image or human inpainting for portions of a digital image that portrays a human. Moreover, in some embodiments, the disclosed systems perform reposing of subjects portrayed within a digital image to generate modified digital images. In addition, the disclosed systems in some embodiments perform facial expression transfer and facial expression animations to generate modified digital images or animations.