Abstract: A computing system including one or more processing devices configured to receive an image generation prompt and a reference image. Over a plurality of denoising timesteps, the one or more processing devices compute a guided image by applying denoising updates to a generated image at a denoising diffusion model. At a subset of the denoising timesteps, computing the guided image further includes applying guidance updates to the generated image based on the image generation prompt, the reference image, and a generated image set. The one or more processing devices compute each guidance update by performing a forward pass and a backward pass in first and second integration timesteps. A size of the generated image set and numbers of the first and second integration timesteps are each equal to a predefined integration timestep count. The one or more processing devices output a final generated image computed in a final denoising timestep.

Abstract: A computing system is described herein that implements a diffusion-based framework for animating reference images. The computing system includes a video diffusion model that is utilized to encode temporal information. The computing system further includes a novel appearance encoder that is utilized to retain the intricate details of the reference image and maintain appearance coherence across frames. The computing system further employs a video fusion technique to smooth transitions between animated segments in long video animation. Potential benefits of the computing system include enhanced temporal consistency, faithful preservation of reference images, and improved animation fidelity in the generated animation sequences.

Abstract: Methods of customizing generation of objects using diffusion models are provided. One or more parameters (e.g., a conditioning signal, network weights, or an initial or starting noise) of the diffusion model can be optimized by a backpropagation process, which can be performed by solving an augmented adjoint ordinary differential equation (ODE) based on an adjoint sensitivity method. The customized diffusion model can generate stylized objects, generate objects with specific visual effect(s), and provide adversary examples to audit security of an object generation system.

Abstract: Generating an object using a diffusion model includes obtaining a first input and a second input, and synthesizing an output object from the first input and the second input. The synthesizing of the output object includes generating a layout of the output object from the first input, injecting the second input as a content conditioner to the layout of the output object, and de-noising the layout of the output object injected with the content conditioner to generate a content of the output object.