Abstract: Embodiments of a generative framework comprise cooperative learning of two generative flow models, in which the two models are iteratively updated based on the jointly synthesized examples. In one or more embodiments, the first flow model is a normalizing flow that transforms an initial simple density into a target density by applying a sequence of invertible transformations, and the second flow model is a Langevin flow that runs finite steps of gradient-based MCMC toward an energy-based model. In learning iterations, synthesized examples are generated by using a normalizing flow initialization followed by a short-run Langevin flow revision toward the current energy-based model. Then, the synthesized examples may be treated as fair samples from the energy-based model and the model parameters are updated, while the normalizing flow directly learns from the synthesized examples by maximizing the tractable likelihood. Also provided are both theoretical and empirical justifications for the embodiments.

Abstract: Training energy-based models (EBMs) by maximum likelihood may require MCMC sampling to approximate the gradient of Kullback-Leibler divergence between data and model distributions, but it is non-trivial to sample from an EBM because of the difficulty of mixing between modes. The present disclosure discloses embodiments to learn a variational auto-encoder (VAE) to initialize a finite-step MCMC derived from an energy function for efficient amortized sampling of the EBM. With these amortized MCMC samples, the EBM may be trained by maximum likelihood, which follows an “analysis by synthesis” scheme; while the VAE learns from MCMC samples via variational Bayes. In this joint training process, the VAE chases the EBM toward data distribution. The learning methodology may be interpreted as a dynamic alternating projection in the context of information geometry.

Abstract: Learning latent variable models with deep top-down architectures typically requires inferring latent variables for each training example based on posterior distribution of these latent variables. The inference step relies on either time-consuming long-run Markov chain Monte Carlo (MCMC) sampling or a separate inference model for variational learning. Embodiments of a short-run MCMC, such as a short-run Langevin dynamics, are used herein as an approximate flow-based inference engine. Bias existing in the output distribution of non-convergent short-run Langevin dynamics may be corrected by optimal transport (OT), which aims at transforming the biased distribution produced by finite-step MCMC to the prior distribution with a minimum transport cost.

Abstract: Different from prior works that model the internal distribution of patches within an image implicitly with a top-down latent variable model (e.g., generator), embodiments explicitly represent the statistical distribution within a single image by using an energy-based generative framework, where a pyramid of energy functions, each parameterized by a bottom-up deep neural network, are used to capture the distributions of patches at different resolutions. Also, embodiments of a coarse-to-fine sequential training and sampling strategy are presented to train the model efficiently. Besides learning to generate random samples from white noise, embodiments can learn in parallel with a self-supervised task (e.g., recover an input image from its corrupted version), which can further improve the descriptive power of the learned model. Embodiments does not require an auxiliary model (e.g.

Abstract: Presented herein are embodiments of energy-based models (EBMs), which may be trained via embodiments of a multistage coarse-to-fine expanding and sampling strategy. Embodiments of the training methodology start with learning a coarse-level EBM from images at low resolution and then gradually transits to learn a finer-level EBM from images at higher resolution by expanding the energy function as the learning progresses. Embodiments are computationally efficient with smooth learning and sampling. Tested embodiments achieved the best performance on image generation amongst all EBMs and successfully synthesized high-fidelity images. Embodiments may also be used for image restoration and out-of-distribution detection. Framework embodiments may be further generalized for one-sided unsupervised image-to-image translation and beat baseline methods in terms of model size and training budget. Also presented herein are embodiments of a gradient-based generative saliency methodology to interpret the translation dynamics.