Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating an output sequence of discrete tokens using a diffusion model. In one aspect, a method includes initializing the output sequence by assigning a respective embedding to each of the plurality of output positions; repeatedly performing the following at each of multiple reverse diffusion steps: a current continuous representation of the output sequence; processing a diffusion model input that comprises the current continuous representation using the diffusion model to generate a diffusion model output; processing the respective initial scores using a softmax function to generate, for each of the plurality of output positions, a probability distribution over the plurality of embeddings in the vocabulary of embeddings; and updating the continuous representation of the output sequence using the probability distributions and the vocabulary of embeddings.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating an output sequence of discrete tokens using a diffusion model. In one aspect, a method includes generating, by using the diffusion model, a final latent representation of the sequence of discrete tokens that includes a determined value for each of a plurality of latent variables; applying a de-embedding matrix to the final latent representation of the output sequence of discrete tokens to generate a de-embedded final latent representation that includes, for each of the plurality of latent variables, a respective numeric score for each discrete token in a vocabulary of multiple discrete tokens; selecting, for each of the plurality of latent variables, a discrete token from among the multiple discrete tokens in the vocabulary that has a highest numeric score; and generating the output sequence of discrete tokens that includes the selected discrete tokens.