Abstract: Diffusion for realistic scene generation includes obtaining a current set of agent state vectors and a map data of a geographic region, and iteratively, through multiple diffusion timesteps, updating the current set of agent state vectors. Iteratively updating includes processing, by a noise prediction model, the current set of agent state vectors, a current diffusion timestep of the plurality of diffusion timesteps, and the map data to obtain a noise prediction value, generating a mean using the noise prediction value, generating a distribution function according to the mean, sampling a revised set of agent state vectors from the distribution function, and replacing the current set of agent state vectors with the revised set of agent state vectors. The current set of agent state vectors are outputted.