Abstract: In various examples, systems and methods are disclosed relating to generating a simulated environment and update a machine learning model to move each of a plurality of human characters having a plurality of body shapes, to follow a corresponding trajectory within the simulated environment as conditioned on a respective body shape. The simulated human characters can have diverse characteristics (such as gender, body proportions, body shape, and so on) as observed in real-life crowds. A machine learning model can determine an action for a human character in a simulated environment, based at least on a humanoid state, a body shape, and task-related features. The task-related features can include an environmental feature and a trajectory.