Abstract: A scene simulation system can use scene data of a scene of a vehicle to generate one or more simulated states and one or more simulated trajectories associated with the one or more simulated states. The system can evaluate the simulated trajectories and select an action for the vehicle based on the evaluation of the simulated trajectories.

Abstract: Techniques for generation of an optimal trajectory for a vehicle include receiving, using one or more processors of the vehicle, an instruction for the vehicle to travel from an initial spatiotemporal location to a destination spatiotemporal location. One or more processors are used to generate a trajectory including travel segments. The trajectory begins at the initial spatiotemporal location and terminates at the destination spatiotemporal location. Each travel segment begins at a first spatiotemporal location and terminates at a second spatiotemporal location. Each travel segment is associated with operational metrics. The operational metrics are associated with navigating the vehicle from the first spatiotemporal location to the second spatiotemporal location. Each operational metric is optimized across the travel segments to generate the trajectory.