Abstract: Aspects of the disclosure provide for controlling an autonomous vehicle. For instance, data identifying an object may be received. A first portion of a trajectory may be generated using a first uncertainty distribution for the object. The first portion of the trajectory may enable the autonomous vehicle to make progress towards a destination of the autonomous vehicle. A fallback portion of the trajectory may be generated using a second uncertainty distribution for the object. The fallback portion may enable the autonomous vehicle to stop. The second uncertainty distribution may be different from the first uncertainty distribution, and the second uncertainty distribution may be based on a predetermined uncertainty distribution if the autonomous vehicle loses a localization improvement process. The autonomous vehicle may be controlled according to the trajectory.