Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting agent response times. One of the methods includes continually updating, at each time step of a plurality of time steps, an accumulated measure of surprise for the agent due to the movements of another entity in the traffic environment. A distribution of previously predicted trajectories is obtained at a previous time step for the other entity in the environment. A measure of surprise is computed from the perspective of the agent. An accumulated measure of surprise is updated for the time step using the computed measure of surprise for the agent. If the accumulated measure of surprise crosses a threshold at a particular point in time, a predicted response time for the agent is generated based on the particular point in time that the accumulated measure of surprise crosses the threshold.