Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for automatically designating traffic scenarios as safety-relevant traffic conflicts between agents in a driving environment. One of the methods includes receiving data representing a traffic scenario involving two agents; computing a safety-relevant metric for a first plurality of time points of the traffic scenario; computing a surprise metric for a second plurality of time points of the traffic scenario; determining that the surprise metric satisfies a surprise threshold within a threshold time window of the safety-relevant metric satisfying a safety-relevant threshold; and in response, designating the traffic scenario as a safety-relevant traffic conflict.