Abstract: Techniques for analyzing driving simulations and determining simulation stability metrics are discussed herein. Various techniques may include receiving a driving simulation and determining a number of perturbed simulations based on the driving simulation. The perturbed simulations may include, for example, one or more perturbed object attributes, perturbed simulation failure conditions, and/or perturbed behaviors of the vehicle system on which the simulation is executed. The perturbed simulations may be executed, and the results may be analyzed to determine a stability score associated with the driving simulation. Stability scores for driving simulations may be determined and maintained based on different simulation data perturbations received based on perturbation models, and the stability scores may be used in various implementations to generate simulation batteries for regression testing and for analyzing and routing simulation failures.

Abstract: Systems and techniques are provided for testing software. A method can generate, based on sensor data reflecting a pose/behavior of an autonomous vehicle (AV) during an event, a simulation of a trip by the AV comprising an evaluation window for testing a simulated pose/behavior of the AV after a software modification; iteratively adjust the time interval to yield an evaluation window for each iteration of adjustments to the time interval, each evaluation window including a time interval including the event and an evaluation start and end; select one of the evaluation windows based on a comparison of AV metrics from each respective simulation of the AV during the evaluation windows and divergences in an AV pose/behavior during each evaluation window and a pose/behavior of the AV during the trip; simulate a pose/behavior of the AV during the evaluation window; and simulate a performance of the AV during the evaluation window.