Abstract: Techniques for determining a metrics associated with a vehicle are disclosed. The techniques may comprise receiving a driving simulation scenario associated with a simulated vehicle. Based at least in part on the scenario and a dataset of recorded driving associated with operation of a test vehicle, a plurality of driving events present in the dataset may be determined. Based at least in part on the plurality of driving events, a first metric associated with a likelihood of occurrence of the driving simulation scenario may be determined. A simulation may be instantiated based at least in part on the driving simulation scenario. Based at least in part on the simulation, a safety metric indicative of the safe performance of the simulated vehicle in the simulation may be determined. An aggregate safety metric may be determined based at least in part on the safety metric and the first metric.

Abstract: An error estimation component may generate contours or a heat map of a likelihood that an object detection or prediction generated by a perception component of an autonomous vehicle is inaccurate. The contours or heat map may be used to lock or unlock actions available to a planning component and the planning component may use the contours or heat for determining a trajectory for controlling the autonomous vehicle. The planning component may additionally or alternatively alter operation of a sensor or determine a trajectory to unlock a target action that is currently locked due to a size/extent of uncertainty indicated by a contour/heat map.

Abstract: Fast simulation of a scenario (e.g., simulating the scenario once as opposed to multiple times) to determine performance metric(s) of a configuration of one or more components of an autonomous vehicle may include training a perception error model based at least in part on a difference between a prediction output by a perception component associated with a future time and a perception output associated with that future time once that future time has arrived. A contour or heat map output by the perception error model may be used to determine one or more performance metric(s) associated with a component of the autonomous vehicle and identify which component may cause a degradation of a performance metric.

Abstract: Techniques associated with generating simulation scenarios for simulating a vehicle controller are discussed herein. Log data may include sensor data captured by sensors of a vehicle. The log data may represent objects in an environment. Objects may be associated with a region of a discretized representation of the environment relative to the vehicle. Specific states of objects (relative position in a region type, velocity, classification, size, etc.) may represent an instance of an occupation. Log data can be aggregated based on similar region type and/or object state. A statistical model over object states can be determined for each region type and can later be sampled to determine simulation parameters. A simulation scenario can be generated based on the simulation parameters, and a vehicle controller can be evaluated based on the simulation scenario.

Abstract: Techniques associated with generating simulation scenarios for simulating a vehicle controller are discussed herein. Log data may include sensor data captured by sensors of a vehicle. The log data may represent objects in an environment. Objects may be associated with a region of a discretized representation of the environment relative to the vehicle. Specific states of objects (relative position in a region type, velocity, classification, size, etc.) may represent an instance of an occupation. Log data can be aggregated based on similar region type and/or object state. A statistical model over object states can be determined for each region type and can later be sampled to determine simulation parameters. A simulation scenario can be generated based on the simulation parameters, and a vehicle controller can be evaluated based on the simulation scenario.