Abstract: System, methods, and computer-readable media for determining metrics associated with simulations of proximity events within autonomous vehicle simulations. In some examples, a simulation may be created using driving log data from an autonomous vehicle. Portions of the simulation containing proximity events may be analyzed. Intent times for both the autonomous vehicle and agent vehicles within the simulation may be determined, and the deceleration required for each vehicle to avoid the proximity event may be calculated. These decelerations may be used to determine metrics for the likeliness of the proximity event and the avoidability of the proximity event by the autonomous vehicle. These metrics may be used to prioritize development tasks, assess performance of the simulation, and assess performance of vehicle control systems.

Abstract: Techniques associated with ingesting data based on the catalog are discussed herein. In some examples, log data associated with a vehicle in an environment can be received. The log data can include at least one of location data, state data, or prediction data. A sequence of data can be identified as corresponding to a driving sequence based on a set of rules. An identification of the driving sequence involving the vehicle in the environment can be associated with the sequence of data in a database. An inquiry for retrieving the sequence of data or information associated with the driving sequence can be received. In response to the inquiry, the sequence of data or information associated with the driving sequence can be returned.

Abstract: Techniques for evaluating and validating progress of training machine-learned models may include storing, in a database, metric data associated with outputs from machine-learned models based on sensor data inputs. For instance, the metric data may include first metric data associated with a first bounding box output by a machine-learned model and second metric data associated with a second bounding box output by an updated version of the machine-learned model. The techniques also include a graphical user interface (GUI) for presenting visualizations of the metric data that improve the ability to evaluate the performance of a machine-learned model. In some examples, an indication of a request to evaluate the updated version of the machine-learned model may be received via the GUI. Based on the indication, the GUI may cause presentation of visualization(s) of difference(s) between first metric(s) of the first metric data and second metric(s) of the second metric data.

Abstract: Techniques for evaluating and validating progress of training machine-learned models are described herein. The techniques may include storing, in a database, metric data associated with outputs from machine-learned models based on sensor data inputs. For instance, the metric data may include first metric data associated with a first bounding box output by a machine-learned model and second metric data associated with a second bounding box output by an updated version of the machine-learned model. The techniques also include a graphical user interface (GUI) for presenting visualizations of the metric data that improve the ability to evaluate the performance of a machine-learned model. In some examples, an indication of a request to evaluate the updated version of the machine-learned model may be received via the GUI. Based on the indication, the GUI may cause presentation of visualization(s) of difference(s) between first metric(s) of the first metric data and second metric(s) of the second metric data.

Abstract: Techniques for validating operation of a perception system that is configured to detect objects in an environment of a vehicle are described herein. The techniques may include receiving log data representing a real scenario in which the vehicle was traversing an environment. Based at least in part on the log data, an error may be identified that is associated with an output received from the perception system while the vehicle was traversing the environment. In some examples, a determination may be made that a magnitude of the error violates a perception system requirement, the requirement established based on a determination that the magnitude of the error would contribute to an adverse event in an alternative scenario. Based on the magnitude of the error contributing to the adverse event, data associated with the error may be output for use in updating the perception system to at least meet the requirement.