Abstract: Techniques for determining whether a machine-learned model has improved or regressed in response to an update, as well as verifying whether those improvements or regressions impact overall vehicle safety, are described herein. The techniques may include running a simulation in which a simulated vehicle traverses a simulated environment. During the simulation, sensor data associated with the simulated environment may be input to the machine-learned model, which is configured for use in the real vehicle. As such, perception data outputs associated with objects detected in the simulated environment may be received from the machine-learned model during the simulation, and one or more error models may be generated based on the outputs and a ground truth. If the error model indicates that an error meets or exceeds a threshold error, the machine-learned model may be updated to reduce the error below the threshold.

Abstract: Techniques associated with ingesting data based on the catalog are discussed herein. In some examples, log data associated with a vehicle in an environment can be received. The log data can include at least one of location data, state data, or prediction data. A sequence of data can be identified as corresponding to a driving sequence based on a set of rules. An identification of the driving sequence involving the vehicle in the environment can be associated with the sequence of data in a database. An inquiry for retrieving the sequence of data or information associated with the driving sequence can be received. In response to the inquiry, the sequence of data or information associated with the driving sequence can be returned.

Abstract: Techniques for correlating perception system errors with unwanted vehicle behavior. The techniques may include receiving log data associated with a vehicle traversing an environment. Based at least in part on the log data, an error associated with an output received from a perception component of the vehicle may be identified. The output of the perception component may be associated with a detection of an object in the environment. The techniques may also include determining that the error contributed to an unwanted behavior of the vehicle. In some examples, a simulation of a planner component of the vehicle may be ran using ground truth data to determine that the error contributed to the unwanted behavior. Based at least in part on the error contributing to the unwanted behavior, a parameter of the perception component may be updated to minimize the error and the unwanted behavior.

Abstract: Techniques for validating operation of a perception system that is configured to detect objects in an environment of a vehicle are described herein. The techniques may include receiving log data representing a real scenario in which the vehicle was traversing an environment. Based at least in part on the log data, an error may be identified that is associated with an output received from the perception system while the vehicle was traversing the environment. In some examples, a determination may be made that a magnitude of the error violates a perception system requirement, the requirement established based on a determination that the magnitude of the error would contribute to an adverse event in an alternative scenario. Based on the magnitude of the error contributing to the adverse event, data associated with the error may be output for use in updating the perception system to at least meet the requirement.

Abstract: Techniques for determining error models for use in simulations are discussed herein. Ground truth perception data and vehicle perception data can be determined from vehicle log data. Further, objects in the log data can be identified as relevant objects by signals output by a planner system or based on the object being located in a driving corridor. Differences between the ground truth perception data and the vehicle perception data can be determined and used to generate error models for the relevant objects. The error models can be applied to objects during simulation to increase realism and test vehicle components.

Abstract: System, methods, and other embodiments described herein relate to improving clustering of points within a point cloud. In one embodiment, a method includes grouping the points into cells of a grid. The grid divides an observed region of a surrounding environment associated with the point cloud into the cells. The method includes computing feature vectors for the cells that use cell features to characterize the points in the cells and relationships between the cells. The method includes analyzing the feature vectors according to a clustering model to identify clusters for the cells. The clustering model evaluates the cells to identify which of the cells belong to common entities. The method includes providing the clusters as assignments of the points to the entities depicted in the point cloud.

Abstract: System, methods, and other embodiments described herein relate to improving clustering of points within a point cloud. In one embodiment, a method includes grouping the points into cells of a grid. The grid divides an observed region of a surrounding environment associated with the point cloud into the cells. The method includes computing feature vectors for the cells that use cell features to characterize the points in the cells and relationships between the cells. The method includes analyzing the feature vectors according to a clustering model to identify clusters for the cells. The clustering model evaluates the cells to identify which of the cells belong to common entities. The method includes providing the clusters as assignments of the points to the entities depicted in the point cloud.