Abstract: Methods are described for automatically identifying perception weaknesses for training data to be used in improving the performance of perception systems. Deficiencies in simulated data are also identified. The methods, which can be incorporated into a system or into instructions placed on storage media, include comparing perception system results between baseline results and results with augmented inputs and identifying perception weaknesses responsive to that comparison. The perception system is retrained using the relabeled data and is improved thereby.

Abstract: A system and method for measuring the performance of a perception system is provided that includes obtaining a pair of sensor data input streams to be evaluated by the perception system. Data augmentation is then added into at least one of the pair of sensor data input streams and the pair of sensor data input streams is then processed with the perception system to obtain a first detection list and a second detection list. The first detection list is then compared with the second detection list to identify one or more perception weaknesses.