Abstract: Methods are described for automatically identifying perception weaknesses for training data to be used in improving the performance of perception systems. Deficiencies in simulated data are also identified. The methods, which can be incorporated into a system or into instructions placed on storage media, include comparing perception system results between baseline results and results with augmented inputs and identifying perception weaknesses responsive to that comparison. The perception system is retrained using the relabeled data and is improved thereby.