Abstract: A method for determining hard example sensor data inputs for training a task neural network is described. The task neural network is configured to receive a sensor data input and to generate a respective output for the sensor data input to perform a machine learning task. The method includes: receiving one or more sensor data inputs depicting a same scene of an environment, wherein the one or more sensor data inputs are taken during a predetermined time period; generating a plurality of predictions about a characteristic of an object of the scene; determining a level of inconsistency between the plurality of predictions; determining that the level of inconsistency exceeds a threshold level; and in response to the determining that the level of inconsistency exceeds a threshold level, determining that the one or more sensor data inputs comprise a hard example sensor data input.

Abstract: Aspects of the disclosure provide methods for controlling a first vehicle having an autonomous driving mode. In one instance, sensor data generated by one or more sensors of the first vehicle may be received. A splash and characteristics of the splash may be detected from the sensor data using a classifier. A severity of a puddle may be determined based on the characteristics of the splash and a speed of a second vehicle that caused the splash. The first vehicle may be controlled based on the severity. In another instance, a location of a puddle relative to a tire of a second vehicle is estimated using sensor data generated by one or more sensors of the first vehicle. A severity of the puddle may be determined based on the estimated location. The first vehicle may be controlled based on the severity.