Abstract: The subject disclosure is directed towards a technology in which a first software version (e.g., build or check-in) that corresponds to a failure/regression is automatically identified. Software versions associated with a development order are automatically loaded and tested according to a search plan that narrows in on which version a failure condition first appears. For example, a binary search may be used that looks back to a previous version when a failure is detected on a tested version, or moves to a subsequent version when the failure is not detected. The search plan allows multiple test machines run tests in parallel on different versions, and adapts to the number of test machines available for testing.