Abstract: A computing device selects new test configurations for testing software. (A) First test configurations are generated using a random seed value. (B) Software under test is executed with the first test configurations to generate a test result for each. (C) Second test configurations are generated from the first test configurations and the test results generated for each. (D) The software under test is executed with the second test configurations to generate the test result for each. (E) When a restart is triggered based on a distance metric value computed between the second test configurations, a next random seed value is selected as the random seed value and (A) through (E) are repeated. (F) When the restart is not triggered, (C) through (F) are repeated until a stop criterion is satisfied. (G) When the stop criterion is satisfied, the test result is output for each test configuration.

Abstract: A computing device selects new test configurations for testing software. (A) First test configurations are generated using a random seed value. (B) Software under test is executed with the first test configurations to generate a test result for each. (C) Second test configurations are generated from the first test configurations and the test results generated for each. (D) The software under test is executed with the second test configurations to generate the test result for each. (E) When a restart is triggered based on a distance metric value computed between the second test configurations, a next random seed value is selected as the random seed value and (A) through (E) are repeated. (F) When the restart is not triggered, (C) through (F) are repeated until a stop criterion is satisfied. (G) When the stop criterion is satisfied, the test result is output for each test configuration.

Abstract: Systems and methods for evaluating the testing of a computer program wherein a test automator generates code to test the computer program with respect to predetermined testing criteria. A test results analyst reviews test results generated by applying the code to test the computer program. The environments of a test automator and a test analyst are separated, thereby relieving a test results analyst from being required to have knowledge of the code that was used to test the computer program.