Abstract: Systems and techniques are provided for validating a simulation framework.

Abstract: Systems and techniques are provided for testing software changes and determining a repeatability of software tests. An example method can perform software tests at different timepoints, each software test being based on a test scenario comprising a test simulation environment configured to test a software; determine one or more software tests from the software tests having a variation in test scores that exceeds a divergence threshold, the one or more software tests comprising at least one test scenario; rerun a software test a number of times, the software test configured to test changes to a code of the software, the changes to the code being associated with the software test and/or the at least one test scenario; and determine, based on test scores generated by the software test performed the number of times, a repeatability score for the software test on the changes to the code of the software.

Abstract: The subject technology provides solutions for evaluating AV performance in simulated test environments. In some aspects, the disclosed technology includes a process for receiving road data comprising autonomous vehicle (AV) sensor data, generating simulation (SIM) data based on the road data, and measuring one or more first performance metrics, wherein the one or more first performance metrics correspond with a performance of an AV in the real-world environment. The process can further include steps for measuring one or more second performance metrics, wherein the one or more second performance metrics correspond with a performance of the AV in the simulated environment, training a machine-learning (ML) model to predict a correspondence between the performance of the AV in the real-world environment and the performance of the AV in the simulated environment. Systems and machine-readable media are also provided.

Abstract: The disclosed technology provides solutions for improving simulation generation, and in particular for diagnosing problems with a simulation renderer configured to generate simulated (or synthetic) environments for use in autonomous vehicle (AV) testing and training. In some aspects, a process of the disclosed technology includes steps for receiving a set of divergence metrics, wherein the divergence metrics comprise performance statistics for one or more components of a simulation renderer, providing the divergence metrics to a machine-learning model, and identifying, using an attribution tool, one or more components of the simulation renderer that contributed to the AV pose divergence based on one or more weights of the machine-learning model. Systems and machine-readable media are also provided.

Abstract: Aspects of the subject technology relate to systems, methods, and computer-readable media for troubleshooting a baseline simulation. A method can include accessing captured real-world data of performed behaviors of an autonomous vehicle in operation and running input data through a simulation template as part of a baseline simulation. The method can also include identifying a divergence of the baseline simulation between the performed behaviors of the AV and simulated behaviors in the baseline simulation of the AV. The method can include running the input data through a modified simulation template and identifying a divergence of the modified simulation between the performed behaviors of the AV and simulated behaviors in the modified simulation of the AV. The method can include troubleshooting the baseline simulation based on a comparison between the divergence of the modified simulation and the divergence of the baseline simulation.

Abstract: Systems and techniques are provided for validating a simulation framework.

Abstract: Systems and techniques are provided for testing software changes and determining a repeatability of software tests. An example method can perform software tests at different timepoints, each software test being based on a test scenario comprising a test simulation environment configured to test a software; determine one or more software tests from the software tests having a variation in test scores that exceeds a divergence threshold, the one or more software tests comprising at least one test scenario; rerun a software test a number of times, the software test configured to test changes to a code of the software, the changes to the code being associated with the software test and/or the at least one test scenario; and determine, based on test scores generated by the software test performed the number of times, a repeatability score for the software test on the changes to the code of the software.

Abstract: Systems and methods for evaluating a fidelity of a simulation in replicating a real-world driving scene are provided. A computer-implemented system including one or more non-transitory computer-readable media storing instructions, when executed by one or more processing units, cause the one or more processing units to perform operations including receiving first data collected from a reference driving scene, the first data associated with the reference driving scene and a driving behavior of a first vehicle in the reference driving scene; receiving second data collected from a driving simulation that generates a simulated driving scene and operations of a second vehicle in the simulated driving scene, wherein the simulated driving scene is a simulation of the reference driving scene; and determining a fidelity of the driving simulation based on a comparison between the first data and the second data.

Abstract: Systems and methods for improving an end-to-end (E2E) vehicle simulation in replicating a real-world driving scene based on a continuous feedback loop are provided. For example, a method includes receiving first data collected from a real-world driving scene, the first data associated with the real-world driving scene and a performance of a real-world vehicle in the real-world driving scene; and adjusting, for a current iteration, a parameter associated with an E2E vehicle simulation that simulates operations of a simulated vehicle driving in a simulated driving scene, wherein the adjustment is based on a fidelity value of the E2E vehicle simulation in a previous iteration, and the fidelity value is based on a comparison between second data collected from the E2E simulation in the previous iteration and the first data.

Abstract: Systems and methods for generation and application of a road data response prediction model for evaluation of vehicle simulation are provided. A computer-implemented system, including one or more non-transitory computer-readable media storing instructions, when executed by one or more processing units, cause the one or more processing units to perform operations including receiving road data collected from a reference driving scene; extracting, from the road data, features of the reference driving scene; obtaining, from the road data, response data associated with a vehicle in the reference driving scene and responsive to the extracted features; calculating a statistical measure of the response data; and generating a road data response prediction model to map the extracted features of the road data to the statistical measure of the response data.