Abstract: Aspects of the disclosed technology provide solutions for autonomous vehicle (AV) testing and in particular, for providing a bot orchestrator to test various use cases on a fleet management system. A process of the disclosed technology can include steps for receiving, at a fleet management system, a first dispatch request, the first dispatch request indicating a first pick-up location and sending, from the fleet management system, a first dispatch command to an autonomous vehicle (AV). In some aspects, the process can further include steps for receiving, at a bot orchestrator, a first provisioning request, the first provisioning requesting specifying one or more AV parameters and instantiating, by the bot orchestrator, one or more AV bots based on the provisioning request. Systems and machine-readable media are also provided.

Abstract: A system may attribute an event, which occurred during an operation of an AV, to one or more AV behavior changes through backtesting of the AV software version. The system may identify one or more critical events from events detected by the AV based on classifications of the events. A critical event may be an event that impairs a performance of the AV. The system, after identifying a critical event, may run backtesting of the AV software version. The system may simulate AV operations, which are controlled by the AV software version and one or more historical software versions, in a virtual scene generated based on the critical event. The system may determine whether there is a regression in the AV software version based on AV performances in the simulated AV operations. The system may determine one or more AV behavior changes based on the regression.