Abstract: A data processing system for A/B testing software product builds herein implements dividing a group of user devices into a first subset and a second subset of user devices to participate in a controlled build rollout of a second version of the software product, sending a first signal to the first subset of user devices to cause the first subset of computing devices to reinstall a first version of the software product which has previously been installed on the first subset of user devices, sending a second signal to the second subset of user devices to cause the second subset of computing devices to install a second version of the software product, collecting telemetry data from the user devices of the first and second subsets of user devices, and comparing the performance of the first and second versions based on the telemetry data.