Abstract: Disclosed examples include an automated online experimentation mechanism that can perform model selection from a large pool of models with a relatively small number of online experiments. The probability distribution of the metric of interest that contains the model uncertainty is derived from a Bayesian surrogate model trained using historical logs. Disclosed techniques can be applied to identify a superior model by sequentially selecting and deploying a list of models from the candidate set that balance exploration-exploitation.