Abstract: Disclosed examples include an automated online experimentation mechanism that can perform model selection from a large pool of models with a relatively small number of online experiments. The probability distribution of the metric of interest that contains the model uncertainty is derived from a Bayesian surrogate model trained using historical logs. Disclosed techniques can be applied to identify a superior model by sequentially selecting and deploying a list of models from the candidate set that balance exploration-exploitation.

Abstract: An electronic device, for a search session, receives one or more user-input search queries and determines, based on interactions with each user-input search query, whether the search session satisfies success criteria. The electronic device generates a graph that includes, for a plurality of search sessions that satisfy the success criteria: a first set of nodes, each node in the first set of nodes corresponding to a respective search query of the plurality of search queries in a respective search session, and a second set of nodes, each node in the second set of nodes corresponding to a respective content item selected from a respective search query of the plurality of search queries in a respective search session. The electronic device converts the first and the second set of nodes of the graph to a vector space and provides a recommendation based on the nodes in the vector space.

Abstract: Disclosed examples include an automated online experimentation mechanism that can perform model selection from a large pool of models with a relatively small number of online experiments. The probability distribution of the metric of interest that contains the model uncertainty is derived from a Bayesian surrogate model trained using historical logs. Disclosed techniques can be applied to identify a superior model by sequentially selecting and deploying a list of models from the candidate set that balance exploration-exploitation.