Abstract: An online concierge system uses a new treatment engine to score users for applying treatments of a new treatment type. The new treatment engine uses treatment models to generate treatment lift scores for the user. The new treatment engine applies an aggregation function model to the treatment lift scores to generate an aggregated lift score for the user. If the aggregated lift score exceeds a threshold, the new treatment engine applies a treatment of the new treatment type to the user. The new treatment engine trains the aggregation function model based on training examples used to train the treatment models. For a training example associated with a particular treatment type, the new treatment engine uses a target lift score generated by the treatment model for the treatment type to evaluate the performance of the aggregation function model, and to update the aggregation function model accordingly.