Abstract: An online concierge system generates an aggregated lift score for a test feature for the online concierge system. The online concierge presents prioritized items from a set of item groups to two sets of users: a test set and a control set. The online concierge system uses the test feature to present prioritized items to users in the test set, and the online concierge system uses existing functionality to present prioritized items to users in the control set. For each test group, the online concierge system creates holdout subsets out of the test set and the control set. The online concierge system tracks user interactions with items in an item group and computes a group lift score for the item group. The online concierge system generates an aggregated lift score for the test feature based on the group lift scores and presents items to users based on the aggregated lift score.