Abstract: Engagement relevance models useful for e-commerce and computer simulation purposes are evaluated offline against ground truth to determine which is the best model at predicting various metrics, such as what products are likely to be purchased, how long a user remains on a web page, and how users will rank e-commerce items. Evaluation includes summing absolute errors between ground truth and model predictions for each of multiple users, normalizing the sum of all prediction statistics to unity. Weights may be applied to reduce skew in evaluation by accounting for biases toward disproportionately popular items.