Abstract: An online system determines an estimated conversion rate for sponsored content items placed on content publishers and on the online system. The estimated conversion rate can be determined by a machine learning model trained using data describing content campaigns, content publishers, and online system users. This data is collected by the online system from content publishers and/or content campaigns that report conversion rates to the online system. By determining a ratio of estimated conversion rates with third party content on the content publisher against those on the online system, the online system can determine a publisher quality score for that content publisher. The online system uses the publisher quality score to normalize third party value contributions toward placing sponsored content on content publishers and the online system. Thus, disparities in the intrinsic value across publishers are diminished as third party value contributions are normalized based on the publisher conversion rates.