Abstract: In various examples, an offline evaluation system obtains log data from a recommendation system and trains an imitation ranker using the log data. The imitation ranker generates a first result including a set of scores associated with document and rank pairs based on a query. The offline evaluation system may then determine a rank distribution indicating propensities associated with the document and rank pairs for a set of impressions which can be used to determine a value associated with the performance of the new recommendation system.

Abstract: Systems and methods for diversity auditing are described. The systems and methods include identifying a plurality of images; detecting a face in each of the plurality of images using a face detection network; classifying the face in each of the plurality of images based on a sensitive attribute using an image classification network; generating a distribution of the sensitive attribute in the plurality of images based on the classification; and computing a diversity score for the plurality of images based on the distribution.