Abstract: A machine learning system is trained to determine scores indicative of a quality of video data based on the characteristics of the video data, without requiring information regarding the correct appearance or other aspects of the video content. To train the machine learning system, users input scores for videos having predetermined quality scores, videos that have been previously seen by the users, and videos that have not been previously seen by the users. The differences between a user's score and a predetermined score or a score previously input by the user are used to determine a consistency metric. The scores and consistency metrics determined for a group of users, and the video characteristics of the videos presented to the users, are used to train the machine learning system to determine scores indicative of the quality of a video based on the characteristics of the video.