Abstract: Systems and methods are provided for evaluating an automatic content scoring technology. A first text and a second text are received. A determination by the first automatic content scoring technology on whether at least a portion of the first text entails the second text, does not entail the second text, or refutes the second text is received. A determination by a human rater on whether at least a portion of the first text, entails the second text, does not entail the second text, or refutes the second text, and a reason for the determination by the human rater are received. The determination by the first automatic content scoring technology and the determination by the human rater are compared. A report is output indicating quality of the determination by the first automatic content scoring technology and showing disagreement between the determination by the first automatic content scoring technology and the determination by the human rater.