Abstract: Described are techniques for determining an accurate ground-truth Bayesian Inter-Reviewer Agreement Rate (BIRAR) from unreliable sources. For instance, a process can include obtaining an initial set of diagnostic imaging exams, wherein each diagnostic imaging exam includes a severity grade associated with an initial radiologist. For each diagnostic imaging exam, two or more secondary quality assurance (QA) reviews can be obtained for each diagnostic imaging exam, wherein the secondary QA reviews are associated with QA'ing radiologists different than the initial radiologist. One or more inter-reviewer agreement rates can be determined for the QA'ing radiologists, based on the secondary QA reviews associated with the QA'ing radiologists.