A 25-reader performance study for hepatic metastasis detection: lessons from unsupervised learning

There is substantial variability in the performance of radiologist readers. We hypothesized that certain readers may have idiosyncratic weaknesses towards certain types of lesions, and unsupervised learning techniques might identify these patterns. After IRB approval, 25 radiologist readers (9 abdominal subspecialists and 16 non-specialists or trainees) read 40 portal phase liver CT exams, marking all metastases and providing a confidence rating on a scale of 1 to 100. We formed a matrix of reader confidence ratings, with rows corresponding to readers, and columns corresponding to metastases, and each matrix entry providing the confidence rating that a reader gave to the metastasis, with zero confidence used for lesions that were not marked. A clustergram was used to permute the rows and columns of this matrix to group similar readers and metastases together. This clustergram was manually interpreted. We found a cluster of lesions with atypical presentation that were missed by several readers, including subspecialists, and a separate cluster of small, subtle lesions where subspecialists were more confident of their diagnosis than trainees. These and other observations from unsupervised learning could inform targeted training and education of future radiologists.