Evaluation of Pseudoreader Study Designs to Estimate Observer Performance Results as an Alternative to Fully Crossed, Multireader, Multicase Studies

Research output: Contribution to journalArticle

Abstract

Rationale and Objectives: To examine the ability of a pseudoreader study design to estimate the observer performance obtained using a traditional fully crossed, multireader, multicase (MRMC) study. Materials and Methods: A 10-reader MRMC study with 20 computed tomography datasets was designed to measure observer performance on four novel noise reduction methods. This study served as the foundation for the empirical evaluation of three different pseudoreader designs, each of which used a similar bootstrap approach for generating 2000 realizations from the fully crossed study. Our three approaches to generating a pseudoreader varied in the degree to which reader performance was matched and integrated into the pseudoreader design. One randomly selected simulation was selected as a “mock study” to represent a hypothetical, prospective implementation of the design. Results: Using the traditional fully crossed design, figures of merit) (95% CIs) for the four noise reductions methods were 68.2 (55.5–81.0), 69.6 (58.4–80.8), 70.8 (60.2–81.4), and 70.9 (60.4–81.3), respectively. When radiologists’ performances on the fourth noise reduction method were used to pair readers during the mock study, there was strong agreement in the estimated figures of merits with estimates using the pseudoreader design being within ±3% of the fully crossed design. Conclusion: Fully crossed MRMC studies require significant investment in resources and time, often resulting in delayed implementation or minimal human testing before dissemination. The pseudoreader approach accelerates study conduct by combining readers judiciously and was found to provide comparable results to the traditional fully crossed design by making strong assumptions about exchangeability of the readers.

Original languageEnglish (US)
JournalAcademic radiology
DOIs
StatePublished - Jan 1 2019

Fingerprint

Noise
antineoplaston A10
Tomography
Radiologists
Datasets

Keywords

  • Multi-case study
  • Multi-reader
  • Observer performance
  • Study design

ASJC Scopus subject areas

  • Radiology Nuclear Medicine and imaging

Cite this

@article{9304016bb77b4ba19f95c2932f82b7fa,
title = "Evaluation of Pseudoreader Study Designs to Estimate Observer Performance Results as an Alternative to Fully Crossed, Multireader, Multicase Studies",
abstract = "Rationale and Objectives: To examine the ability of a pseudoreader study design to estimate the observer performance obtained using a traditional fully crossed, multireader, multicase (MRMC) study. Materials and Methods: A 10-reader MRMC study with 20 computed tomography datasets was designed to measure observer performance on four novel noise reduction methods. This study served as the foundation for the empirical evaluation of three different pseudoreader designs, each of which used a similar bootstrap approach for generating 2000 realizations from the fully crossed study. Our three approaches to generating a pseudoreader varied in the degree to which reader performance was matched and integrated into the pseudoreader design. One randomly selected simulation was selected as a “mock study” to represent a hypothetical, prospective implementation of the design. Results: Using the traditional fully crossed design, figures of merit) (95{\%} CIs) for the four noise reductions methods were 68.2 (55.5–81.0), 69.6 (58.4–80.8), 70.8 (60.2–81.4), and 70.9 (60.4–81.3), respectively. When radiologists’ performances on the fourth noise reduction method were used to pair readers during the mock study, there was strong agreement in the estimated figures of merits with estimates using the pseudoreader design being within ±3{\%} of the fully crossed design. Conclusion: Fully crossed MRMC studies require significant investment in resources and time, often resulting in delayed implementation or minimal human testing before dissemination. The pseudoreader approach accelerates study conduct by combining readers judiciously and was found to provide comparable results to the traditional fully crossed design by making strong assumptions about exchangeability of the readers.",
keywords = "Multi-case study, Multi-reader, Observer performance, Study design",
author = "Carter, {Rickey E.} and {Holmes III}, {David R.} and Fletcher, {Joel Garland} and McCollough, {Cynthia H}",
year = "2019",
month = "1",
day = "1",
doi = "10.1016/j.acra.2019.03.014",
language = "English (US)",
journal = "Academic Radiology",
issn = "1076-6332",
publisher = "Elsevier USA",

}

TY - JOUR

T1 - Evaluation of Pseudoreader Study Designs to Estimate Observer Performance Results as an Alternative to Fully Crossed, Multireader, Multicase Studies

AU - Carter, Rickey E.

AU - Holmes III, David R.

AU - Fletcher, Joel Garland

AU - McCollough, Cynthia H

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Rationale and Objectives: To examine the ability of a pseudoreader study design to estimate the observer performance obtained using a traditional fully crossed, multireader, multicase (MRMC) study. Materials and Methods: A 10-reader MRMC study with 20 computed tomography datasets was designed to measure observer performance on four novel noise reduction methods. This study served as the foundation for the empirical evaluation of three different pseudoreader designs, each of which used a similar bootstrap approach for generating 2000 realizations from the fully crossed study. Our three approaches to generating a pseudoreader varied in the degree to which reader performance was matched and integrated into the pseudoreader design. One randomly selected simulation was selected as a “mock study” to represent a hypothetical, prospective implementation of the design. Results: Using the traditional fully crossed design, figures of merit) (95% CIs) for the four noise reductions methods were 68.2 (55.5–81.0), 69.6 (58.4–80.8), 70.8 (60.2–81.4), and 70.9 (60.4–81.3), respectively. When radiologists’ performances on the fourth noise reduction method were used to pair readers during the mock study, there was strong agreement in the estimated figures of merits with estimates using the pseudoreader design being within ±3% of the fully crossed design. Conclusion: Fully crossed MRMC studies require significant investment in resources and time, often resulting in delayed implementation or minimal human testing before dissemination. The pseudoreader approach accelerates study conduct by combining readers judiciously and was found to provide comparable results to the traditional fully crossed design by making strong assumptions about exchangeability of the readers.

AB - Rationale and Objectives: To examine the ability of a pseudoreader study design to estimate the observer performance obtained using a traditional fully crossed, multireader, multicase (MRMC) study. Materials and Methods: A 10-reader MRMC study with 20 computed tomography datasets was designed to measure observer performance on four novel noise reduction methods. This study served as the foundation for the empirical evaluation of three different pseudoreader designs, each of which used a similar bootstrap approach for generating 2000 realizations from the fully crossed study. Our three approaches to generating a pseudoreader varied in the degree to which reader performance was matched and integrated into the pseudoreader design. One randomly selected simulation was selected as a “mock study” to represent a hypothetical, prospective implementation of the design. Results: Using the traditional fully crossed design, figures of merit) (95% CIs) for the four noise reductions methods were 68.2 (55.5–81.0), 69.6 (58.4–80.8), 70.8 (60.2–81.4), and 70.9 (60.4–81.3), respectively. When radiologists’ performances on the fourth noise reduction method were used to pair readers during the mock study, there was strong agreement in the estimated figures of merits with estimates using the pseudoreader design being within ±3% of the fully crossed design. Conclusion: Fully crossed MRMC studies require significant investment in resources and time, often resulting in delayed implementation or minimal human testing before dissemination. The pseudoreader approach accelerates study conduct by combining readers judiciously and was found to provide comparable results to the traditional fully crossed design by making strong assumptions about exchangeability of the readers.

KW - Multi-case study

KW - Multi-reader

KW - Observer performance

KW - Study design

UR - http://www.scopus.com/inward/record.url?scp=85065146239&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85065146239&partnerID=8YFLogxK

U2 - 10.1016/j.acra.2019.03.014

DO - 10.1016/j.acra.2019.03.014

M3 - Article

AN - SCOPUS:85065146239

JO - Academic Radiology

JF - Academic Radiology

SN - 1076-6332

ER -