Inter-laboratory comparison of channelized hotelling observer computation

Alexandre Ba, Craig K. Abbey, Jongduk Baek, Minah Han, Ramona W. Bouwman, Christiana Balta, Jovan Brankov, Francesc Massanes, Howard C. Gifford, Irene Hernandez-Giron, Wouter J.H. Veldkamp, Dimitar Petrov, Nicholas Marshall, Frank W. Samuelson, Rongping Zeng, Justin B. Solomon, Ehsan Samei, Pontus Timberg, Hannie Förnvik, Ingrid ReiserLifeng Yu, Hao Gong, François O. Bochud

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

Purpose: The task-based assessment of image quality using model observers is increasingly used for the assessment of different imaging modalities. However, the performance computation of model observers needs standardization as well as a well-established trust in its implementation methodology and uncertainty estimation. The purpose of this work was to determine the degree of equivalence of the channelized Hotelling observer performance and uncertainty estimation using an intercomparison exercise. Materials and Methods: Image samples to estimate model observer performance for detection tasks were generated from two-dimensional CT image slices of a uniform water phantom. A common set of images was sent to participating laboratories to perform and document the following tasks: (a) estimate the detectability index of a well-defined CHO and its uncertainty in three conditions involving different sized targets all at the same dose, and (b) apply this CHO to an image set where ground truth was unknown to participants (lower image dose). In addition, and on an optional basis, we asked the participating laboratories to (c) estimate the performance of real human observers from a psychophysical experiment of their choice. Each of the 13 participating laboratories was confidentially assigned a participant number and image sets could be downloaded through a secure server. Results were distributed with each participant recognizable by its number and then each laboratory was able to modify their results with justification as model observer calculation are not yet a routine and potentially error prone. Results: Detectability index increased with signal size for all participants and was very consistent for 6 mm sized target while showing higher variability for 8 and 10 mm sized target. There was one order of magnitude between the lowest and the largest uncertainty estimation. Conclusions: This intercomparison helped define the state of the art of model observer performance computation and with thirteen participants, reflects openness and trust within the medical imaging community. The performance of a CHO with explicitly defined channels and a relatively large number of test images was consistently estimated by all participants. In contrast, the paper demonstrates that there is no agreement on estimating the variance of detectability in the training and testing setting.

Original languageEnglish (US)
Pages (from-to)3019-3030
Number of pages12
JournalMedical Physics
Volume45
Issue number7
DOIs
StatePublished - Jul 1 2018

Fingerprint

Uncertainty
Task Performance and Analysis
Diagnostic Imaging
Water

Keywords

  • channelized hotelling observer
  • computed tomography
  • image quality
  • intercomparison
  • model observers

ASJC Scopus subject areas

  • Biophysics
  • Radiology Nuclear Medicine and imaging

Cite this

Ba, A., Abbey, C. K., Baek, J., Han, M., Bouwman, R. W., Balta, C., ... Bochud, F. O. (2018). Inter-laboratory comparison of channelized hotelling observer computation. Medical Physics, 45(7), 3019-3030. https://doi.org/10.1002/mp.12940

Inter-laboratory comparison of channelized hotelling observer computation. / Ba, Alexandre; Abbey, Craig K.; Baek, Jongduk; Han, Minah; Bouwman, Ramona W.; Balta, Christiana; Brankov, Jovan; Massanes, Francesc; Gifford, Howard C.; Hernandez-Giron, Irene; Veldkamp, Wouter J.H.; Petrov, Dimitar; Marshall, Nicholas; Samuelson, Frank W.; Zeng, Rongping; Solomon, Justin B.; Samei, Ehsan; Timberg, Pontus; Förnvik, Hannie; Reiser, Ingrid; Yu, Lifeng; Gong, Hao; Bochud, François O.

In: Medical Physics, Vol. 45, No. 7, 01.07.2018, p. 3019-3030.

Research output: Contribution to journalArticle

Ba, A, Abbey, CK, Baek, J, Han, M, Bouwman, RW, Balta, C, Brankov, J, Massanes, F, Gifford, HC, Hernandez-Giron, I, Veldkamp, WJH, Petrov, D, Marshall, N, Samuelson, FW, Zeng, R, Solomon, JB, Samei, E, Timberg, P, Förnvik, H, Reiser, I, Yu, L, Gong, H & Bochud, FO 2018, 'Inter-laboratory comparison of channelized hotelling observer computation', Medical Physics, vol. 45, no. 7, pp. 3019-3030. https://doi.org/10.1002/mp.12940
Ba A, Abbey CK, Baek J, Han M, Bouwman RW, Balta C et al. Inter-laboratory comparison of channelized hotelling observer computation. Medical Physics. 2018 Jul 1;45(7):3019-3030. https://doi.org/10.1002/mp.12940
Ba, Alexandre ; Abbey, Craig K. ; Baek, Jongduk ; Han, Minah ; Bouwman, Ramona W. ; Balta, Christiana ; Brankov, Jovan ; Massanes, Francesc ; Gifford, Howard C. ; Hernandez-Giron, Irene ; Veldkamp, Wouter J.H. ; Petrov, Dimitar ; Marshall, Nicholas ; Samuelson, Frank W. ; Zeng, Rongping ; Solomon, Justin B. ; Samei, Ehsan ; Timberg, Pontus ; Förnvik, Hannie ; Reiser, Ingrid ; Yu, Lifeng ; Gong, Hao ; Bochud, François O. / Inter-laboratory comparison of channelized hotelling observer computation. In: Medical Physics. 2018 ; Vol. 45, No. 7. pp. 3019-3030.
@article{5d0645555f7241e08b227d8d48d2fd1e,
title = "Inter-laboratory comparison of channelized hotelling observer computation",
abstract = "Purpose: The task-based assessment of image quality using model observers is increasingly used for the assessment of different imaging modalities. However, the performance computation of model observers needs standardization as well as a well-established trust in its implementation methodology and uncertainty estimation. The purpose of this work was to determine the degree of equivalence of the channelized Hotelling observer performance and uncertainty estimation using an intercomparison exercise. Materials and Methods: Image samples to estimate model observer performance for detection tasks were generated from two-dimensional CT image slices of a uniform water phantom. A common set of images was sent to participating laboratories to perform and document the following tasks: (a) estimate the detectability index of a well-defined CHO and its uncertainty in three conditions involving different sized targets all at the same dose, and (b) apply this CHO to an image set where ground truth was unknown to participants (lower image dose). In addition, and on an optional basis, we asked the participating laboratories to (c) estimate the performance of real human observers from a psychophysical experiment of their choice. Each of the 13 participating laboratories was confidentially assigned a participant number and image sets could be downloaded through a secure server. Results were distributed with each participant recognizable by its number and then each laboratory was able to modify their results with justification as model observer calculation are not yet a routine and potentially error prone. Results: Detectability index increased with signal size for all participants and was very consistent for 6 mm sized target while showing higher variability for 8 and 10 mm sized target. There was one order of magnitude between the lowest and the largest uncertainty estimation. Conclusions: This intercomparison helped define the state of the art of model observer performance computation and with thirteen participants, reflects openness and trust within the medical imaging community. The performance of a CHO with explicitly defined channels and a relatively large number of test images was consistently estimated by all participants. In contrast, the paper demonstrates that there is no agreement on estimating the variance of detectability in the training and testing setting.",
keywords = "channelized hotelling observer, computed tomography, image quality, intercomparison, model observers",
author = "Alexandre Ba and Abbey, {Craig K.} and Jongduk Baek and Minah Han and Bouwman, {Ramona W.} and Christiana Balta and Jovan Brankov and Francesc Massanes and Gifford, {Howard C.} and Irene Hernandez-Giron and Veldkamp, {Wouter J.H.} and Dimitar Petrov and Nicholas Marshall and Samuelson, {Frank W.} and Rongping Zeng and Solomon, {Justin B.} and Ehsan Samei and Pontus Timberg and Hannie F{\"o}rnvik and Ingrid Reiser and Lifeng Yu and Hao Gong and Bochud, {Fran{\cc}ois O.}",
year = "2018",
month = "7",
day = "1",
doi = "10.1002/mp.12940",
language = "English (US)",
volume = "45",
pages = "3019--3030",
journal = "Medical Physics",
issn = "0094-2405",
publisher = "AAPM - American Association of Physicists in Medicine",
number = "7",

}

TY - JOUR

T1 - Inter-laboratory comparison of channelized hotelling observer computation

AU - Ba, Alexandre

AU - Abbey, Craig K.

AU - Baek, Jongduk

AU - Han, Minah

AU - Bouwman, Ramona W.

AU - Balta, Christiana

AU - Brankov, Jovan

AU - Massanes, Francesc

AU - Gifford, Howard C.

AU - Hernandez-Giron, Irene

AU - Veldkamp, Wouter J.H.

AU - Petrov, Dimitar

AU - Marshall, Nicholas

AU - Samuelson, Frank W.

AU - Zeng, Rongping

AU - Solomon, Justin B.

AU - Samei, Ehsan

AU - Timberg, Pontus

AU - Förnvik, Hannie

AU - Reiser, Ingrid

AU - Yu, Lifeng

AU - Gong, Hao

AU - Bochud, François O.

PY - 2018/7/1

Y1 - 2018/7/1

N2 - Purpose: The task-based assessment of image quality using model observers is increasingly used for the assessment of different imaging modalities. However, the performance computation of model observers needs standardization as well as a well-established trust in its implementation methodology and uncertainty estimation. The purpose of this work was to determine the degree of equivalence of the channelized Hotelling observer performance and uncertainty estimation using an intercomparison exercise. Materials and Methods: Image samples to estimate model observer performance for detection tasks were generated from two-dimensional CT image slices of a uniform water phantom. A common set of images was sent to participating laboratories to perform and document the following tasks: (a) estimate the detectability index of a well-defined CHO and its uncertainty in three conditions involving different sized targets all at the same dose, and (b) apply this CHO to an image set where ground truth was unknown to participants (lower image dose). In addition, and on an optional basis, we asked the participating laboratories to (c) estimate the performance of real human observers from a psychophysical experiment of their choice. Each of the 13 participating laboratories was confidentially assigned a participant number and image sets could be downloaded through a secure server. Results were distributed with each participant recognizable by its number and then each laboratory was able to modify their results with justification as model observer calculation are not yet a routine and potentially error prone. Results: Detectability index increased with signal size for all participants and was very consistent for 6 mm sized target while showing higher variability for 8 and 10 mm sized target. There was one order of magnitude between the lowest and the largest uncertainty estimation. Conclusions: This intercomparison helped define the state of the art of model observer performance computation and with thirteen participants, reflects openness and trust within the medical imaging community. The performance of a CHO with explicitly defined channels and a relatively large number of test images was consistently estimated by all participants. In contrast, the paper demonstrates that there is no agreement on estimating the variance of detectability in the training and testing setting.

AB - Purpose: The task-based assessment of image quality using model observers is increasingly used for the assessment of different imaging modalities. However, the performance computation of model observers needs standardization as well as a well-established trust in its implementation methodology and uncertainty estimation. The purpose of this work was to determine the degree of equivalence of the channelized Hotelling observer performance and uncertainty estimation using an intercomparison exercise. Materials and Methods: Image samples to estimate model observer performance for detection tasks were generated from two-dimensional CT image slices of a uniform water phantom. A common set of images was sent to participating laboratories to perform and document the following tasks: (a) estimate the detectability index of a well-defined CHO and its uncertainty in three conditions involving different sized targets all at the same dose, and (b) apply this CHO to an image set where ground truth was unknown to participants (lower image dose). In addition, and on an optional basis, we asked the participating laboratories to (c) estimate the performance of real human observers from a psychophysical experiment of their choice. Each of the 13 participating laboratories was confidentially assigned a participant number and image sets could be downloaded through a secure server. Results were distributed with each participant recognizable by its number and then each laboratory was able to modify their results with justification as model observer calculation are not yet a routine and potentially error prone. Results: Detectability index increased with signal size for all participants and was very consistent for 6 mm sized target while showing higher variability for 8 and 10 mm sized target. There was one order of magnitude between the lowest and the largest uncertainty estimation. Conclusions: This intercomparison helped define the state of the art of model observer performance computation and with thirteen participants, reflects openness and trust within the medical imaging community. The performance of a CHO with explicitly defined channels and a relatively large number of test images was consistently estimated by all participants. In contrast, the paper demonstrates that there is no agreement on estimating the variance of detectability in the training and testing setting.

KW - channelized hotelling observer

KW - computed tomography

KW - image quality

KW - intercomparison

KW - model observers

UR - http://www.scopus.com/inward/record.url?scp=85049789575&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85049789575&partnerID=8YFLogxK

U2 - 10.1002/mp.12940

DO - 10.1002/mp.12940

M3 - Article

C2 - 29704868

AN - SCOPUS:85049789575

VL - 45

SP - 3019

EP - 3030

JO - Medical Physics

JF - Medical Physics

SN - 0094-2405

IS - 7

ER -