Evaluating the impact on operator performance of quantification algorithms

Quantifying changes in the number and extent of lesions in MR images has been used to indicate disease activity in Multiple Sclerosis. However, quantification is often long and arduous. This has prompted research into new quantification and image processing algorithms to speed up and simplify lesion quantification. Nevertheless, many algorithms rely upon an experienced operator, and are thus subject to inter- and intra-operator variability. We present a new technique to measure operator variability which is largely independent of the lesions selected for analysis, and is expressed in the measurement units. This new measure allows researchers to predict and compare absolute uncertainty for operators using new algorithms. We used this technique to examine the impact on operator performance of computer assisted lesion quantification and anisotropic filtering of patient exams to reduce image noise. In both cases repeated measurements of lesions in MR exams were performed and analysed. Results indicate that assisted quantification reduced inter-operator variability by 1/2 (from 0.34 cm³ to 0.17 cm³) and reduced intra-operator variability by 1/3 (from 0.23 cm³ to 0.15 cm³). Anisotropic filtering reduced inter- and intra-operator variability by 1/5 (from 0.34 cm³ to 0.27 cm³, and from 0.23 cm³ to 0.19 cm³, respectively). These results may have practical implications for clinical trials which rely on quantitative measurements to assess therapeutic effect.