Effects of hardware heterogeneity on the performance of SVM Alzheimer's disease classifier

Ahmed Abdulkadir, Bénédicte Mortamet, Prashanthi D Vemuri, Clifford R Jr. Jack, Gunnar Krueger, Stefan Klöppel

Research output: Contribution to journalArticle

53 Citations (Scopus)

Abstract

Fully automated machine learning methods based on structural magnetic resonance imaging (MRI) data can assist radiologists in the diagnosis of Alzheimer's disease (AD). These algorithms require large data sets to learn the separation of subjects with and without AD. Training and test data may come from heterogeneous hardware settings, which can potentially affect the performance of disease classification.A total of 518 MRI sessions from 226 healthy controls and 191 individuals with probable AD from the multicenter Alzheimer's Disease Neuroimaging Initiative (ADNI) were used to investigate whether grouping data by acquisition hardware (i.e. vendor, field strength, coil system) is beneficial for the performance of a support vector machine (SVM) classifier, compared to the case where data from different hardware is mixed. We compared the change of the SVM decision value resulting from (a) changes in hardware against the effect of disease and (b) changes resulting simply from rescanning the same subject on the same machine.Maximum accuracy of 87% was obtained with a training set of all 417 subjects. Classifiers trained with 95 subjects in each diagnostic group and acquired with heterogeneous scanner settings had an empirical detection accuracy of 84.2 ± 2.4% when tested on an independent set of the same size. These results mirror the accuracy reported in recent studies. Encouragingly, classifiers trained on images acquired with homogenous and heterogeneous hardware settings had equivalent cross-validation performances. Two scans of the same subject acquired on the same machine had very similar decision values and were generally classified into the same group. Higher variation was introduced when two acquisitions of the same subject were performed on two scanners with different field strengths. The variation was unbiased and similar for both diagnostic groups.The findings of the study encourage the pooling of data from different sites to increase the number of training samples and thereby improving performance of disease classifiers. Although small, a change in hardware could lead to a change of the decision value and thus diagnostic grouping. The findings of this study provide estimators for diagnostic accuracy of an automated disease diagnosis method involving scans acquired with different sets of hardware. Furthermore, we show that the level of confidence in the performance estimation significantly depends on the size of the training sample, and hence should be taken into account in a clinical setting.

Original languageEnglish (US)
Pages (from-to)785-792
Number of pages8
JournalNeuroImage
Volume58
Issue number3
DOIs
StatePublished - Oct 1 2011

Fingerprint

Alzheimer Disease
Magnetic Resonance Imaging
Neuroimaging
Sample Size
Meta-Analysis
Support Vector Machine

Keywords

  • Alzheimer's disease
  • Magnetic resonance imaging
  • MRI
  • Multi-site study
  • Support vector machines (SVM)

ASJC Scopus subject areas

  • Cognitive Neuroscience
  • Neurology

Cite this

Effects of hardware heterogeneity on the performance of SVM Alzheimer's disease classifier. / Abdulkadir, Ahmed; Mortamet, Bénédicte; Vemuri, Prashanthi D; Jack, Clifford R Jr.; Krueger, Gunnar; Klöppel, Stefan.

In: NeuroImage, Vol. 58, No. 3, 01.10.2011, p. 785-792.

Research output: Contribution to journalArticle

Abdulkadir, Ahmed ; Mortamet, Bénédicte ; Vemuri, Prashanthi D ; Jack, Clifford R Jr. ; Krueger, Gunnar ; Klöppel, Stefan. / Effects of hardware heterogeneity on the performance of SVM Alzheimer's disease classifier. In: NeuroImage. 2011 ; Vol. 58, No. 3. pp. 785-792.
@article{19d318f991834ecb9989bcdaa9d4dc00,
title = "Effects of hardware heterogeneity on the performance of SVM Alzheimer's disease classifier",
abstract = "Fully automated machine learning methods based on structural magnetic resonance imaging (MRI) data can assist radiologists in the diagnosis of Alzheimer's disease (AD). These algorithms require large data sets to learn the separation of subjects with and without AD. Training and test data may come from heterogeneous hardware settings, which can potentially affect the performance of disease classification.A total of 518 MRI sessions from 226 healthy controls and 191 individuals with probable AD from the multicenter Alzheimer's Disease Neuroimaging Initiative (ADNI) were used to investigate whether grouping data by acquisition hardware (i.e. vendor, field strength, coil system) is beneficial for the performance of a support vector machine (SVM) classifier, compared to the case where data from different hardware is mixed. We compared the change of the SVM decision value resulting from (a) changes in hardware against the effect of disease and (b) changes resulting simply from rescanning the same subject on the same machine.Maximum accuracy of 87{\%} was obtained with a training set of all 417 subjects. Classifiers trained with 95 subjects in each diagnostic group and acquired with heterogeneous scanner settings had an empirical detection accuracy of 84.2 ± 2.4{\%} when tested on an independent set of the same size. These results mirror the accuracy reported in recent studies. Encouragingly, classifiers trained on images acquired with homogenous and heterogeneous hardware settings had equivalent cross-validation performances. Two scans of the same subject acquired on the same machine had very similar decision values and were generally classified into the same group. Higher variation was introduced when two acquisitions of the same subject were performed on two scanners with different field strengths. The variation was unbiased and similar for both diagnostic groups.The findings of the study encourage the pooling of data from different sites to increase the number of training samples and thereby improving performance of disease classifiers. Although small, a change in hardware could lead to a change of the decision value and thus diagnostic grouping. The findings of this study provide estimators for diagnostic accuracy of an automated disease diagnosis method involving scans acquired with different sets of hardware. Furthermore, we show that the level of confidence in the performance estimation significantly depends on the size of the training sample, and hence should be taken into account in a clinical setting.",
keywords = "Alzheimer's disease, Magnetic resonance imaging, MRI, Multi-site study, Support vector machines (SVM)",
author = "Ahmed Abdulkadir and B{\'e}n{\'e}dicte Mortamet and Vemuri, {Prashanthi D} and Jack, {Clifford R Jr.} and Gunnar Krueger and Stefan Kl{\"o}ppel",
year = "2011",
month = "10",
day = "1",
doi = "10.1016/j.neuroimage.2011.06.029",
language = "English (US)",
volume = "58",
pages = "785--792",
journal = "NeuroImage",
issn = "1053-8119",
publisher = "Academic Press Inc.",
number = "3",

}

TY - JOUR

T1 - Effects of hardware heterogeneity on the performance of SVM Alzheimer's disease classifier

AU - Abdulkadir, Ahmed

AU - Mortamet, Bénédicte

AU - Vemuri, Prashanthi D

AU - Jack, Clifford R Jr.

AU - Krueger, Gunnar

AU - Klöppel, Stefan

PY - 2011/10/1

Y1 - 2011/10/1

N2 - Fully automated machine learning methods based on structural magnetic resonance imaging (MRI) data can assist radiologists in the diagnosis of Alzheimer's disease (AD). These algorithms require large data sets to learn the separation of subjects with and without AD. Training and test data may come from heterogeneous hardware settings, which can potentially affect the performance of disease classification.A total of 518 MRI sessions from 226 healthy controls and 191 individuals with probable AD from the multicenter Alzheimer's Disease Neuroimaging Initiative (ADNI) were used to investigate whether grouping data by acquisition hardware (i.e. vendor, field strength, coil system) is beneficial for the performance of a support vector machine (SVM) classifier, compared to the case where data from different hardware is mixed. We compared the change of the SVM decision value resulting from (a) changes in hardware against the effect of disease and (b) changes resulting simply from rescanning the same subject on the same machine.Maximum accuracy of 87% was obtained with a training set of all 417 subjects. Classifiers trained with 95 subjects in each diagnostic group and acquired with heterogeneous scanner settings had an empirical detection accuracy of 84.2 ± 2.4% when tested on an independent set of the same size. These results mirror the accuracy reported in recent studies. Encouragingly, classifiers trained on images acquired with homogenous and heterogeneous hardware settings had equivalent cross-validation performances. Two scans of the same subject acquired on the same machine had very similar decision values and were generally classified into the same group. Higher variation was introduced when two acquisitions of the same subject were performed on two scanners with different field strengths. The variation was unbiased and similar for both diagnostic groups.The findings of the study encourage the pooling of data from different sites to increase the number of training samples and thereby improving performance of disease classifiers. Although small, a change in hardware could lead to a change of the decision value and thus diagnostic grouping. The findings of this study provide estimators for diagnostic accuracy of an automated disease diagnosis method involving scans acquired with different sets of hardware. Furthermore, we show that the level of confidence in the performance estimation significantly depends on the size of the training sample, and hence should be taken into account in a clinical setting.

AB - Fully automated machine learning methods based on structural magnetic resonance imaging (MRI) data can assist radiologists in the diagnosis of Alzheimer's disease (AD). These algorithms require large data sets to learn the separation of subjects with and without AD. Training and test data may come from heterogeneous hardware settings, which can potentially affect the performance of disease classification.A total of 518 MRI sessions from 226 healthy controls and 191 individuals with probable AD from the multicenter Alzheimer's Disease Neuroimaging Initiative (ADNI) were used to investigate whether grouping data by acquisition hardware (i.e. vendor, field strength, coil system) is beneficial for the performance of a support vector machine (SVM) classifier, compared to the case where data from different hardware is mixed. We compared the change of the SVM decision value resulting from (a) changes in hardware against the effect of disease and (b) changes resulting simply from rescanning the same subject on the same machine.Maximum accuracy of 87% was obtained with a training set of all 417 subjects. Classifiers trained with 95 subjects in each diagnostic group and acquired with heterogeneous scanner settings had an empirical detection accuracy of 84.2 ± 2.4% when tested on an independent set of the same size. These results mirror the accuracy reported in recent studies. Encouragingly, classifiers trained on images acquired with homogenous and heterogeneous hardware settings had equivalent cross-validation performances. Two scans of the same subject acquired on the same machine had very similar decision values and were generally classified into the same group. Higher variation was introduced when two acquisitions of the same subject were performed on two scanners with different field strengths. The variation was unbiased and similar for both diagnostic groups.The findings of the study encourage the pooling of data from different sites to increase the number of training samples and thereby improving performance of disease classifiers. Although small, a change in hardware could lead to a change of the decision value and thus diagnostic grouping. The findings of this study provide estimators for diagnostic accuracy of an automated disease diagnosis method involving scans acquired with different sets of hardware. Furthermore, we show that the level of confidence in the performance estimation significantly depends on the size of the training sample, and hence should be taken into account in a clinical setting.

KW - Alzheimer's disease

KW - Magnetic resonance imaging

KW - MRI

KW - Multi-site study

KW - Support vector machines (SVM)

UR - http://www.scopus.com/inward/record.url?scp=80052169077&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=80052169077&partnerID=8YFLogxK

U2 - 10.1016/j.neuroimage.2011.06.029

DO - 10.1016/j.neuroimage.2011.06.029

M3 - Article

C2 - 21708272

AN - SCOPUS:80052169077

VL - 58

SP - 785

EP - 792

JO - NeuroImage

JF - NeuroImage

SN - 1053-8119

IS - 3

ER -