e-ASPECTS software improves interobserver agreement and accuracy of interpretation of aspects score

Waleed Brinjikji; Mehdi Abbasi; Catherine Arnold; John C. Benson; Sherry A. Braksick; Norbert Campeau; Carrie M. Carr; Petrice M. Cogswell; James P. Klaas; Greta B. Liebo; Jason T. Little; Patrick H. Luetmer; Steven A. Messina; Alex A. Nagelschneider; Kara M. Schwartz; Christopher P. Wood; Deena M. Nasr; David F. Kallmes

doi:10.1177/15910199211011861

e-ASPECTS software improves interobserver agreement and accuracy of interpretation of aspects score

Waleed Brinjikji, Mehdi Abbasi, Catherine Arnold, John C. Benson, Sherry A. Braksick, Norbert Campeau, Carrie M. Carr, Petrice M. Cogswell, James P. Klaas, Greta B. Liebo, Jason T. Little, Patrick H. Luetmer, Steven A. Messina, Alex A. Nagelschneider, Kara M. Schwartz, Christopher P. Wood, Deena M. Nasr, David F. Kallmes

Research output: Contribution to journal › Article › peer-review

Abstract

Introduction: There is increased interest in the use of artificial intelligence-based (AI) software packages in the evaluation of neuroimaging studies for acute ischemic stroke. We studied whether, compared to standard image interpretation without AI, Brainomix e-ASPECTS software improved interobserver agreement and accuracy in detecting ASPECTS regions affected in anterior circulation LVO. Methods: We included 60 consecutive patients with anterior circulation LVO who had TICI 3 revascularization within 60 minutes of their baseline CT. A total of 16 readers, including senior neuroradiologists, junior neuroradiologists and vascular neurologists participated. Readers interpreted CT scans on independent workstations and assessed final ASPECTS and evaluated whether each individual ASPECTS region was affected. Two months later, readers again evaluated the CT scans, but with assistance of e-ASPECTS software. We assessed interclass correlation coefficient for total ASPECTS and interobserver agreement with Fleiss’ Kappa for each ASPECTS region with and without assistance of the e-ASPECTS. We also assessed accuracy for the readers with and without e-ASPECTS assistance. In our assessment of accuracy, ground truth was the 24 hour CT in this cohort of patients who had prompt and complete revascularization. Results: Interclass correlation coefficient for total ASPECTS without e-ASPECTS assistance was 0.395, indicating fair agreement compared, to 0.574 with e-ASPECTS assistance, indicating good agreement (P < 0.01). There was significant improvement in inter-rater agreement with e-ASPECTS assistance for each individual region with the exception of M6 and caudate. The e-ASPECTS software had higher accuracy than the overall cohort of readers (with and without e-ASPECTS assistance) for every region except the caudate. Conclusions: Use of Brainomix e-ASPECTS software resulted in significant improvements in inter-rater agreement and accuracy of ASPECTS score evaluation in a large group of neuroradiologists and neurologists. e-ASPECTS software was more predictive of final infarct/ASPECTS than the overall group interpreting the CT scans with and without e-ASPECTS assistance.

Original language	English (US)
Pages (from-to)	781-787
Number of pages	7
Journal	Interventional Neuroradiology
Volume	27
Issue number	6
DOIs	https://doi.org/10.1177/15910199211011861
State	Published - Dec 2021

Keywords

ASPECTS
CT
Stroke
artificial intelligence
large vessel occlusion

ASJC Scopus subject areas

Radiology Nuclear Medicine and imaging
Clinical Neurology
Cardiology and Cardiovascular Medicine

Access to Document

10.1177/15910199211011861

Cite this

Brinjikji, W., Abbasi, M., Arnold, C., Benson, J. C., Braksick, S. A., Campeau, N., Carr, C. M., Cogswell, P. M., Klaas, J. P., Liebo, G. B., Little, J. T., Luetmer, P. H., Messina, S. A., Nagelschneider, A. A., Schwartz, K. M., Wood, C. P., Nasr, D. M., & Kallmes, D. F. (2021). e-ASPECTS software improves interobserver agreement and accuracy of interpretation of aspects score. Interventional Neuroradiology, 27(6), 781-787. https://doi.org/10.1177/15910199211011861

Brinjikji, W, Abbasi, M, Arnold, C, Benson, JC, Braksick, SA, Campeau, N , Carr, CM, Cogswell, PM, Klaas, JP, Liebo, GB, Little, JT, Luetmer, PH, Messina, SA, Nagelschneider, AA, Schwartz, KM, Wood, CP, Nasr, DM & Kallmes, DF 2021, 'e-ASPECTS software improves interobserver agreement and accuracy of interpretation of aspects score', Interventional Neuroradiology, vol. 27, no. 6, pp. 781-787. https://doi.org/10.1177/15910199211011861

@article{ff90612b8b564c8380d77e1dde6e3846,

title = "e-ASPECTS software improves interobserver agreement and accuracy of interpretation of aspects score",

abstract = "Introduction: There is increased interest in the use of artificial intelligence-based (AI) software packages in the evaluation of neuroimaging studies for acute ischemic stroke. We studied whether, compared to standard image interpretation without AI, Brainomix e-ASPECTS software improved interobserver agreement and accuracy in detecting ASPECTS regions affected in anterior circulation LVO. Methods: We included 60 consecutive patients with anterior circulation LVO who had TICI 3 revascularization within 60 minutes of their baseline CT. A total of 16 readers, including senior neuroradiologists, junior neuroradiologists and vascular neurologists participated. Readers interpreted CT scans on independent workstations and assessed final ASPECTS and evaluated whether each individual ASPECTS region was affected. Two months later, readers again evaluated the CT scans, but with assistance of e-ASPECTS software. We assessed interclass correlation coefficient for total ASPECTS and interobserver agreement with Fleiss{\textquoteright} Kappa for each ASPECTS region with and without assistance of the e-ASPECTS. We also assessed accuracy for the readers with and without e-ASPECTS assistance. In our assessment of accuracy, ground truth was the 24 hour CT in this cohort of patients who had prompt and complete revascularization. Results: Interclass correlation coefficient for total ASPECTS without e-ASPECTS assistance was 0.395, indicating fair agreement compared, to 0.574 with e-ASPECTS assistance, indicating good agreement (P < 0.01). There was significant improvement in inter-rater agreement with e-ASPECTS assistance for each individual region with the exception of M6 and caudate. The e-ASPECTS software had higher accuracy than the overall cohort of readers (with and without e-ASPECTS assistance) for every region except the caudate. Conclusions: Use of Brainomix e-ASPECTS software resulted in significant improvements in inter-rater agreement and accuracy of ASPECTS score evaluation in a large group of neuroradiologists and neurologists. e-ASPECTS software was more predictive of final infarct/ASPECTS than the overall group interpreting the CT scans with and without e-ASPECTS assistance.",

keywords = "ASPECTS, CT, Stroke, artificial intelligence, large vessel occlusion",

author = "Waleed Brinjikji and Mehdi Abbasi and Catherine Arnold and Benson, {John C.} and Braksick, {Sherry A.} and Norbert Campeau and Carr, {Carrie M.} and Cogswell, {Petrice M.} and Klaas, {James P.} and Liebo, {Greta B.} and Little, {Jason T.} and Luetmer, {Patrick H.} and Messina, {Steven A.} and Nagelschneider, {Alex A.} and Schwartz, {Kara M.} and Wood, {Christopher P.} and Nasr, {Deena M.} and Kallmes, {David F.}",

note = "Publisher Copyright: {\textcopyright} The Author(s) 2021.",

year = "2021",

month = dec,

doi = "10.1177/15910199211011861",

language = "English (US)",

volume = "27",

pages = "781--787",

journal = "Interventional Neuroradiology",

issn = "1591-0199",

publisher = "Centauro srl",

number = "6",

}

TY - JOUR

T1 - e-ASPECTS software improves interobserver agreement and accuracy of interpretation of aspects score

AU - Brinjikji, Waleed

AU - Abbasi, Mehdi

AU - Arnold, Catherine

AU - Benson, John C.

AU - Braksick, Sherry A.

AU - Campeau, Norbert

AU - Carr, Carrie M.

AU - Cogswell, Petrice M.

AU - Klaas, James P.

AU - Liebo, Greta B.

AU - Little, Jason T.

AU - Luetmer, Patrick H.

AU - Messina, Steven A.

AU - Nagelschneider, Alex A.

AU - Schwartz, Kara M.

AU - Wood, Christopher P.

AU - Nasr, Deena M.

AU - Kallmes, David F.

PY - 2021/12

Y1 - 2021/12

N2 - Introduction: There is increased interest in the use of artificial intelligence-based (AI) software packages in the evaluation of neuroimaging studies for acute ischemic stroke. We studied whether, compared to standard image interpretation without AI, Brainomix e-ASPECTS software improved interobserver agreement and accuracy in detecting ASPECTS regions affected in anterior circulation LVO. Methods: We included 60 consecutive patients with anterior circulation LVO who had TICI 3 revascularization within 60 minutes of their baseline CT. A total of 16 readers, including senior neuroradiologists, junior neuroradiologists and vascular neurologists participated. Readers interpreted CT scans on independent workstations and assessed final ASPECTS and evaluated whether each individual ASPECTS region was affected. Two months later, readers again evaluated the CT scans, but with assistance of e-ASPECTS software. We assessed interclass correlation coefficient for total ASPECTS and interobserver agreement with Fleiss’ Kappa for each ASPECTS region with and without assistance of the e-ASPECTS. We also assessed accuracy for the readers with and without e-ASPECTS assistance. In our assessment of accuracy, ground truth was the 24 hour CT in this cohort of patients who had prompt and complete revascularization. Results: Interclass correlation coefficient for total ASPECTS without e-ASPECTS assistance was 0.395, indicating fair agreement compared, to 0.574 with e-ASPECTS assistance, indicating good agreement (P < 0.01). There was significant improvement in inter-rater agreement with e-ASPECTS assistance for each individual region with the exception of M6 and caudate. The e-ASPECTS software had higher accuracy than the overall cohort of readers (with and without e-ASPECTS assistance) for every region except the caudate. Conclusions: Use of Brainomix e-ASPECTS software resulted in significant improvements in inter-rater agreement and accuracy of ASPECTS score evaluation in a large group of neuroradiologists and neurologists. e-ASPECTS software was more predictive of final infarct/ASPECTS than the overall group interpreting the CT scans with and without e-ASPECTS assistance.

AB - Introduction: There is increased interest in the use of artificial intelligence-based (AI) software packages in the evaluation of neuroimaging studies for acute ischemic stroke. We studied whether, compared to standard image interpretation without AI, Brainomix e-ASPECTS software improved interobserver agreement and accuracy in detecting ASPECTS regions affected in anterior circulation LVO. Methods: We included 60 consecutive patients with anterior circulation LVO who had TICI 3 revascularization within 60 minutes of their baseline CT. A total of 16 readers, including senior neuroradiologists, junior neuroradiologists and vascular neurologists participated. Readers interpreted CT scans on independent workstations and assessed final ASPECTS and evaluated whether each individual ASPECTS region was affected. Two months later, readers again evaluated the CT scans, but with assistance of e-ASPECTS software. We assessed interclass correlation coefficient for total ASPECTS and interobserver agreement with Fleiss’ Kappa for each ASPECTS region with and without assistance of the e-ASPECTS. We also assessed accuracy for the readers with and without e-ASPECTS assistance. In our assessment of accuracy, ground truth was the 24 hour CT in this cohort of patients who had prompt and complete revascularization. Results: Interclass correlation coefficient for total ASPECTS without e-ASPECTS assistance was 0.395, indicating fair agreement compared, to 0.574 with e-ASPECTS assistance, indicating good agreement (P < 0.01). There was significant improvement in inter-rater agreement with e-ASPECTS assistance for each individual region with the exception of M6 and caudate. The e-ASPECTS software had higher accuracy than the overall cohort of readers (with and without e-ASPECTS assistance) for every region except the caudate. Conclusions: Use of Brainomix e-ASPECTS software resulted in significant improvements in inter-rater agreement and accuracy of ASPECTS score evaluation in a large group of neuroradiologists and neurologists. e-ASPECTS software was more predictive of final infarct/ASPECTS than the overall group interpreting the CT scans with and without e-ASPECTS assistance.

KW - ASPECTS

KW - CT

KW - Stroke

KW - artificial intelligence

KW - large vessel occlusion

UR - http://www.scopus.com/inward/record.url?scp=85104440160&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85104440160&partnerID=8YFLogxK

U2 - 10.1177/15910199211011861

DO - 10.1177/15910199211011861

M3 - Article

C2 - 33853441

AN - SCOPUS:85104440160

SN - 1591-0199

VL - 27

SP - 781

EP - 787

JO - Interventional Neuroradiology

JF - Interventional Neuroradiology

IS - 6

ER -

e-ASPECTS software improves interobserver agreement and accuracy of interpretation of aspects score

Abstract

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this