Development and validation pathways of artificial intelligence tools evaluated in randomised clinical trials

George C.M. Siontis; Romy Sweda; Peter A. Noseworthy; Paul A. Friedman; Konstantinos C. Siontis; Chirag J. Patel

doi:10.1136/bmjhci-2021-100466

Development and validation pathways of artificial intelligence tools evaluated in randomised clinical trials

George C.M. Siontis, Romy Sweda, Peter A. Noseworthy, Paul A. Friedman, Konstantinos C. Siontis, Chirag J. Patel

Cardiovascular Medicine

Research output: Contribution to journal › Article › peer-review

Abstract

ObjectiveGiven the complexities of testing the translational capability of new artificial intelligence (AI) tools, we aimed to map the pathways of training/validation/testing in development process and external validation of AI tools evaluated in dedicated randomised controlled trials (AI-RCTs).MethodsWe searched for peer-reviewed protocols and completed AI-RCTs evaluating the clinical effectiveness of AI tools and identified development and validation studies of AI tools. We collected detailed information, and evaluated patterns of development and external validation of AI tools.ResultsWe found 23 AI-RCTs evaluating the clinical impact of 18 unique AI tools (2009–2021). Standard-of-care interventions were used in the control arms in all but one AI-RCT. Investigators did not provide access to the software code of the AI tool in any of the studies. Considering the primary outcome, the results were in favour of the AI intervention in 82% of the completed AI-RCTs (14 out of 17). We identified significant variation in the patterns of development, external validation and clinical evaluation approaches among different AI tools. A published development study was found only for 10 of the 18 AI tools. Median time from the publication of a development study to the respective AI-RCT was 1.4 years (IQR 0.2–2.2).ConclusionsWe found significant variation in the patterns of development and validation for AI tools before their evaluation in dedicated AI-RCTs. Published peer-reviewed protocols and completed AI-RCTs were also heterogeneous in design and reporting. Upcoming guidelines providing guidance for the development and clinical translation process aim to improve these aspects.

Original language	English (US)
Article number	e100466
Journal	BMJ Health and Care Informatics
Volume	28
Issue number	1
DOIs	https://doi.org/10.1136/bmjhci-2021-100466
State	Published - Dec 27 2021

Keywords

artificial intelligence
clinical
data science
decision support systems
machine learning
medical informatics

ASJC Scopus subject areas

Computer Science Applications
Health Informatics
Health Information Management

Access to Document

10.1136/bmjhci-2021-100466

Cite this

@article{cf6128264e6e44bda7d5cf6ef8a9ea75,

title = "Development and validation pathways of artificial intelligence tools evaluated in randomised clinical trials",

abstract = "ObjectiveGiven the complexities of testing the translational capability of new artificial intelligence (AI) tools, we aimed to map the pathways of training/validation/testing in development process and external validation of AI tools evaluated in dedicated randomised controlled trials (AI-RCTs).MethodsWe searched for peer-reviewed protocols and completed AI-RCTs evaluating the clinical effectiveness of AI tools and identified development and validation studies of AI tools. We collected detailed information, and evaluated patterns of development and external validation of AI tools.ResultsWe found 23 AI-RCTs evaluating the clinical impact of 18 unique AI tools (2009–2021). Standard-of-care interventions were used in the control arms in all but one AI-RCT. Investigators did not provide access to the software code of the AI tool in any of the studies. Considering the primary outcome, the results were in favour of the AI intervention in 82% of the completed AI-RCTs (14 out of 17). We identified significant variation in the patterns of development, external validation and clinical evaluation approaches among different AI tools. A published development study was found only for 10 of the 18 AI tools. Median time from the publication of a development study to the respective AI-RCT was 1.4 years (IQR 0.2–2.2).ConclusionsWe found significant variation in the patterns of development and validation for AI tools before their evaluation in dedicated AI-RCTs. Published peer-reviewed protocols and completed AI-RCTs were also heterogeneous in design and reporting. Upcoming guidelines providing guidance for the development and clinical translation process aim to improve these aspects.",

keywords = "artificial intelligence, clinical, data science, decision support systems, machine learning, medical informatics",

author = "Siontis, {George C.M.} and Romy Sweda and Noseworthy, {Peter A.} and Friedman, {Paul A.} and Siontis, {Konstantinos C.} and Patel, {Chirag J.}",

note = "Publisher Copyright: {\textcopyright} Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.",

year = "2021",

month = dec,

day = "27",

doi = "10.1136/bmjhci-2021-100466",

language = "English (US)",

volume = "28",

journal = "BMJ Health and Care Informatics",

issn = "2058-4555",

publisher = "BMJ Publishing Group",

number = "1",

}

TY - JOUR

T1 - Development and validation pathways of artificial intelligence tools evaluated in randomised clinical trials

AU - Siontis, George C.M.

AU - Sweda, Romy

AU - Noseworthy, Peter A.

AU - Friedman, Paul A.

AU - Siontis, Konstantinos C.

AU - Patel, Chirag J.

N1 - Publisher Copyright: © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

PY - 2021/12/27

Y1 - 2021/12/27

N2 - ObjectiveGiven the complexities of testing the translational capability of new artificial intelligence (AI) tools, we aimed to map the pathways of training/validation/testing in development process and external validation of AI tools evaluated in dedicated randomised controlled trials (AI-RCTs).MethodsWe searched for peer-reviewed protocols and completed AI-RCTs evaluating the clinical effectiveness of AI tools and identified development and validation studies of AI tools. We collected detailed information, and evaluated patterns of development and external validation of AI tools.ResultsWe found 23 AI-RCTs evaluating the clinical impact of 18 unique AI tools (2009–2021). Standard-of-care interventions were used in the control arms in all but one AI-RCT. Investigators did not provide access to the software code of the AI tool in any of the studies. Considering the primary outcome, the results were in favour of the AI intervention in 82% of the completed AI-RCTs (14 out of 17). We identified significant variation in the patterns of development, external validation and clinical evaluation approaches among different AI tools. A published development study was found only for 10 of the 18 AI tools. Median time from the publication of a development study to the respective AI-RCT was 1.4 years (IQR 0.2–2.2).ConclusionsWe found significant variation in the patterns of development and validation for AI tools before their evaluation in dedicated AI-RCTs. Published peer-reviewed protocols and completed AI-RCTs were also heterogeneous in design and reporting. Upcoming guidelines providing guidance for the development and clinical translation process aim to improve these aspects.

AB - ObjectiveGiven the complexities of testing the translational capability of new artificial intelligence (AI) tools, we aimed to map the pathways of training/validation/testing in development process and external validation of AI tools evaluated in dedicated randomised controlled trials (AI-RCTs).MethodsWe searched for peer-reviewed protocols and completed AI-RCTs evaluating the clinical effectiveness of AI tools and identified development and validation studies of AI tools. We collected detailed information, and evaluated patterns of development and external validation of AI tools.ResultsWe found 23 AI-RCTs evaluating the clinical impact of 18 unique AI tools (2009–2021). Standard-of-care interventions were used in the control arms in all but one AI-RCT. Investigators did not provide access to the software code of the AI tool in any of the studies. Considering the primary outcome, the results were in favour of the AI intervention in 82% of the completed AI-RCTs (14 out of 17). We identified significant variation in the patterns of development, external validation and clinical evaluation approaches among different AI tools. A published development study was found only for 10 of the 18 AI tools. Median time from the publication of a development study to the respective AI-RCT was 1.4 years (IQR 0.2–2.2).ConclusionsWe found significant variation in the patterns of development and validation for AI tools before their evaluation in dedicated AI-RCTs. Published peer-reviewed protocols and completed AI-RCTs were also heterogeneous in design and reporting. Upcoming guidelines providing guidance for the development and clinical translation process aim to improve these aspects.

KW - artificial intelligence

KW - clinical

KW - data science

KW - decision support systems

KW - machine learning

KW - medical informatics

UR - http://www.scopus.com/inward/record.url?scp=85122271050&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85122271050&partnerID=8YFLogxK

U2 - 10.1136/bmjhci-2021-100466

DO - 10.1136/bmjhci-2021-100466

M3 - Article

C2 - 34969668

AN - SCOPUS:85122271050

SN - 2058-4555

VL - 28

JO - BMJ Health and Care Informatics

JF - BMJ Health and Care Informatics

IS - 1

M1 - e100466

ER -

Development and validation pathways of artificial intelligence tools evaluated in randomised clinical trials

Abstract

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this