Multicenter derivation and validation of an early warning score for acute respiratory failure or death in the hospital

Mikhail A. Dziadzko, Paul J. Novotny, Jeff A Sloan, Ognjen Gajic, Vitaly D Herasevich, Parsa Mirhaji, Yiyuan Wu, Michelle Ng Gong

Research output: Contribution to journalArticle

Abstract

Background: Acute respiratory failure occurs frequently in hospitalized patients and often starts before ICU admission. A risk stratification tool to predict mortality and risk for mechanical ventilation (MV) may allow for earlier evaluation and intervention. We developed and validated an automated electronic health record (EHR)-based model - Accurate Prediction of Prolonged Ventilation (APPROVE) - to identify patients at risk of death or respiratory failure requiring >= 48 h of MV. Methods: This was an observational study of adults admitted to four hospitals in 2013 or a fifth hospital in 2017. Clinical data were extracted from the EHRs. The 2013 patients were randomly split 50:50 into a derivation/validation cohort. The qualifying event was death or intubation leading to MV >= 48 h. Random forest method was used in model derivation. APPROVE was calculated retrospectively whenever data were available in 2013, and prospectively every 4 h after hospital admission in 2017. The Modified Early Warning Score (MEWS) and National Early Warning Score (NEWS) were calculated at the same times as APPROVE. Clinicians were not alerted except for APPROVE in 2017cohort. Results: There were 68,775 admissions in 2013 and 2258 in 2017. APPROVE had an area under the receiver operator curve of 0.87 (95% CI 0.85-0.88) in 2013 and 0.90 (95% CI 0.84-0.95) in 2017, which is significantly better than the MEWS and NEWS in 2013 but similar to the MEWS and NEWS in 2017. At a threshold of > 0.25, APPROVE had similar sensitivity and positive predictive value (PPV) (sensitivity 63% and PPV 21% in 2013 vs 64% and 16%, respectively, in 2017). Compared to APPROVE in 2013, at a threshold to achieve comparable PPV (19% at MEWS > 4 and 22% at NEWS > 6), the MEWS and NEWS had lower sensitivity (16% for MEWS and NEWS). Similarly in 2017, at a comparable sensitivity threshold (64% for APPROVE > 0.25 and 67% for MEWS and NEWS > 4), more patients who triggered an alert developed the event with APPROVE (PPV 16%) while achieving a lower false positive rate (FPR 5%) compared to the MEWS (PPV 7%, FPR 14%) and NEWS (PPV 4%, FPR 25%). Conclusions: An automated EHR model to identify patients at high risk of MV or death was validated retrospectively and prospectively, and was determined to be feasible for real-time risk identification. Trial registration: ClinicalTrials.gov, NCT02488174. Registered on 18 March 2015.

Original languageEnglish (US)
Article number286
JournalCritical Care
Volume22
Issue number1
DOIs
StatePublished - Oct 30 2018

Fingerprint

Respiratory Insufficiency
Ventilation
Artificial Respiration
Electronic Health Records
Intubation
Observational Studies
Mortality

Keywords

  • Acute respiratory failure
  • Early warning scores
  • Electronic health records
  • Prediction
  • Random forest

ASJC Scopus subject areas

  • Critical Care and Intensive Care Medicine

Cite this

Multicenter derivation and validation of an early warning score for acute respiratory failure or death in the hospital. / Dziadzko, Mikhail A.; Novotny, Paul J.; Sloan, Jeff A; Gajic, Ognjen; Herasevich, Vitaly D; Mirhaji, Parsa; Wu, Yiyuan; Gong, Michelle Ng.

In: Critical Care, Vol. 22, No. 1, 286, 30.10.2018.

Research output: Contribution to journalArticle

@article{19ed8aa49f1f481897a7b5695db8ba03,
title = "Multicenter derivation and validation of an early warning score for acute respiratory failure or death in the hospital",
abstract = "Background: Acute respiratory failure occurs frequently in hospitalized patients and often starts before ICU admission. A risk stratification tool to predict mortality and risk for mechanical ventilation (MV) may allow for earlier evaluation and intervention. We developed and validated an automated electronic health record (EHR)-based model - Accurate Prediction of Prolonged Ventilation (APPROVE) - to identify patients at risk of death or respiratory failure requiring >= 48 h of MV. Methods: This was an observational study of adults admitted to four hospitals in 2013 or a fifth hospital in 2017. Clinical data were extracted from the EHRs. The 2013 patients were randomly split 50:50 into a derivation/validation cohort. The qualifying event was death or intubation leading to MV >= 48 h. Random forest method was used in model derivation. APPROVE was calculated retrospectively whenever data were available in 2013, and prospectively every 4 h after hospital admission in 2017. The Modified Early Warning Score (MEWS) and National Early Warning Score (NEWS) were calculated at the same times as APPROVE. Clinicians were not alerted except for APPROVE in 2017cohort. Results: There were 68,775 admissions in 2013 and 2258 in 2017. APPROVE had an area under the receiver operator curve of 0.87 (95{\%} CI 0.85-0.88) in 2013 and 0.90 (95{\%} CI 0.84-0.95) in 2017, which is significantly better than the MEWS and NEWS in 2013 but similar to the MEWS and NEWS in 2017. At a threshold of > 0.25, APPROVE had similar sensitivity and positive predictive value (PPV) (sensitivity 63{\%} and PPV 21{\%} in 2013 vs 64{\%} and 16{\%}, respectively, in 2017). Compared to APPROVE in 2013, at a threshold to achieve comparable PPV (19{\%} at MEWS > 4 and 22{\%} at NEWS > 6), the MEWS and NEWS had lower sensitivity (16{\%} for MEWS and NEWS). Similarly in 2017, at a comparable sensitivity threshold (64{\%} for APPROVE > 0.25 and 67{\%} for MEWS and NEWS > 4), more patients who triggered an alert developed the event with APPROVE (PPV 16{\%}) while achieving a lower false positive rate (FPR 5{\%}) compared to the MEWS (PPV 7{\%}, FPR 14{\%}) and NEWS (PPV 4{\%}, FPR 25{\%}). Conclusions: An automated EHR model to identify patients at high risk of MV or death was validated retrospectively and prospectively, and was determined to be feasible for real-time risk identification. Trial registration: ClinicalTrials.gov, NCT02488174. Registered on 18 March 2015.",
keywords = "Acute respiratory failure, Early warning scores, Electronic health records, Prediction, Random forest",
author = "Dziadzko, {Mikhail A.} and Novotny, {Paul J.} and Sloan, {Jeff A} and Ognjen Gajic and Herasevich, {Vitaly D} and Parsa Mirhaji and Yiyuan Wu and Gong, {Michelle Ng}",
year = "2018",
month = "10",
day = "30",
doi = "10.1186/s13054-018-2194-7",
language = "English (US)",
volume = "22",
journal = "Critical Care",
issn = "1364-8535",
publisher = "BioMed Central Ltd.",
number = "1",

}

TY - JOUR

T1 - Multicenter derivation and validation of an early warning score for acute respiratory failure or death in the hospital

AU - Dziadzko, Mikhail A.

AU - Novotny, Paul J.

AU - Sloan, Jeff A

AU - Gajic, Ognjen

AU - Herasevich, Vitaly D

AU - Mirhaji, Parsa

AU - Wu, Yiyuan

AU - Gong, Michelle Ng

PY - 2018/10/30

Y1 - 2018/10/30

N2 - Background: Acute respiratory failure occurs frequently in hospitalized patients and often starts before ICU admission. A risk stratification tool to predict mortality and risk for mechanical ventilation (MV) may allow for earlier evaluation and intervention. We developed and validated an automated electronic health record (EHR)-based model - Accurate Prediction of Prolonged Ventilation (APPROVE) - to identify patients at risk of death or respiratory failure requiring >= 48 h of MV. Methods: This was an observational study of adults admitted to four hospitals in 2013 or a fifth hospital in 2017. Clinical data were extracted from the EHRs. The 2013 patients were randomly split 50:50 into a derivation/validation cohort. The qualifying event was death or intubation leading to MV >= 48 h. Random forest method was used in model derivation. APPROVE was calculated retrospectively whenever data were available in 2013, and prospectively every 4 h after hospital admission in 2017. The Modified Early Warning Score (MEWS) and National Early Warning Score (NEWS) were calculated at the same times as APPROVE. Clinicians were not alerted except for APPROVE in 2017cohort. Results: There were 68,775 admissions in 2013 and 2258 in 2017. APPROVE had an area under the receiver operator curve of 0.87 (95% CI 0.85-0.88) in 2013 and 0.90 (95% CI 0.84-0.95) in 2017, which is significantly better than the MEWS and NEWS in 2013 but similar to the MEWS and NEWS in 2017. At a threshold of > 0.25, APPROVE had similar sensitivity and positive predictive value (PPV) (sensitivity 63% and PPV 21% in 2013 vs 64% and 16%, respectively, in 2017). Compared to APPROVE in 2013, at a threshold to achieve comparable PPV (19% at MEWS > 4 and 22% at NEWS > 6), the MEWS and NEWS had lower sensitivity (16% for MEWS and NEWS). Similarly in 2017, at a comparable sensitivity threshold (64% for APPROVE > 0.25 and 67% for MEWS and NEWS > 4), more patients who triggered an alert developed the event with APPROVE (PPV 16%) while achieving a lower false positive rate (FPR 5%) compared to the MEWS (PPV 7%, FPR 14%) and NEWS (PPV 4%, FPR 25%). Conclusions: An automated EHR model to identify patients at high risk of MV or death was validated retrospectively and prospectively, and was determined to be feasible for real-time risk identification. Trial registration: ClinicalTrials.gov, NCT02488174. Registered on 18 March 2015.

AB - Background: Acute respiratory failure occurs frequently in hospitalized patients and often starts before ICU admission. A risk stratification tool to predict mortality and risk for mechanical ventilation (MV) may allow for earlier evaluation and intervention. We developed and validated an automated electronic health record (EHR)-based model - Accurate Prediction of Prolonged Ventilation (APPROVE) - to identify patients at risk of death or respiratory failure requiring >= 48 h of MV. Methods: This was an observational study of adults admitted to four hospitals in 2013 or a fifth hospital in 2017. Clinical data were extracted from the EHRs. The 2013 patients were randomly split 50:50 into a derivation/validation cohort. The qualifying event was death or intubation leading to MV >= 48 h. Random forest method was used in model derivation. APPROVE was calculated retrospectively whenever data were available in 2013, and prospectively every 4 h after hospital admission in 2017. The Modified Early Warning Score (MEWS) and National Early Warning Score (NEWS) were calculated at the same times as APPROVE. Clinicians were not alerted except for APPROVE in 2017cohort. Results: There were 68,775 admissions in 2013 and 2258 in 2017. APPROVE had an area under the receiver operator curve of 0.87 (95% CI 0.85-0.88) in 2013 and 0.90 (95% CI 0.84-0.95) in 2017, which is significantly better than the MEWS and NEWS in 2013 but similar to the MEWS and NEWS in 2017. At a threshold of > 0.25, APPROVE had similar sensitivity and positive predictive value (PPV) (sensitivity 63% and PPV 21% in 2013 vs 64% and 16%, respectively, in 2017). Compared to APPROVE in 2013, at a threshold to achieve comparable PPV (19% at MEWS > 4 and 22% at NEWS > 6), the MEWS and NEWS had lower sensitivity (16% for MEWS and NEWS). Similarly in 2017, at a comparable sensitivity threshold (64% for APPROVE > 0.25 and 67% for MEWS and NEWS > 4), more patients who triggered an alert developed the event with APPROVE (PPV 16%) while achieving a lower false positive rate (FPR 5%) compared to the MEWS (PPV 7%, FPR 14%) and NEWS (PPV 4%, FPR 25%). Conclusions: An automated EHR model to identify patients at high risk of MV or death was validated retrospectively and prospectively, and was determined to be feasible for real-time risk identification. Trial registration: ClinicalTrials.gov, NCT02488174. Registered on 18 March 2015.

KW - Acute respiratory failure

KW - Early warning scores

KW - Electronic health records

KW - Prediction

KW - Random forest

UR - http://www.scopus.com/inward/record.url?scp=85055615148&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85055615148&partnerID=8YFLogxK

U2 - 10.1186/s13054-018-2194-7

DO - 10.1186/s13054-018-2194-7

M3 - Article

C2 - 30373653

AN - SCOPUS:85055615148

VL - 22

JO - Critical Care

JF - Critical Care

SN - 1364-8535

IS - 1

M1 - 286

ER -