Impact of unaccounted risk factors on the interpretation of surgical outcomes

David A. Etzioni, Nabil Wasif, Amit Mathur, Elizabeth B Habermann, Robert R. Cima, Yu Hui H Chang

Research output: Contribution to journalArticle

5 Citations (Scopus)

Abstract

Background Systems that report hospital-based risk-adjusted surgical outcomes are potentially sensitive to the underlying methods used for risk adjustment. If a body of operations has a true level of risk that is higher than the estimated risk, then these operations might generate bias in the output of these reports. The objective of this study was to quantify the impact of unaccounted risk on the results of a surgical outcomes report. Study Design We constructed a model simulating a universe of 500 hospitals, each providing care to 1,500 patients in a given year. The likelihood of morbidity and mortality for each of these patients was drawn from a random sampling of patients in the American College of Surgeons NSQIP. A single additional hospital was also simulated, within which a certain proportion (proportion varied from 2% to 10%) of patients had a significantly higher (odds ratio varied from 1 to 5) actual likelihood of mortality. Results The presence of even a small proportion (2%) of patients with unaccounted risk had the potential to greatly increase the likelihood of a hospital being considered a statistical outlier (poor performer). This impact was greater in the assessment of complications than mortality. Conclusions This study shows that even a small proportion of patients with substantial levels of unaccounted risk can have a dramatic impact on the assessment of hospital-level risk-adjusted surgical outcomes. To avoid the unintended consequences associated with risk-averse behavior from providers, policy should be constructed to address this potential source of bias.

Original languageEnglish (US)
Pages (from-to)821-827
Number of pages7
JournalJournal of the American College of Surgeons
Volume221
Issue number4
DOIs
StatePublished - Oct 1 2015

Fingerprint

Mortality
Risk Adjustment
Odds Ratio
Morbidity

ASJC Scopus subject areas

  • Surgery

Cite this

Impact of unaccounted risk factors on the interpretation of surgical outcomes. / Etzioni, David A.; Wasif, Nabil; Mathur, Amit; Habermann, Elizabeth B; Cima, Robert R.; Chang, Yu Hui H.

In: Journal of the American College of Surgeons, Vol. 221, No. 4, 01.10.2015, p. 821-827.

Research output: Contribution to journalArticle

@article{d09381b6cb97444cb7f586cc6813bd86,
title = "Impact of unaccounted risk factors on the interpretation of surgical outcomes",
abstract = "Background Systems that report hospital-based risk-adjusted surgical outcomes are potentially sensitive to the underlying methods used for risk adjustment. If a body of operations has a true level of risk that is higher than the estimated risk, then these operations might generate bias in the output of these reports. The objective of this study was to quantify the impact of unaccounted risk on the results of a surgical outcomes report. Study Design We constructed a model simulating a universe of 500 hospitals, each providing care to 1,500 patients in a given year. The likelihood of morbidity and mortality for each of these patients was drawn from a random sampling of patients in the American College of Surgeons NSQIP. A single additional hospital was also simulated, within which a certain proportion (proportion varied from 2{\%} to 10{\%}) of patients had a significantly higher (odds ratio varied from 1 to 5) actual likelihood of mortality. Results The presence of even a small proportion (2{\%}) of patients with unaccounted risk had the potential to greatly increase the likelihood of a hospital being considered a statistical outlier (poor performer). This impact was greater in the assessment of complications than mortality. Conclusions This study shows that even a small proportion of patients with substantial levels of unaccounted risk can have a dramatic impact on the assessment of hospital-level risk-adjusted surgical outcomes. To avoid the unintended consequences associated with risk-averse behavior from providers, policy should be constructed to address this potential source of bias.",
author = "Etzioni, {David A.} and Nabil Wasif and Amit Mathur and Habermann, {Elizabeth B} and Cima, {Robert R.} and Chang, {Yu Hui H}",
year = "2015",
month = "10",
day = "1",
doi = "10.1016/j.jamcollsurg.2015.06.015",
language = "English (US)",
volume = "221",
pages = "821--827",
journal = "Journal of the American College of Surgeons",
issn = "1072-7515",
publisher = "Elsevier Inc.",
number = "4",

}

TY - JOUR

T1 - Impact of unaccounted risk factors on the interpretation of surgical outcomes

AU - Etzioni, David A.

AU - Wasif, Nabil

AU - Mathur, Amit

AU - Habermann, Elizabeth B

AU - Cima, Robert R.

AU - Chang, Yu Hui H

PY - 2015/10/1

Y1 - 2015/10/1

N2 - Background Systems that report hospital-based risk-adjusted surgical outcomes are potentially sensitive to the underlying methods used for risk adjustment. If a body of operations has a true level of risk that is higher than the estimated risk, then these operations might generate bias in the output of these reports. The objective of this study was to quantify the impact of unaccounted risk on the results of a surgical outcomes report. Study Design We constructed a model simulating a universe of 500 hospitals, each providing care to 1,500 patients in a given year. The likelihood of morbidity and mortality for each of these patients was drawn from a random sampling of patients in the American College of Surgeons NSQIP. A single additional hospital was also simulated, within which a certain proportion (proportion varied from 2% to 10%) of patients had a significantly higher (odds ratio varied from 1 to 5) actual likelihood of mortality. Results The presence of even a small proportion (2%) of patients with unaccounted risk had the potential to greatly increase the likelihood of a hospital being considered a statistical outlier (poor performer). This impact was greater in the assessment of complications than mortality. Conclusions This study shows that even a small proportion of patients with substantial levels of unaccounted risk can have a dramatic impact on the assessment of hospital-level risk-adjusted surgical outcomes. To avoid the unintended consequences associated with risk-averse behavior from providers, policy should be constructed to address this potential source of bias.

AB - Background Systems that report hospital-based risk-adjusted surgical outcomes are potentially sensitive to the underlying methods used for risk adjustment. If a body of operations has a true level of risk that is higher than the estimated risk, then these operations might generate bias in the output of these reports. The objective of this study was to quantify the impact of unaccounted risk on the results of a surgical outcomes report. Study Design We constructed a model simulating a universe of 500 hospitals, each providing care to 1,500 patients in a given year. The likelihood of morbidity and mortality for each of these patients was drawn from a random sampling of patients in the American College of Surgeons NSQIP. A single additional hospital was also simulated, within which a certain proportion (proportion varied from 2% to 10%) of patients had a significantly higher (odds ratio varied from 1 to 5) actual likelihood of mortality. Results The presence of even a small proportion (2%) of patients with unaccounted risk had the potential to greatly increase the likelihood of a hospital being considered a statistical outlier (poor performer). This impact was greater in the assessment of complications than mortality. Conclusions This study shows that even a small proportion of patients with substantial levels of unaccounted risk can have a dramatic impact on the assessment of hospital-level risk-adjusted surgical outcomes. To avoid the unintended consequences associated with risk-averse behavior from providers, policy should be constructed to address this potential source of bias.

UR - http://www.scopus.com/inward/record.url?scp=84941414743&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84941414743&partnerID=8YFLogxK

U2 - 10.1016/j.jamcollsurg.2015.06.015

DO - 10.1016/j.jamcollsurg.2015.06.015

M3 - Article

C2 - 26253561

AN - SCOPUS:84941414743

VL - 221

SP - 821

EP - 827

JO - Journal of the American College of Surgeons

JF - Journal of the American College of Surgeons

SN - 1072-7515

IS - 4

ER -