Impact of unaccounted risk factors on the interpretation of surgical outcomes

Background Systems that report hospital-based risk-adjusted surgical outcomes are potentially sensitive to the underlying methods used for risk adjustment. If a body of operations has a true level of risk that is higher than the estimated risk, then these operations might generate bias in the output of these reports. The objective of this study was to quantify the impact of unaccounted risk on the results of a surgical outcomes report. Study Design We constructed a model simulating a universe of 500 hospitals, each providing care to 1,500 patients in a given year. The likelihood of morbidity and mortality for each of these patients was drawn from a random sampling of patients in the American College of Surgeons NSQIP. A single additional hospital was also simulated, within which a certain proportion (proportion varied from 2% to 10%) of patients had a significantly higher (odds ratio varied from 1 to 5) actual likelihood of mortality. Results The presence of even a small proportion (2%) of patients with unaccounted risk had the potential to greatly increase the likelihood of a hospital being considered a statistical outlier (poor performer). This impact was greater in the assessment of complications than mortality. Conclusions This study shows that even a small proportion of patients with substantial levels of unaccounted risk can have a dramatic impact on the assessment of hospital-level risk-adjusted surgical outcomes. To avoid the unintended consequences associated with risk-averse behavior from providers, policy should be constructed to address this potential source of bias.