Testing the Relative Performance of Data Adaptive Prediction Algorithms: A Generalized Test of Conditional Risk Differences

Benjamin A. Goldstein, Eric C. Polley, Farren B.S. Briggs, Mark J. Van Der Laan, Alan Hubbard

Research output: Contribution to journalArticlepeer-review

1 Scopus citations

Abstract

Comparing the relative fit of competing models can be used to address many different scientific questions. In classical statistics one can, if appropriate, use likelihood ratio tests and information based criterion, whereas clinical medicine has tended to rely on comparisons of fit metrics like C-statistics. However, for many data adaptive modelling procedures such approaches are not suitable. In these cases, statisticians have used cross-validation, which can make inference challenging. In this paper we propose a general approach that focuses on the "conditional" risk difference (conditional on the model fits being fixed) for the improvement in prediction risk. Specifically, we derive a Wald-type test statistic and associated confidence intervals for cross-validated test sets utilizing the independent validation within cross-validation in conjunction with a test for multiple comparisons. We show that this test maintains proper Type I Error under the null fit, and can be used as a general test of relative fit for any semiparametric model alternative. We apply the test to a candidate gene study to test for the association of a set of genes in a genetic pathway.

Original languageEnglish (US)
Pages (from-to)117-129
Number of pages13
JournalInternational Journal of Biostatistics
Volume12
Issue number1
DOIs
StatePublished - May 1 2016

Keywords

  • Risk prediction
  • cross-validation
  • machine learning
  • semi-parametric models

ASJC Scopus subject areas

  • Statistics and Probability
  • Statistics, Probability and Uncertainty

Fingerprint

Dive into the research topics of 'Testing the Relative Performance of Data Adaptive Prediction Algorithms: A Generalized Test of Conditional Risk Differences'. Together they form a unique fingerprint.

Cite this