TY - JOUR
T1 - Testing the Relative Performance of Data Adaptive Prediction Algorithms
T2 - A Generalized Test of Conditional Risk Differences
AU - Goldstein, Benjamin A.
AU - Polley, Eric C.
AU - Briggs, Farren B.S.
AU - Van Der Laan, Mark J.
AU - Hubbard, Alan
N1 - Publisher Copyright:
© 2016 by De Gruyter.
PY - 2016/5/1
Y1 - 2016/5/1
N2 - Comparing the relative fit of competing models can be used to address many different scientific questions. In classical statistics one can, if appropriate, use likelihood ratio tests and information based criterion, whereas clinical medicine has tended to rely on comparisons of fit metrics like C-statistics. However, for many data adaptive modelling procedures such approaches are not suitable. In these cases, statisticians have used cross-validation, which can make inference challenging. In this paper we propose a general approach that focuses on the "conditional" risk difference (conditional on the model fits being fixed) for the improvement in prediction risk. Specifically, we derive a Wald-type test statistic and associated confidence intervals for cross-validated test sets utilizing the independent validation within cross-validation in conjunction with a test for multiple comparisons. We show that this test maintains proper Type I Error under the null fit, and can be used as a general test of relative fit for any semiparametric model alternative. We apply the test to a candidate gene study to test for the association of a set of genes in a genetic pathway.
AB - Comparing the relative fit of competing models can be used to address many different scientific questions. In classical statistics one can, if appropriate, use likelihood ratio tests and information based criterion, whereas clinical medicine has tended to rely on comparisons of fit metrics like C-statistics. However, for many data adaptive modelling procedures such approaches are not suitable. In these cases, statisticians have used cross-validation, which can make inference challenging. In this paper we propose a general approach that focuses on the "conditional" risk difference (conditional on the model fits being fixed) for the improvement in prediction risk. Specifically, we derive a Wald-type test statistic and associated confidence intervals for cross-validated test sets utilizing the independent validation within cross-validation in conjunction with a test for multiple comparisons. We show that this test maintains proper Type I Error under the null fit, and can be used as a general test of relative fit for any semiparametric model alternative. We apply the test to a candidate gene study to test for the association of a set of genes in a genetic pathway.
KW - Risk prediction
KW - cross-validation
KW - machine learning
KW - semi-parametric models
UR - http://www.scopus.com/inward/record.url?scp=84975253355&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84975253355&partnerID=8YFLogxK
U2 - 10.1515/ijb-2015-0014
DO - 10.1515/ijb-2015-0014
M3 - Article
C2 - 26529567
AN - SCOPUS:84975253355
SN - 1557-4679
VL - 12
SP - 117
EP - 129
JO - International Journal of Biostatistics
JF - International Journal of Biostatistics
IS - 1
ER -