Risk of bias: a simulation study of power to detect study-level moderator effects in meta-analysis.

Susanne Hempel, Jeremy N V Miles, Marika J. Booth, Zhen Wang, Sally C. Morton, Paul G. Shekelle

Research output: Contribution to journalArticle

45 Citations (Scopus)

Abstract

There are both theoretical and empirical reasons to believe that design and execution factors are associated with bias in controlled trials. Statistically significant moderator effects, such as the effect of trial quality on treatment effect sizes, are rarely detected in individual meta-analyses, and evidence from meta-epidemiological datasets is inconsistent. The reasons for the disconnect between theory and empirical observation are unclear. The study objective was to explore the power to detect study level moderator effects in meta-analyses. We generated meta-analyses using Monte-Carlo simulations and investigated the effect of number of trials, trial sample size, moderator effect size, heterogeneity, and moderator distribution on power to detect moderator effects. The simulations provide a reference guide for investigators to estimate power when planning meta-regressions. The power to detect moderator effects in meta-analyses, for example, effects of study quality on effect sizes, is largely determined by the degree of residual heterogeneity present in the dataset (noise not explained by the moderator). Larger trial sample sizes increase power only when residual heterogeneity is low. A large number of trials or low residual heterogeneity are necessary to detect effects. When the proportion of the moderator is not equal (for example, 25% 'high quality', 75% 'low quality' trials), power of 80% was rarely achieved in investigated scenarios. Application to an empirical meta-epidemiological dataset with substantial heterogeneity (I(2) = 92%, τ(2) = 0.285) estimated >200 trials are needed for a power of 80% to show a statistically significant result, even for a substantial moderator effect (0.2), and the number of trials with the less common feature (for example, few 'high quality' studies) affects power extensively. Although study characteristics, such as trial quality, may explain some proportion of heterogeneity across study results in meta-analyses, residual heterogeneity is a crucial factor in determining when associations between moderator variables and effect sizes can be statistically detected. Detecting moderator effects requires more powerful analyses than are employed in most published investigations; hence negative findings should not be considered evidence of a lack of effect, and investigations are not hypothesis-proving unless power calculations show sufficient ability to detect effects.

Original languageEnglish (US)
Pages (from-to)107
Number of pages1
JournalSystematic Reviews
Volume2
StatePublished - 2013

Fingerprint

Meta-Analysis
Sample Size
Epidemiologic Effect Modifiers
Power (Psychology)
Aptitude
Noise
Research Personnel
Observation
Datasets

ASJC Scopus subject areas

  • Medicine (miscellaneous)

Cite this

Hempel, S., Miles, J. N. V., Booth, M. J., Wang, Z., Morton, S. C., & Shekelle, P. G. (2013). Risk of bias: a simulation study of power to detect study-level moderator effects in meta-analysis. Systematic Reviews, 2, 107.

Risk of bias : a simulation study of power to detect study-level moderator effects in meta-analysis. / Hempel, Susanne; Miles, Jeremy N V; Booth, Marika J.; Wang, Zhen; Morton, Sally C.; Shekelle, Paul G.

In: Systematic Reviews, Vol. 2, 2013, p. 107.

Research output: Contribution to journalArticle

Hempel, S, Miles, JNV, Booth, MJ, Wang, Z, Morton, SC & Shekelle, PG 2013, 'Risk of bias: a simulation study of power to detect study-level moderator effects in meta-analysis.', Systematic Reviews, vol. 2, pp. 107.
Hempel, Susanne ; Miles, Jeremy N V ; Booth, Marika J. ; Wang, Zhen ; Morton, Sally C. ; Shekelle, Paul G. / Risk of bias : a simulation study of power to detect study-level moderator effects in meta-analysis. In: Systematic Reviews. 2013 ; Vol. 2. pp. 107.
@article{ecc0db084f6e45d89691e481bbcae4c2,
title = "Risk of bias: a simulation study of power to detect study-level moderator effects in meta-analysis.",
abstract = "There are both theoretical and empirical reasons to believe that design and execution factors are associated with bias in controlled trials. Statistically significant moderator effects, such as the effect of trial quality on treatment effect sizes, are rarely detected in individual meta-analyses, and evidence from meta-epidemiological datasets is inconsistent. The reasons for the disconnect between theory and empirical observation are unclear. The study objective was to explore the power to detect study level moderator effects in meta-analyses. We generated meta-analyses using Monte-Carlo simulations and investigated the effect of number of trials, trial sample size, moderator effect size, heterogeneity, and moderator distribution on power to detect moderator effects. The simulations provide a reference guide for investigators to estimate power when planning meta-regressions. The power to detect moderator effects in meta-analyses, for example, effects of study quality on effect sizes, is largely determined by the degree of residual heterogeneity present in the dataset (noise not explained by the moderator). Larger trial sample sizes increase power only when residual heterogeneity is low. A large number of trials or low residual heterogeneity are necessary to detect effects. When the proportion of the moderator is not equal (for example, 25{\%} 'high quality', 75{\%} 'low quality' trials), power of 80{\%} was rarely achieved in investigated scenarios. Application to an empirical meta-epidemiological dataset with substantial heterogeneity (I(2) = 92{\%}, τ(2) = 0.285) estimated >200 trials are needed for a power of 80{\%} to show a statistically significant result, even for a substantial moderator effect (0.2), and the number of trials with the less common feature (for example, few 'high quality' studies) affects power extensively. Although study characteristics, such as trial quality, may explain some proportion of heterogeneity across study results in meta-analyses, residual heterogeneity is a crucial factor in determining when associations between moderator variables and effect sizes can be statistically detected. Detecting moderator effects requires more powerful analyses than are employed in most published investigations; hence negative findings should not be considered evidence of a lack of effect, and investigations are not hypothesis-proving unless power calculations show sufficient ability to detect effects.",
author = "Susanne Hempel and Miles, {Jeremy N V} and Booth, {Marika J.} and Zhen Wang and Morton, {Sally C.} and Shekelle, {Paul G.}",
year = "2013",
language = "English (US)",
volume = "2",
pages = "107",
journal = "Systematic Reviews",
issn = "2046-4053",
publisher = "BioMed Central",

}

TY - JOUR

T1 - Risk of bias

T2 - a simulation study of power to detect study-level moderator effects in meta-analysis.

AU - Hempel, Susanne

AU - Miles, Jeremy N V

AU - Booth, Marika J.

AU - Wang, Zhen

AU - Morton, Sally C.

AU - Shekelle, Paul G.

PY - 2013

Y1 - 2013

N2 - There are both theoretical and empirical reasons to believe that design and execution factors are associated with bias in controlled trials. Statistically significant moderator effects, such as the effect of trial quality on treatment effect sizes, are rarely detected in individual meta-analyses, and evidence from meta-epidemiological datasets is inconsistent. The reasons for the disconnect between theory and empirical observation are unclear. The study objective was to explore the power to detect study level moderator effects in meta-analyses. We generated meta-analyses using Monte-Carlo simulations and investigated the effect of number of trials, trial sample size, moderator effect size, heterogeneity, and moderator distribution on power to detect moderator effects. The simulations provide a reference guide for investigators to estimate power when planning meta-regressions. The power to detect moderator effects in meta-analyses, for example, effects of study quality on effect sizes, is largely determined by the degree of residual heterogeneity present in the dataset (noise not explained by the moderator). Larger trial sample sizes increase power only when residual heterogeneity is low. A large number of trials or low residual heterogeneity are necessary to detect effects. When the proportion of the moderator is not equal (for example, 25% 'high quality', 75% 'low quality' trials), power of 80% was rarely achieved in investigated scenarios. Application to an empirical meta-epidemiological dataset with substantial heterogeneity (I(2) = 92%, τ(2) = 0.285) estimated >200 trials are needed for a power of 80% to show a statistically significant result, even for a substantial moderator effect (0.2), and the number of trials with the less common feature (for example, few 'high quality' studies) affects power extensively. Although study characteristics, such as trial quality, may explain some proportion of heterogeneity across study results in meta-analyses, residual heterogeneity is a crucial factor in determining when associations between moderator variables and effect sizes can be statistically detected. Detecting moderator effects requires more powerful analyses than are employed in most published investigations; hence negative findings should not be considered evidence of a lack of effect, and investigations are not hypothesis-proving unless power calculations show sufficient ability to detect effects.

AB - There are both theoretical and empirical reasons to believe that design and execution factors are associated with bias in controlled trials. Statistically significant moderator effects, such as the effect of trial quality on treatment effect sizes, are rarely detected in individual meta-analyses, and evidence from meta-epidemiological datasets is inconsistent. The reasons for the disconnect between theory and empirical observation are unclear. The study objective was to explore the power to detect study level moderator effects in meta-analyses. We generated meta-analyses using Monte-Carlo simulations and investigated the effect of number of trials, trial sample size, moderator effect size, heterogeneity, and moderator distribution on power to detect moderator effects. The simulations provide a reference guide for investigators to estimate power when planning meta-regressions. The power to detect moderator effects in meta-analyses, for example, effects of study quality on effect sizes, is largely determined by the degree of residual heterogeneity present in the dataset (noise not explained by the moderator). Larger trial sample sizes increase power only when residual heterogeneity is low. A large number of trials or low residual heterogeneity are necessary to detect effects. When the proportion of the moderator is not equal (for example, 25% 'high quality', 75% 'low quality' trials), power of 80% was rarely achieved in investigated scenarios. Application to an empirical meta-epidemiological dataset with substantial heterogeneity (I(2) = 92%, τ(2) = 0.285) estimated >200 trials are needed for a power of 80% to show a statistically significant result, even for a substantial moderator effect (0.2), and the number of trials with the less common feature (for example, few 'high quality' studies) affects power extensively. Although study characteristics, such as trial quality, may explain some proportion of heterogeneity across study results in meta-analyses, residual heterogeneity is a crucial factor in determining when associations between moderator variables and effect sizes can be statistically detected. Detecting moderator effects requires more powerful analyses than are employed in most published investigations; hence negative findings should not be considered evidence of a lack of effect, and investigations are not hypothesis-proving unless power calculations show sufficient ability to detect effects.

UR - http://www.scopus.com/inward/record.url?scp=84899497147&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84899497147&partnerID=8YFLogxK

M3 - Article

C2 - 24286208

AN - SCOPUS:84899497147

VL - 2

SP - 107

JO - Systematic Reviews

JF - Systematic Reviews

SN - 2046-4053

ER -