Abstract
Publication bias, more generally termed as small-study effect, is a major threat to the validity of meta-analyses. Most meta-analysts rely on the p values from statistical tests to make a binary decision about the presence or absence of small-study effects. Measures are available to quantify small-study effects' magnitude, but the current literature lacks clear rules to help evidence users in judging whether such effects are minimal or substantial. This article aims to provide rules of thumb for interpreting the measures. We use six measures to evaluate small-study effects in 29 932 meta-analyses from the Cochrane Database of Systematic Reviews. They include Egger's regression intercept and the skewness under both the fixed-effect and random-effects settings, the proportion of suppressed studies, and the relative change of the estimated overall result due to small-study effects. The cut-offs for different extents of small-study effects are determined based on the quantiles in these distributions. We present the empirical distributions of the six measures and propose a rough guide to interpret the measures' magnitude. The proposed rules of thumb may help evidence users grade the certainty in evidence as impacted by small-study effects.
| Original language | English (US) |
|---|---|
| Journal | BMJ evidence-based medicine |
| DOIs | |
| State | Published - Jan 1 2019 |
Fingerprint
Keywords
- epidemiology
ASJC Scopus subject areas
- Medicine(all)
Cite this
The magnitude of small-study effects in the Cochrane Database of Systematic Reviews : An empirical study of nearly 30 000 meta-analyses. / Lin, Lifeng; Shi, Linyu; Chu, Haitao; Murad, Mohammad H.
In: BMJ evidence-based medicine, 01.01.2019.Research output: Contribution to journal › Article
}
TY - JOUR
T1 - The magnitude of small-study effects in the Cochrane Database of Systematic Reviews
T2 - An empirical study of nearly 30 000 meta-analyses
AU - Lin, Lifeng
AU - Shi, Linyu
AU - Chu, Haitao
AU - Murad, Mohammad H
PY - 2019/1/1
Y1 - 2019/1/1
N2 - Publication bias, more generally termed as small-study effect, is a major threat to the validity of meta-analyses. Most meta-analysts rely on the p values from statistical tests to make a binary decision about the presence or absence of small-study effects. Measures are available to quantify small-study effects' magnitude, but the current literature lacks clear rules to help evidence users in judging whether such effects are minimal or substantial. This article aims to provide rules of thumb for interpreting the measures. We use six measures to evaluate small-study effects in 29 932 meta-analyses from the Cochrane Database of Systematic Reviews. They include Egger's regression intercept and the skewness under both the fixed-effect and random-effects settings, the proportion of suppressed studies, and the relative change of the estimated overall result due to small-study effects. The cut-offs for different extents of small-study effects are determined based on the quantiles in these distributions. We present the empirical distributions of the six measures and propose a rough guide to interpret the measures' magnitude. The proposed rules of thumb may help evidence users grade the certainty in evidence as impacted by small-study effects.
AB - Publication bias, more generally termed as small-study effect, is a major threat to the validity of meta-analyses. Most meta-analysts rely on the p values from statistical tests to make a binary decision about the presence or absence of small-study effects. Measures are available to quantify small-study effects' magnitude, but the current literature lacks clear rules to help evidence users in judging whether such effects are minimal or substantial. This article aims to provide rules of thumb for interpreting the measures. We use six measures to evaluate small-study effects in 29 932 meta-analyses from the Cochrane Database of Systematic Reviews. They include Egger's regression intercept and the skewness under both the fixed-effect and random-effects settings, the proportion of suppressed studies, and the relative change of the estimated overall result due to small-study effects. The cut-offs for different extents of small-study effects are determined based on the quantiles in these distributions. We present the empirical distributions of the six measures and propose a rough guide to interpret the measures' magnitude. The proposed rules of thumb may help evidence users grade the certainty in evidence as impacted by small-study effects.
KW - epidemiology
UR - http://www.scopus.com/inward/record.url?scp=85068595022&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85068595022&partnerID=8YFLogxK
U2 - 10.1136/bmjebm-2019-111191
DO - 10.1136/bmjebm-2019-111191
M3 - Article
AN - SCOPUS:85068595022
JO - BMJ Evidence-Based Medicine
JF - BMJ Evidence-Based Medicine
SN - 2515-446X
ER -