TY - JOUR
T1 - The magnitude of small-study effects in the Cochrane Database of Systematic Reviews
T2 - An empirical study of nearly 30 000 meta-analyses
AU - Lin, Lifeng
AU - Shi, Linyu
AU - Chu, Haitao
AU - Murad, Mohammad Hassan
N1 - Funding Information:
Funding This research was supported in part by NIH NLM
Funding Information:
This research was supported in part by NIH NLM R21 012197 (HC, LL), NLM R21 012744 (HC), and AHRQ R03 HS024743 (HC, LL).
Publisher Copyright:
� Author(s) (or their employer(s)) 2020. No commercial re-use. See rights and permissions. Published by BMJ.
PY - 2020/2/1
Y1 - 2020/2/1
N2 - Publication bias, more generally termed as small-study effect, is a major threat to the validity of meta-analyses. Most meta-analysts rely on the p values from statistical tests to make a binary decision about the presence or absence of small-study effects. Measures are available to quantify small-study effects' magnitude, but the current literature lacks clear rules to help evidence users in judging whether such effects are minimal or substantial. This article aims to provide rules of thumb for interpreting the measures. We use six measures to evaluate small-study effects in 29 932 meta-analyses from the Cochrane Database of Systematic Reviews. They include Egger's regression intercept and the skewness under both the fixed-effect and random-effects settings, the proportion of suppressed studies, and the relative change of the estimated overall result due to small-study effects. The cut-offs for different extents of small-study effects are determined based on the quantiles in these distributions. We present the empirical distributions of the six measures and propose a rough guide to interpret the measures' magnitude. The proposed rules of thumb may help evidence users grade the certainty in evidence as impacted by small-study effects.
AB - Publication bias, more generally termed as small-study effect, is a major threat to the validity of meta-analyses. Most meta-analysts rely on the p values from statistical tests to make a binary decision about the presence or absence of small-study effects. Measures are available to quantify small-study effects' magnitude, but the current literature lacks clear rules to help evidence users in judging whether such effects are minimal or substantial. This article aims to provide rules of thumb for interpreting the measures. We use six measures to evaluate small-study effects in 29 932 meta-analyses from the Cochrane Database of Systematic Reviews. They include Egger's regression intercept and the skewness under both the fixed-effect and random-effects settings, the proportion of suppressed studies, and the relative change of the estimated overall result due to small-study effects. The cut-offs for different extents of small-study effects are determined based on the quantiles in these distributions. We present the empirical distributions of the six measures and propose a rough guide to interpret the measures' magnitude. The proposed rules of thumb may help evidence users grade the certainty in evidence as impacted by small-study effects.
KW - epidemiology
UR - http://www.scopus.com/inward/record.url?scp=85068595022&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85068595022&partnerID=8YFLogxK
U2 - 10.1136/bmjebm-2019-111191
DO - 10.1136/bmjebm-2019-111191
M3 - Article
C2 - 31273125
AN - SCOPUS:85068595022
VL - 25
SP - 27
EP - 32
JO - BMJ Evidence-Based Medicine
JF - BMJ Evidence-Based Medicine
SN - 2515-446X
IS - 1
ER -