Different meta-analysis methods can change judgements about imprecision of effect estimates: a meta-epidemiological study

Objectives: To empirically evaluate five commonly used meta-analysis methods and their impact on imprecision judgements about effect estimates. The two fixed-effect model methods were the inverse variance method based on normal distribution and the Mantel-Haenszel method. The three random-effects model methods were the DerSimonian and Laird, the Hartung-Knapp-Sidik-Jonkman and the profile likelihood approaches. Design: Meta-epidemiological study. Setting: Meta-analyses published between 2007 and 2019 in the 10 general medical journals with the highest impact factors that evaluated a medication or device for chronic medical conditions and included at least 5 randomised trials. Main outcome measures: Discordance in the judgements of imprecision of effect estimates based on two definitions: when either boundary of 95% CI of the OR changed by more than 15% or changed in relation to the null. Results: We analysed 88 meta-analyses including 1114 trials with an average of 12.60 trials per meta-analysis and average I2 of 26% (range: 0%-96%). The profile likelihood failed to converge in three meta-analyses (3%). Discordance in imprecision judgements based on the two definitions, respectively, occurred between the fixed normal distribution and fixed Mantel-Haenszel method (8% and 2%), between the DerSimonian and Laird and Hartung-Knapp-Sidik-Jonkman methods (19% and 10%), between the DerSimonian and Laird and profile likelihood methods (9% and 5%), and between the Hartung-Knapp-Sidik-Jonkman and profile likelihood methods (5% and 13%). Discordance was greater when fewer studies and greater heterogeneity was present. Conclusion: Empirical evaluation of studies of chronic medical conditions showed that conclusions about the precision of the estimates of the efficacy of a drug or device frequently changed when different pooling methods were used, particularly when the number of studies within a meta-analysis was small and statistical heterogeneity was substantial. Sensitivity analyses using more than one method may need to be considered in these two scenarios.