GRADE approach to rate the certainty from a network meta-analysis: avoiding spurious judgments of imprecision in sparse networks

GRADE Working Group, Romina Brignardello-Petersen, Mohammad H Murad, Stephen D. Walter, Shelley McLeod, Alonso Carrasco-Labra, Bram Rochwerg, Holger J. Schünemann, George Tomlinson, Gordon H. Guyatt

Research output: Contribution to journalArticle

Abstract

When direct and indirect estimates of treatment effects are coherent, network meta-analysis (NMA) estimates should have increased precision (narrower confidence or credible intervals compared with relying on direct estimates alone), a benefit of NMA. We have, however, observed cases of sparse networks in which combining direct and indirect estimates results in marked widening of the confidence intervals. In many cases, the assumption of common between-study heterogeneity across the network seems to be responsible for this counterintuitive result. Although the assumption of common between-study heterogeneity across paired comparisons may, in many cases, not be appropriate, it is required to ensure the feasibility of estimating NMA treatment effects. This is especially the case in sparse networks, in which data are insufficient to reliably estimate different variances across the network. The result, however, may be spuriously wide confidence intervals for some of the comparisons in the network (and, in the Grading of Recommendations Assessment, Development, and Evaluation approach, inappropriately low ratings of the certainty of the evidence through rating down for serious imprecision). Systematic reviewers should be aware of the problem and plan sensitivity analyses that produce intuitively sensible confidence intervals. These sensitivity analyses may include using informative priors for the between-study heterogeneity parameter in the Bayesian framework and the use of fixed effects models.

LanguageEnglish (US)
Pages60-67
Number of pages8
JournalJournal of Clinical Epidemiology
Volume105
DOIs
StatePublished - Jan 1 2019

Fingerprint

Confidence Intervals
Matched-Pair Analysis
Network Meta-Analysis

Keywords

  • certainty
  • clinical practice guidelines
  • evidence-based medicine
  • GRADE
  • imprecision
  • meta-analysis
  • Network meta-analysis
  • quality of evidence

ASJC Scopus subject areas

  • Epidemiology

Cite this

GRADE approach to rate the certainty from a network meta-analysis : avoiding spurious judgments of imprecision in sparse networks. / GRADE Working Group.

In: Journal of Clinical Epidemiology, Vol. 105, 01.01.2019, p. 60-67.

Research output: Contribution to journalArticle

@article{7c9c21b9d18e4bc9899a90478de32abd,
title = "GRADE approach to rate the certainty from a network meta-analysis: avoiding spurious judgments of imprecision in sparse networks",
abstract = "When direct and indirect estimates of treatment effects are coherent, network meta-analysis (NMA) estimates should have increased precision (narrower confidence or credible intervals compared with relying on direct estimates alone), a benefit of NMA. We have, however, observed cases of sparse networks in which combining direct and indirect estimates results in marked widening of the confidence intervals. In many cases, the assumption of common between-study heterogeneity across the network seems to be responsible for this counterintuitive result. Although the assumption of common between-study heterogeneity across paired comparisons may, in many cases, not be appropriate, it is required to ensure the feasibility of estimating NMA treatment effects. This is especially the case in sparse networks, in which data are insufficient to reliably estimate different variances across the network. The result, however, may be spuriously wide confidence intervals for some of the comparisons in the network (and, in the Grading of Recommendations Assessment, Development, and Evaluation approach, inappropriately low ratings of the certainty of the evidence through rating down for serious imprecision). Systematic reviewers should be aware of the problem and plan sensitivity analyses that produce intuitively sensible confidence intervals. These sensitivity analyses may include using informative priors for the between-study heterogeneity parameter in the Bayesian framework and the use of fixed effects models.",
keywords = "certainty, clinical practice guidelines, evidence-based medicine, GRADE, imprecision, meta-analysis, Network meta-analysis, quality of evidence",
author = "{GRADE Working Group} and Romina Brignardello-Petersen and Murad, {Mohammad H} and Walter, {Stephen D.} and Shelley McLeod and Alonso Carrasco-Labra and Bram Rochwerg and Sch{\"u}nemann, {Holger J.} and George Tomlinson and Guyatt, {Gordon H.}",
year = "2019",
month = "1",
day = "1",
doi = "10.1016/j.jclinepi.2018.08.022",
language = "English (US)",
volume = "105",
pages = "60--67",
journal = "Journal of Clinical Epidemiology",
issn = "0895-4356",
publisher = "Elsevier USA",

}

TY - JOUR

T1 - GRADE approach to rate the certainty from a network meta-analysis

T2 - Journal of Clinical Epidemiology

AU - GRADE Working Group

AU - Brignardello-Petersen, Romina

AU - Murad, Mohammad H

AU - Walter, Stephen D.

AU - McLeod, Shelley

AU - Carrasco-Labra, Alonso

AU - Rochwerg, Bram

AU - Schünemann, Holger J.

AU - Tomlinson, George

AU - Guyatt, Gordon H.

PY - 2019/1/1

Y1 - 2019/1/1

N2 - When direct and indirect estimates of treatment effects are coherent, network meta-analysis (NMA) estimates should have increased precision (narrower confidence or credible intervals compared with relying on direct estimates alone), a benefit of NMA. We have, however, observed cases of sparse networks in which combining direct and indirect estimates results in marked widening of the confidence intervals. In many cases, the assumption of common between-study heterogeneity across the network seems to be responsible for this counterintuitive result. Although the assumption of common between-study heterogeneity across paired comparisons may, in many cases, not be appropriate, it is required to ensure the feasibility of estimating NMA treatment effects. This is especially the case in sparse networks, in which data are insufficient to reliably estimate different variances across the network. The result, however, may be spuriously wide confidence intervals for some of the comparisons in the network (and, in the Grading of Recommendations Assessment, Development, and Evaluation approach, inappropriately low ratings of the certainty of the evidence through rating down for serious imprecision). Systematic reviewers should be aware of the problem and plan sensitivity analyses that produce intuitively sensible confidence intervals. These sensitivity analyses may include using informative priors for the between-study heterogeneity parameter in the Bayesian framework and the use of fixed effects models.

AB - When direct and indirect estimates of treatment effects are coherent, network meta-analysis (NMA) estimates should have increased precision (narrower confidence or credible intervals compared with relying on direct estimates alone), a benefit of NMA. We have, however, observed cases of sparse networks in which combining direct and indirect estimates results in marked widening of the confidence intervals. In many cases, the assumption of common between-study heterogeneity across the network seems to be responsible for this counterintuitive result. Although the assumption of common between-study heterogeneity across paired comparisons may, in many cases, not be appropriate, it is required to ensure the feasibility of estimating NMA treatment effects. This is especially the case in sparse networks, in which data are insufficient to reliably estimate different variances across the network. The result, however, may be spuriously wide confidence intervals for some of the comparisons in the network (and, in the Grading of Recommendations Assessment, Development, and Evaluation approach, inappropriately low ratings of the certainty of the evidence through rating down for serious imprecision). Systematic reviewers should be aware of the problem and plan sensitivity analyses that produce intuitively sensible confidence intervals. These sensitivity analyses may include using informative priors for the between-study heterogeneity parameter in the Bayesian framework and the use of fixed effects models.

KW - certainty

KW - clinical practice guidelines

KW - evidence-based medicine

KW - GRADE

KW - imprecision

KW - meta-analysis

KW - Network meta-analysis

KW - quality of evidence

UR - http://www.scopus.com/inward/record.url?scp=85054793352&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85054793352&partnerID=8YFLogxK

U2 - 10.1016/j.jclinepi.2018.08.022

DO - 10.1016/j.jclinepi.2018.08.022

M3 - Article

VL - 105

SP - 60

EP - 67

JO - Journal of Clinical Epidemiology

JF - Journal of Clinical Epidemiology

SN - 0895-4356

ER -