World Health Organization strong recommendations based on low-quality evidence (study quality) are frequent and often inconsistent with GRADE guidance

Paul E. Alexander, Juan Brito Campana, Ignacio Neumann, Michael R. Gionfriddo, Lisa Bero, Benjamin Djulbegovic, Rebecca Stoltzfus, Victor Manuel Montori, Susan L. Norris, Holger J. Schünemann, Gordon H. Guyatt

Research output: Contribution to journalArticle

31 Citations (Scopus)

Abstract

Objectives In 2007 the World Health Organization (WHO) adopted the GRADE system for development of public health guidelines. Previously we found that many strong recommendations issued by WHO are based on evidence for which there is only low or very low confidence in the estimates of effect (discordant recommendations). GRADE guidance indicates that such discordant recommendations are rarely appropriate but suggests five paradigmatic situations in which discordant recommendations may be warranted. We sought to provide insight into the many discordant recommendations in WHO guidelines. Study Design and Setting We examined all guidelines that used the GRADE method and were approved by the WHO Guideline Review Committee between 2007 and 2012. Teams of reviewers independently abstracted data from eligible guidelines and classified recommendations either into one of the five paradigms for appropriately-formulated discordant recommendations or into three additional categories in which discordant recommendations were inconsistent with GRADE guidance: 1) the evidence warranted moderate or high confidence (a misclassification of evidence) rather than low or very low confidence; 2) good practice statements; or 3) uncertainty in the estimates of effect would best lead to a conditional (weak) recommendation. Results The 33 eligible guidelines included 160 discordant recommendations, of which 98 (61.3%) addressed drug interventions and 132 (82.5%) provided some rationale (though not entirely explicit at times) for the strong recommendation. Of 160 discordant recommendations, 25 (15.6%) were judged consistent with one of the five paradigms for appropriate recommendations; 33 (21%) were based on evidence warranting moderate or high confidence in the estimates of effect; 29 (18%) were good practice statements; and 73 (46%) warranted a conditional, rather than a strong recommendation. Conclusion WHO discordant recommendations are often inconsistent with GRADE guidance, possibly threatening the integrity of the process. Further training in GRADE methods for WHO guideline development group members may be necessary, along with further research on what motivates the formulation of such recommendations.

Original languageEnglish (US)
Pages (from-to)98-106
Number of pages9
JournalJournal of Clinical Epidemiology
Volume72
DOIs
StatePublished - Apr 1 2016

Fingerprint

Guidelines
Advisory Committees
Uncertainty
Public Health
Research
Pharmaceutical Preparations

Keywords

  • Confidence in effect estimates
  • Discordant recommendations
  • GRADE
  • Low study quality
  • Public health guidelines
  • Strong recommendations
  • World Health Organization

ASJC Scopus subject areas

  • Epidemiology

Cite this

World Health Organization strong recommendations based on low-quality evidence (study quality) are frequent and often inconsistent with GRADE guidance. / Alexander, Paul E.; Brito Campana, Juan; Neumann, Ignacio; Gionfriddo, Michael R.; Bero, Lisa; Djulbegovic, Benjamin; Stoltzfus, Rebecca; Montori, Victor Manuel; Norris, Susan L.; Schünemann, Holger J.; Guyatt, Gordon H.

In: Journal of Clinical Epidemiology, Vol. 72, 01.04.2016, p. 98-106.

Research output: Contribution to journalArticle

Alexander, Paul E. ; Brito Campana, Juan ; Neumann, Ignacio ; Gionfriddo, Michael R. ; Bero, Lisa ; Djulbegovic, Benjamin ; Stoltzfus, Rebecca ; Montori, Victor Manuel ; Norris, Susan L. ; Schünemann, Holger J. ; Guyatt, Gordon H. / World Health Organization strong recommendations based on low-quality evidence (study quality) are frequent and often inconsistent with GRADE guidance. In: Journal of Clinical Epidemiology. 2016 ; Vol. 72. pp. 98-106.
@article{a729950c39ab4c04b4c5cab3f4c9e4e0,
title = "World Health Organization strong recommendations based on low-quality evidence (study quality) are frequent and often inconsistent with GRADE guidance",
abstract = "Objectives In 2007 the World Health Organization (WHO) adopted the GRADE system for development of public health guidelines. Previously we found that many strong recommendations issued by WHO are based on evidence for which there is only low or very low confidence in the estimates of effect (discordant recommendations). GRADE guidance indicates that such discordant recommendations are rarely appropriate but suggests five paradigmatic situations in which discordant recommendations may be warranted. We sought to provide insight into the many discordant recommendations in WHO guidelines. Study Design and Setting We examined all guidelines that used the GRADE method and were approved by the WHO Guideline Review Committee between 2007 and 2012. Teams of reviewers independently abstracted data from eligible guidelines and classified recommendations either into one of the five paradigms for appropriately-formulated discordant recommendations or into three additional categories in which discordant recommendations were inconsistent with GRADE guidance: 1) the evidence warranted moderate or high confidence (a misclassification of evidence) rather than low or very low confidence; 2) good practice statements; or 3) uncertainty in the estimates of effect would best lead to a conditional (weak) recommendation. Results The 33 eligible guidelines included 160 discordant recommendations, of which 98 (61.3{\%}) addressed drug interventions and 132 (82.5{\%}) provided some rationale (though not entirely explicit at times) for the strong recommendation. Of 160 discordant recommendations, 25 (15.6{\%}) were judged consistent with one of the five paradigms for appropriate recommendations; 33 (21{\%}) were based on evidence warranting moderate or high confidence in the estimates of effect; 29 (18{\%}) were good practice statements; and 73 (46{\%}) warranted a conditional, rather than a strong recommendation. Conclusion WHO discordant recommendations are often inconsistent with GRADE guidance, possibly threatening the integrity of the process. Further training in GRADE methods for WHO guideline development group members may be necessary, along with further research on what motivates the formulation of such recommendations.",
keywords = "Confidence in effect estimates, Discordant recommendations, GRADE, Low study quality, Public health guidelines, Strong recommendations, World Health Organization",
author = "Alexander, {Paul E.} and {Brito Campana}, Juan and Ignacio Neumann and Gionfriddo, {Michael R.} and Lisa Bero and Benjamin Djulbegovic and Rebecca Stoltzfus and Montori, {Victor Manuel} and Norris, {Susan L.} and Sch{\"u}nemann, {Holger J.} and Guyatt, {Gordon H.}",
year = "2016",
month = "4",
day = "1",
doi = "10.1016/j.jclinepi.2014.10.011",
language = "English (US)",
volume = "72",
pages = "98--106",
journal = "Journal of Clinical Epidemiology",
issn = "0895-4356",
publisher = "Elsevier USA",

}

TY - JOUR

T1 - World Health Organization strong recommendations based on low-quality evidence (study quality) are frequent and often inconsistent with GRADE guidance

AU - Alexander, Paul E.

AU - Brito Campana, Juan

AU - Neumann, Ignacio

AU - Gionfriddo, Michael R.

AU - Bero, Lisa

AU - Djulbegovic, Benjamin

AU - Stoltzfus, Rebecca

AU - Montori, Victor Manuel

AU - Norris, Susan L.

AU - Schünemann, Holger J.

AU - Guyatt, Gordon H.

PY - 2016/4/1

Y1 - 2016/4/1

N2 - Objectives In 2007 the World Health Organization (WHO) adopted the GRADE system for development of public health guidelines. Previously we found that many strong recommendations issued by WHO are based on evidence for which there is only low or very low confidence in the estimates of effect (discordant recommendations). GRADE guidance indicates that such discordant recommendations are rarely appropriate but suggests five paradigmatic situations in which discordant recommendations may be warranted. We sought to provide insight into the many discordant recommendations in WHO guidelines. Study Design and Setting We examined all guidelines that used the GRADE method and were approved by the WHO Guideline Review Committee between 2007 and 2012. Teams of reviewers independently abstracted data from eligible guidelines and classified recommendations either into one of the five paradigms for appropriately-formulated discordant recommendations or into three additional categories in which discordant recommendations were inconsistent with GRADE guidance: 1) the evidence warranted moderate or high confidence (a misclassification of evidence) rather than low or very low confidence; 2) good practice statements; or 3) uncertainty in the estimates of effect would best lead to a conditional (weak) recommendation. Results The 33 eligible guidelines included 160 discordant recommendations, of which 98 (61.3%) addressed drug interventions and 132 (82.5%) provided some rationale (though not entirely explicit at times) for the strong recommendation. Of 160 discordant recommendations, 25 (15.6%) were judged consistent with one of the five paradigms for appropriate recommendations; 33 (21%) were based on evidence warranting moderate or high confidence in the estimates of effect; 29 (18%) were good practice statements; and 73 (46%) warranted a conditional, rather than a strong recommendation. Conclusion WHO discordant recommendations are often inconsistent with GRADE guidance, possibly threatening the integrity of the process. Further training in GRADE methods for WHO guideline development group members may be necessary, along with further research on what motivates the formulation of such recommendations.

AB - Objectives In 2007 the World Health Organization (WHO) adopted the GRADE system for development of public health guidelines. Previously we found that many strong recommendations issued by WHO are based on evidence for which there is only low or very low confidence in the estimates of effect (discordant recommendations). GRADE guidance indicates that such discordant recommendations are rarely appropriate but suggests five paradigmatic situations in which discordant recommendations may be warranted. We sought to provide insight into the many discordant recommendations in WHO guidelines. Study Design and Setting We examined all guidelines that used the GRADE method and were approved by the WHO Guideline Review Committee between 2007 and 2012. Teams of reviewers independently abstracted data from eligible guidelines and classified recommendations either into one of the five paradigms for appropriately-formulated discordant recommendations or into three additional categories in which discordant recommendations were inconsistent with GRADE guidance: 1) the evidence warranted moderate or high confidence (a misclassification of evidence) rather than low or very low confidence; 2) good practice statements; or 3) uncertainty in the estimates of effect would best lead to a conditional (weak) recommendation. Results The 33 eligible guidelines included 160 discordant recommendations, of which 98 (61.3%) addressed drug interventions and 132 (82.5%) provided some rationale (though not entirely explicit at times) for the strong recommendation. Of 160 discordant recommendations, 25 (15.6%) were judged consistent with one of the five paradigms for appropriate recommendations; 33 (21%) were based on evidence warranting moderate or high confidence in the estimates of effect; 29 (18%) were good practice statements; and 73 (46%) warranted a conditional, rather than a strong recommendation. Conclusion WHO discordant recommendations are often inconsistent with GRADE guidance, possibly threatening the integrity of the process. Further training in GRADE methods for WHO guideline development group members may be necessary, along with further research on what motivates the formulation of such recommendations.

KW - Confidence in effect estimates

KW - Discordant recommendations

KW - GRADE

KW - Low study quality

KW - Public health guidelines

KW - Strong recommendations

KW - World Health Organization

UR - http://www.scopus.com/inward/record.url?scp=84961058099&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84961058099&partnerID=8YFLogxK

U2 - 10.1016/j.jclinepi.2014.10.011

DO - 10.1016/j.jclinepi.2014.10.011

M3 - Article

C2 - 25618534

AN - SCOPUS:84961058099

VL - 72

SP - 98

EP - 106

JO - Journal of Clinical Epidemiology

JF - Journal of Clinical Epidemiology

SN - 0895-4356

ER -