TY - JOUR
T1 - The GRADE approach is reproducible in assessing the quality of evidence of quantitative evidence syntheses
AU - Mustafa, Reem A.
AU - Santesso, Nancy
AU - Brozek, Jan
AU - Akl, Elie A.
AU - Walter, Stephen D.
AU - Norman, Geoff
AU - Kulasegaram, Mahan
AU - Christensen, Robin
AU - Guyatt, Gordon H.
AU - Falck-Ytter, Yngve
AU - Chang, Stephanie
AU - Murad, Mohammad Hassan
AU - Vist, Gunn E.
AU - Lasserson, Toby
AU - Gartlehner, Gerald
AU - Shukla, Vijay
AU - Sun, Xin
AU - Whittington, Craig
AU - Post, Piet N.
AU - Lang, Eddy
AU - Thaler, Kylie
AU - Kunnamo, Ilkka
AU - Alenius, Heidi
AU - Meerpohl, Joerg J.
AU - Alba, Ana C.
AU - Nevis, Immaculate F.
AU - Gentles, Stephen
AU - Ethier, Marie Chantal
AU - Carrasco-Labra, Alonso
AU - Khatib, Rasha
AU - Nesrallah, Gihad
AU - Kroft, Jamie
AU - Selk, Amanda
AU - Brignardello-Petersen, Romina
AU - Schünemann, Holger J.
PY - 2013/7
Y1 - 2013/7
N2 - Objective: We evaluated the inter-rater reliability (IRR) of assessing the quality of evidence (QoE) using the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach. Study Design and Setting: On completing two training exercises, participants worked independently as individual raters to assess the QoE of 16 outcomes. After recording their initial impression using a global rating, raters graded the QoE following the GRADE approach. Subsequently, randomly paired raters submitted a consensus rating. Results: The IRR without using the GRADE approach for two individual raters was 0.31 (95% confidence interval [95% CI] = 0.21-0.42) among Health Research Methodology students (n = 10) and 0.27 (95% CI = 0.19-0.37) among the GRADE working group members (n = 15). The corresponding IRR of the GRADE approach in assessing the QoE was significantly higher, that is, 0.66 (95% CI = 0.56-0.75) and 0.72 (95% CI = 0.61-0.79), respectively. The IRR further increased for three (0.80 [95% CI = 0.73-0.86] and 0.74 [95% CI = 0.65-0.81]) or four raters (0.84 [95% CI = 0.78-0.89] and 0.79 [95% CI = 0.71-0.85]). The IRR did not improve when QoE was assessed through a consensus rating. Conclusion: Our findings suggest that trained individuals using the GRADE approach improves reliability in comparison to intuitive judgments about the QoE and that two individual raters can reliably assess the QoE using the GRADE system.
AB - Objective: We evaluated the inter-rater reliability (IRR) of assessing the quality of evidence (QoE) using the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach. Study Design and Setting: On completing two training exercises, participants worked independently as individual raters to assess the QoE of 16 outcomes. After recording their initial impression using a global rating, raters graded the QoE following the GRADE approach. Subsequently, randomly paired raters submitted a consensus rating. Results: The IRR without using the GRADE approach for two individual raters was 0.31 (95% confidence interval [95% CI] = 0.21-0.42) among Health Research Methodology students (n = 10) and 0.27 (95% CI = 0.19-0.37) among the GRADE working group members (n = 15). The corresponding IRR of the GRADE approach in assessing the QoE was significantly higher, that is, 0.66 (95% CI = 0.56-0.75) and 0.72 (95% CI = 0.61-0.79), respectively. The IRR further increased for three (0.80 [95% CI = 0.73-0.86] and 0.74 [95% CI = 0.65-0.81]) or four raters (0.84 [95% CI = 0.78-0.89] and 0.79 [95% CI = 0.71-0.85]). The IRR did not improve when QoE was assessed through a consensus rating. Conclusion: Our findings suggest that trained individuals using the GRADE approach improves reliability in comparison to intuitive judgments about the QoE and that two individual raters can reliably assess the QoE using the GRADE system.
KW - Evidence-based medicine
KW - GRADE
KW - Inter-rater reliability
KW - Levels of evidence
KW - Reproducibility
KW - Validation studies
UR - http://www.scopus.com/inward/record.url?scp=84878261195&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84878261195&partnerID=8YFLogxK
U2 - 10.1016/j.jclinepi.2013.02.004
DO - 10.1016/j.jclinepi.2013.02.004
M3 - Article
C2 - 23623694
AN - SCOPUS:84878261195
SN - 0895-4356
VL - 66
SP - 736-742.e5
JO - Journal of Clinical Epidemiology
JF - Journal of Clinical Epidemiology
IS - 7
ER -