An efficient strategy allowed English-speaking reviewers to identify foreign-language articles eligible for a systematic review

Jason W. Busse, Paul Bruno, Keshena Malik, Gaelan Connell, David Torrance, Trung Ngo, Karin Kirmayr, Daniel Avrahami, John J. Riva, Shanil Ebrahim, Peter A A Struijs, David Brunarski, Stephen J. Burnie, Frances Leblanc, Eric A. Coomes, Ivan A. Steenstra, Tesha Slack, Robert Rodine, Janey Jim, Victor Manuel Montori & 1 others Gordon H. Guyatt

Research output: Contribution to journalArticle

11 Citations (Scopus)

Abstract

Objective To assess English-speaking reviewers' accuracy in determining the eligibility of foreign-language articles for a systematic review. Study Design and Settings Systematic review of randomized controlled trials of therapy for fibromyalgia. Guided by 10 questions, English-speaking reviewers screened non-English-language articles for eligibility. Teams of two native-language speakers provided reference standard judgments of eligibility. Results Of 15,466 potentially eligible articles, we retrieved 763 in full text, of which 133 were published in 19 non-English languages; 53 trials published in 11 languages other than English proved eligible. Of the 53 eligible articles, English-language reviewers guided by the 10 questions mistakenly judged 6 as ineligible; of the 80 ineligible articles, 8 were incorrectly judged eligible by English-language reviewers (sensitivity = 0.89; specificity = 0.90). Use of a simple three-step rule (excluding languages with less than three articles, reviewing titles and abstracts for clear indications of eligibility, and noting the lack of a clearly reported statistical analysis unless the word "random" appears) led to accurate classification of 51 of 53 articles (sensitivity = 0.96; specificity = 0.70). Conclusion Our findings show promise for limiting the need for non-English-language review teams in systematic reviews with large numbers of potentially eligible non-English-language articles.

Original languageEnglish (US)
Pages (from-to)547-553
Number of pages7
JournalJournal of Clinical Epidemiology
Volume67
Issue number5
DOIs
StatePublished - 2014

Fingerprint

Language
Sensitivity and Specificity
Fibromyalgia
Randomized Controlled Trials

Keywords

  • Bias
  • English-language bias
  • Fibromyalgia
  • Language of publication
  • Methodology
  • Systematic review

ASJC Scopus subject areas

  • Epidemiology

Cite this

An efficient strategy allowed English-speaking reviewers to identify foreign-language articles eligible for a systematic review. / Busse, Jason W.; Bruno, Paul; Malik, Keshena; Connell, Gaelan; Torrance, David; Ngo, Trung; Kirmayr, Karin; Avrahami, Daniel; Riva, John J.; Ebrahim, Shanil; Struijs, Peter A A; Brunarski, David; Burnie, Stephen J.; Leblanc, Frances; Coomes, Eric A.; Steenstra, Ivan A.; Slack, Tesha; Rodine, Robert; Jim, Janey; Montori, Victor Manuel; Guyatt, Gordon H.

In: Journal of Clinical Epidemiology, Vol. 67, No. 5, 2014, p. 547-553.

Research output: Contribution to journalArticle

Busse, JW, Bruno, P, Malik, K, Connell, G, Torrance, D, Ngo, T, Kirmayr, K, Avrahami, D, Riva, JJ, Ebrahim, S, Struijs, PAA, Brunarski, D, Burnie, SJ, Leblanc, F, Coomes, EA, Steenstra, IA, Slack, T, Rodine, R, Jim, J, Montori, VM & Guyatt, GH 2014, 'An efficient strategy allowed English-speaking reviewers to identify foreign-language articles eligible for a systematic review', Journal of Clinical Epidemiology, vol. 67, no. 5, pp. 547-553. https://doi.org/10.1016/j.jclinepi.2013.07.022
Busse, Jason W. ; Bruno, Paul ; Malik, Keshena ; Connell, Gaelan ; Torrance, David ; Ngo, Trung ; Kirmayr, Karin ; Avrahami, Daniel ; Riva, John J. ; Ebrahim, Shanil ; Struijs, Peter A A ; Brunarski, David ; Burnie, Stephen J. ; Leblanc, Frances ; Coomes, Eric A. ; Steenstra, Ivan A. ; Slack, Tesha ; Rodine, Robert ; Jim, Janey ; Montori, Victor Manuel ; Guyatt, Gordon H. / An efficient strategy allowed English-speaking reviewers to identify foreign-language articles eligible for a systematic review. In: Journal of Clinical Epidemiology. 2014 ; Vol. 67, No. 5. pp. 547-553.
@article{8f923c42bddb4f9996004ec7f01bb9ca,
title = "An efficient strategy allowed English-speaking reviewers to identify foreign-language articles eligible for a systematic review",
abstract = "Objective To assess English-speaking reviewers' accuracy in determining the eligibility of foreign-language articles for a systematic review. Study Design and Settings Systematic review of randomized controlled trials of therapy for fibromyalgia. Guided by 10 questions, English-speaking reviewers screened non-English-language articles for eligibility. Teams of two native-language speakers provided reference standard judgments of eligibility. Results Of 15,466 potentially eligible articles, we retrieved 763 in full text, of which 133 were published in 19 non-English languages; 53 trials published in 11 languages other than English proved eligible. Of the 53 eligible articles, English-language reviewers guided by the 10 questions mistakenly judged 6 as ineligible; of the 80 ineligible articles, 8 were incorrectly judged eligible by English-language reviewers (sensitivity = 0.89; specificity = 0.90). Use of a simple three-step rule (excluding languages with less than three articles, reviewing titles and abstracts for clear indications of eligibility, and noting the lack of a clearly reported statistical analysis unless the word {"}random{"} appears) led to accurate classification of 51 of 53 articles (sensitivity = 0.96; specificity = 0.70). Conclusion Our findings show promise for limiting the need for non-English-language review teams in systematic reviews with large numbers of potentially eligible non-English-language articles.",
keywords = "Bias, English-language bias, Fibromyalgia, Language of publication, Methodology, Systematic review",
author = "Busse, {Jason W.} and Paul Bruno and Keshena Malik and Gaelan Connell and David Torrance and Trung Ngo and Karin Kirmayr and Daniel Avrahami and Riva, {John J.} and Shanil Ebrahim and Struijs, {Peter A A} and David Brunarski and Burnie, {Stephen J.} and Frances Leblanc and Coomes, {Eric A.} and Steenstra, {Ivan A.} and Tesha Slack and Robert Rodine and Janey Jim and Montori, {Victor Manuel} and Guyatt, {Gordon H.}",
year = "2014",
doi = "10.1016/j.jclinepi.2013.07.022",
language = "English (US)",
volume = "67",
pages = "547--553",
journal = "Journal of Clinical Epidemiology",
issn = "0895-4356",
publisher = "Elsevier USA",
number = "5",

}

TY - JOUR

T1 - An efficient strategy allowed English-speaking reviewers to identify foreign-language articles eligible for a systematic review

AU - Busse, Jason W.

AU - Bruno, Paul

AU - Malik, Keshena

AU - Connell, Gaelan

AU - Torrance, David

AU - Ngo, Trung

AU - Kirmayr, Karin

AU - Avrahami, Daniel

AU - Riva, John J.

AU - Ebrahim, Shanil

AU - Struijs, Peter A A

AU - Brunarski, David

AU - Burnie, Stephen J.

AU - Leblanc, Frances

AU - Coomes, Eric A.

AU - Steenstra, Ivan A.

AU - Slack, Tesha

AU - Rodine, Robert

AU - Jim, Janey

AU - Montori, Victor Manuel

AU - Guyatt, Gordon H.

PY - 2014

Y1 - 2014

N2 - Objective To assess English-speaking reviewers' accuracy in determining the eligibility of foreign-language articles for a systematic review. Study Design and Settings Systematic review of randomized controlled trials of therapy for fibromyalgia. Guided by 10 questions, English-speaking reviewers screened non-English-language articles for eligibility. Teams of two native-language speakers provided reference standard judgments of eligibility. Results Of 15,466 potentially eligible articles, we retrieved 763 in full text, of which 133 were published in 19 non-English languages; 53 trials published in 11 languages other than English proved eligible. Of the 53 eligible articles, English-language reviewers guided by the 10 questions mistakenly judged 6 as ineligible; of the 80 ineligible articles, 8 were incorrectly judged eligible by English-language reviewers (sensitivity = 0.89; specificity = 0.90). Use of a simple three-step rule (excluding languages with less than three articles, reviewing titles and abstracts for clear indications of eligibility, and noting the lack of a clearly reported statistical analysis unless the word "random" appears) led to accurate classification of 51 of 53 articles (sensitivity = 0.96; specificity = 0.70). Conclusion Our findings show promise for limiting the need for non-English-language review teams in systematic reviews with large numbers of potentially eligible non-English-language articles.

AB - Objective To assess English-speaking reviewers' accuracy in determining the eligibility of foreign-language articles for a systematic review. Study Design and Settings Systematic review of randomized controlled trials of therapy for fibromyalgia. Guided by 10 questions, English-speaking reviewers screened non-English-language articles for eligibility. Teams of two native-language speakers provided reference standard judgments of eligibility. Results Of 15,466 potentially eligible articles, we retrieved 763 in full text, of which 133 were published in 19 non-English languages; 53 trials published in 11 languages other than English proved eligible. Of the 53 eligible articles, English-language reviewers guided by the 10 questions mistakenly judged 6 as ineligible; of the 80 ineligible articles, 8 were incorrectly judged eligible by English-language reviewers (sensitivity = 0.89; specificity = 0.90). Use of a simple three-step rule (excluding languages with less than three articles, reviewing titles and abstracts for clear indications of eligibility, and noting the lack of a clearly reported statistical analysis unless the word "random" appears) led to accurate classification of 51 of 53 articles (sensitivity = 0.96; specificity = 0.70). Conclusion Our findings show promise for limiting the need for non-English-language review teams in systematic reviews with large numbers of potentially eligible non-English-language articles.

KW - Bias

KW - English-language bias

KW - Fibromyalgia

KW - Language of publication

KW - Methodology

KW - Systematic review

UR - http://www.scopus.com/inward/record.url?scp=84897514797&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84897514797&partnerID=8YFLogxK

U2 - 10.1016/j.jclinepi.2013.07.022

DO - 10.1016/j.jclinepi.2013.07.022

M3 - Article

VL - 67

SP - 547

EP - 553

JO - Journal of Clinical Epidemiology

JF - Journal of Clinical Epidemiology

SN - 0895-4356

IS - 5

ER -