Trends in P Value, Confidence Interval, and Power Analysis Reporting in Health Professions Education Research Reports: A Systematic Appraisal

Eduardo F. Abbott, Valentina P. Serrano, Melissa L. Rethlefsen, T. K. Pandian, Nimesh D. Naik, Colin Patrick West, Shane S. Pankratz, David Allan Cook

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

PURPOSE: To characterize reporting of P values, confidence intervals (CIs), and statistical power in health professions education research (HPER) through manual and computerized analysis of published research reports. METHOD: The authors searched PubMed, Embase, and CINAHL on May 7, 2016, for comparative research studies. For manual analysis of abstracts and main texts, they randomly sampled 250 HPER reports published in 1985, 1995, 2005, and 2015, and 100 biomedical research reports published in 1985 and 2015. Automated computerized analysis of abstracts included all HPER reports published 1970–2015. RESULTS: In the 2015 HPER sample, P values were reported in 69/100 abstracts and 94 main texts. CIs were reported in 6 abstracts and 22 main texts. Most P values (≥ 77%) were ≤ .05. Across all years, 60/164 two-group HPER studies had ≥ 80% power to detect a between-group difference of 0.5 standard deviations. From 1985 to 2015, the proportion of HPER abstracts reporting a CI did not change significantly (odds ratio [OR] 2.87; 95% CI 1.04, 7.88) whereas that of main texts reporting a CI increased (OR 1.96; 95% CI 1.39, 2.78). Comparison with biomedical studies revealed similar reporting of P values, but more frequent use of CIs in biomedicine. Automated analysis of 56,440 HPER abstracts found 14,867 (26.3%) reporting a P value, 3,024 (5.4%) reporting a CI, and increased reporting of P values and CIs from 1970 to 2015. CONCLUSIONS: P values are ubiquitous in HPER, CIs are rarely reported, and most studies are underpowered. Most reported P values would be considered statistically significant.

Original languageEnglish (US)
JournalAcademic Medicine
DOIs
StateAccepted/In press - Jun 20 2017

Fingerprint

Health Occupations
Health Education
confidence
profession
Confidence Intervals
trend
health
Values
education
Research
Odds Ratio
biomedicine
comparative research
PubMed
Biomedical Research
Group

ASJC Scopus subject areas

  • Medicine(all)
  • Education

Cite this

Trends in P Value, Confidence Interval, and Power Analysis Reporting in Health Professions Education Research Reports : A Systematic Appraisal. / Abbott, Eduardo F.; Serrano, Valentina P.; Rethlefsen, Melissa L.; Pandian, T. K.; Naik, Nimesh D.; West, Colin Patrick; Pankratz, Shane S.; Cook, David Allan.

In: Academic Medicine, 20.06.2017.

Research output: Contribution to journalArticle

Abbott, Eduardo F. ; Serrano, Valentina P. ; Rethlefsen, Melissa L. ; Pandian, T. K. ; Naik, Nimesh D. ; West, Colin Patrick ; Pankratz, Shane S. ; Cook, David Allan. / Trends in P Value, Confidence Interval, and Power Analysis Reporting in Health Professions Education Research Reports : A Systematic Appraisal. In: Academic Medicine. 2017.
@article{f98b5e66f6dd4235847f18e1136820b3,
title = "Trends in P Value, Confidence Interval, and Power Analysis Reporting in Health Professions Education Research Reports: A Systematic Appraisal",
abstract = "PURPOSE: To characterize reporting of P values, confidence intervals (CIs), and statistical power in health professions education research (HPER) through manual and computerized analysis of published research reports. METHOD: The authors searched PubMed, Embase, and CINAHL on May 7, 2016, for comparative research studies. For manual analysis of abstracts and main texts, they randomly sampled 250 HPER reports published in 1985, 1995, 2005, and 2015, and 100 biomedical research reports published in 1985 and 2015. Automated computerized analysis of abstracts included all HPER reports published 1970–2015. RESULTS: In the 2015 HPER sample, P values were reported in 69/100 abstracts and 94 main texts. CIs were reported in 6 abstracts and 22 main texts. Most P values (≥ 77{\%}) were ≤ .05. Across all years, 60/164 two-group HPER studies had ≥ 80{\%} power to detect a between-group difference of 0.5 standard deviations. From 1985 to 2015, the proportion of HPER abstracts reporting a CI did not change significantly (odds ratio [OR] 2.87; 95{\%} CI 1.04, 7.88) whereas that of main texts reporting a CI increased (OR 1.96; 95{\%} CI 1.39, 2.78). Comparison with biomedical studies revealed similar reporting of P values, but more frequent use of CIs in biomedicine. Automated analysis of 56,440 HPER abstracts found 14,867 (26.3{\%}) reporting a P value, 3,024 (5.4{\%}) reporting a CI, and increased reporting of P values and CIs from 1970 to 2015. CONCLUSIONS: P values are ubiquitous in HPER, CIs are rarely reported, and most studies are underpowered. Most reported P values would be considered statistically significant.",
author = "Abbott, {Eduardo F.} and Serrano, {Valentina P.} and Rethlefsen, {Melissa L.} and Pandian, {T. K.} and Naik, {Nimesh D.} and West, {Colin Patrick} and Pankratz, {Shane S.} and Cook, {David Allan}",
year = "2017",
month = "6",
day = "20",
doi = "10.1097/ACM.0000000000001773",
language = "English (US)",
journal = "Academic Medicine",
issn = "1040-2446",
publisher = "Lippincott Williams and Wilkins",

}

TY - JOUR

T1 - Trends in P Value, Confidence Interval, and Power Analysis Reporting in Health Professions Education Research Reports

T2 - A Systematic Appraisal

AU - Abbott, Eduardo F.

AU - Serrano, Valentina P.

AU - Rethlefsen, Melissa L.

AU - Pandian, T. K.

AU - Naik, Nimesh D.

AU - West, Colin Patrick

AU - Pankratz, Shane S.

AU - Cook, David Allan

PY - 2017/6/20

Y1 - 2017/6/20

N2 - PURPOSE: To characterize reporting of P values, confidence intervals (CIs), and statistical power in health professions education research (HPER) through manual and computerized analysis of published research reports. METHOD: The authors searched PubMed, Embase, and CINAHL on May 7, 2016, for comparative research studies. For manual analysis of abstracts and main texts, they randomly sampled 250 HPER reports published in 1985, 1995, 2005, and 2015, and 100 biomedical research reports published in 1985 and 2015. Automated computerized analysis of abstracts included all HPER reports published 1970–2015. RESULTS: In the 2015 HPER sample, P values were reported in 69/100 abstracts and 94 main texts. CIs were reported in 6 abstracts and 22 main texts. Most P values (≥ 77%) were ≤ .05. Across all years, 60/164 two-group HPER studies had ≥ 80% power to detect a between-group difference of 0.5 standard deviations. From 1985 to 2015, the proportion of HPER abstracts reporting a CI did not change significantly (odds ratio [OR] 2.87; 95% CI 1.04, 7.88) whereas that of main texts reporting a CI increased (OR 1.96; 95% CI 1.39, 2.78). Comparison with biomedical studies revealed similar reporting of P values, but more frequent use of CIs in biomedicine. Automated analysis of 56,440 HPER abstracts found 14,867 (26.3%) reporting a P value, 3,024 (5.4%) reporting a CI, and increased reporting of P values and CIs from 1970 to 2015. CONCLUSIONS: P values are ubiquitous in HPER, CIs are rarely reported, and most studies are underpowered. Most reported P values would be considered statistically significant.

AB - PURPOSE: To characterize reporting of P values, confidence intervals (CIs), and statistical power in health professions education research (HPER) through manual and computerized analysis of published research reports. METHOD: The authors searched PubMed, Embase, and CINAHL on May 7, 2016, for comparative research studies. For manual analysis of abstracts and main texts, they randomly sampled 250 HPER reports published in 1985, 1995, 2005, and 2015, and 100 biomedical research reports published in 1985 and 2015. Automated computerized analysis of abstracts included all HPER reports published 1970–2015. RESULTS: In the 2015 HPER sample, P values were reported in 69/100 abstracts and 94 main texts. CIs were reported in 6 abstracts and 22 main texts. Most P values (≥ 77%) were ≤ .05. Across all years, 60/164 two-group HPER studies had ≥ 80% power to detect a between-group difference of 0.5 standard deviations. From 1985 to 2015, the proportion of HPER abstracts reporting a CI did not change significantly (odds ratio [OR] 2.87; 95% CI 1.04, 7.88) whereas that of main texts reporting a CI increased (OR 1.96; 95% CI 1.39, 2.78). Comparison with biomedical studies revealed similar reporting of P values, but more frequent use of CIs in biomedicine. Automated analysis of 56,440 HPER abstracts found 14,867 (26.3%) reporting a P value, 3,024 (5.4%) reporting a CI, and increased reporting of P values and CIs from 1970 to 2015. CONCLUSIONS: P values are ubiquitous in HPER, CIs are rarely reported, and most studies are underpowered. Most reported P values would be considered statistically significant.

UR - http://www.scopus.com/inward/record.url?scp=85021161822&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85021161822&partnerID=8YFLogxK

U2 - 10.1097/ACM.0000000000001773

DO - 10.1097/ACM.0000000000001773

M3 - Article

C2 - 28640032

AN - SCOPUS:85021161822

JO - Academic Medicine

JF - Academic Medicine

SN - 1040-2446

ER -