Milestone-compatible neurology resident assessments: A role for observable practice activities

Lyell Jones, Elliot L. Dimberg, Christopher J. Boes, Scott Daniel Eggers, David William Dodick, Jeremy K. Cutsforth-Gregory, Andrea N Leep Hunderfund, David J. Capobianco

Research output: Contribution to journalArticle

8 Citations (Scopus)

Abstract

Objective: Beginning in 2014, US neurology residency programs were required to report each trainee's educational progression within 29 neurology Milestone competency domains. Trainee assessment systems will need to be adapted to inform these requirements. The primary aims of this study were to validate neurology resident assessment content using observable practice activities (OPAs) and to develop assessment formats easily translated to the Neurology Milestones. Methods: A modified Delphi technique was used to establish consensus perceptions of importance of 73 neurology OPAs among neurology educators and trainees at 3 neurology residency programs. A content validity score (CVS) was derived for each neurology OPA, with scores ≥4.0 determined in advance to indicate sufficient content validity. Results: The mean CVS for all OPAs was 4.4 (range 3.5-5.0). Fifty-seven (78%) OPAs had a CVS ≥4.0 leaving 16 (22%) below the pre-established threshold for content validity. Trainees assigned a higher importance to individual OPAs (mean CVS 4.6) compared to faculty (mean 4.4, p = 0.016), but the effect size was small (η<inf>2</inf> = 0.10). There was no demonstrated effect of length of education experience on perceived importance of neurology OPAs (p = 0.23). Two sample resident assessment formats were developed, one using neurology OPAs alone and another using a combination of neurology OPAs and the Neurology Milestones. Conclusions: This study provides neurology training programs with content validity evidence for items to include in resident assessments, and sample assessment formats that directly translate to the Neurology Milestones. Length of education experience has little effect on perceptions of neurology OPA importance.

Original languageEnglish (US)
Pages (from-to)2279-2283
Number of pages5
JournalNeurology
Volume84
Issue number22
DOIs
StatePublished - Jun 2 2015

Fingerprint

Neurology
Internship and Residency
Education
Delphi Technique

ASJC Scopus subject areas

  • Clinical Neurology

Cite this

Milestone-compatible neurology resident assessments : A role for observable practice activities. / Jones, Lyell; Dimberg, Elliot L.; Boes, Christopher J.; Eggers, Scott Daniel; Dodick, David William; Cutsforth-Gregory, Jeremy K.; Hunderfund, Andrea N Leep; Capobianco, David J.

In: Neurology, Vol. 84, No. 22, 02.06.2015, p. 2279-2283.

Research output: Contribution to journalArticle

Jones, L, Dimberg, EL, Boes, CJ, Eggers, SD, Dodick, DW, Cutsforth-Gregory, JK, Hunderfund, ANL & Capobianco, DJ 2015, 'Milestone-compatible neurology resident assessments: A role for observable practice activities', Neurology, vol. 84, no. 22, pp. 2279-2283. https://doi.org/10.1212/WNL.0000000000001641
Jones, Lyell ; Dimberg, Elliot L. ; Boes, Christopher J. ; Eggers, Scott Daniel ; Dodick, David William ; Cutsforth-Gregory, Jeremy K. ; Hunderfund, Andrea N Leep ; Capobianco, David J. / Milestone-compatible neurology resident assessments : A role for observable practice activities. In: Neurology. 2015 ; Vol. 84, No. 22. pp. 2279-2283.
@article{1fa4e4a8fbdd4dda89e4296fc5b1b7ce,
title = "Milestone-compatible neurology resident assessments: A role for observable practice activities",
abstract = "Objective: Beginning in 2014, US neurology residency programs were required to report each trainee's educational progression within 29 neurology Milestone competency domains. Trainee assessment systems will need to be adapted to inform these requirements. The primary aims of this study were to validate neurology resident assessment content using observable practice activities (OPAs) and to develop assessment formats easily translated to the Neurology Milestones. Methods: A modified Delphi technique was used to establish consensus perceptions of importance of 73 neurology OPAs among neurology educators and trainees at 3 neurology residency programs. A content validity score (CVS) was derived for each neurology OPA, with scores ≥4.0 determined in advance to indicate sufficient content validity. Results: The mean CVS for all OPAs was 4.4 (range 3.5-5.0). Fifty-seven (78{\%}) OPAs had a CVS ≥4.0 leaving 16 (22{\%}) below the pre-established threshold for content validity. Trainees assigned a higher importance to individual OPAs (mean CVS 4.6) compared to faculty (mean 4.4, p = 0.016), but the effect size was small (η2 = 0.10). There was no demonstrated effect of length of education experience on perceived importance of neurology OPAs (p = 0.23). Two sample resident assessment formats were developed, one using neurology OPAs alone and another using a combination of neurology OPAs and the Neurology Milestones. Conclusions: This study provides neurology training programs with content validity evidence for items to include in resident assessments, and sample assessment formats that directly translate to the Neurology Milestones. Length of education experience has little effect on perceptions of neurology OPA importance.",
author = "Lyell Jones and Dimberg, {Elliot L.} and Boes, {Christopher J.} and Eggers, {Scott Daniel} and Dodick, {David William} and Cutsforth-Gregory, {Jeremy K.} and Hunderfund, {Andrea N Leep} and Capobianco, {David J.}",
year = "2015",
month = "6",
day = "2",
doi = "10.1212/WNL.0000000000001641",
language = "English (US)",
volume = "84",
pages = "2279--2283",
journal = "Neurology",
issn = "0028-3878",
publisher = "Lippincott Williams and Wilkins",
number = "22",

}

TY - JOUR

T1 - Milestone-compatible neurology resident assessments

T2 - A role for observable practice activities

AU - Jones, Lyell

AU - Dimberg, Elliot L.

AU - Boes, Christopher J.

AU - Eggers, Scott Daniel

AU - Dodick, David William

AU - Cutsforth-Gregory, Jeremy K.

AU - Hunderfund, Andrea N Leep

AU - Capobianco, David J.

PY - 2015/6/2

Y1 - 2015/6/2

N2 - Objective: Beginning in 2014, US neurology residency programs were required to report each trainee's educational progression within 29 neurology Milestone competency domains. Trainee assessment systems will need to be adapted to inform these requirements. The primary aims of this study were to validate neurology resident assessment content using observable practice activities (OPAs) and to develop assessment formats easily translated to the Neurology Milestones. Methods: A modified Delphi technique was used to establish consensus perceptions of importance of 73 neurology OPAs among neurology educators and trainees at 3 neurology residency programs. A content validity score (CVS) was derived for each neurology OPA, with scores ≥4.0 determined in advance to indicate sufficient content validity. Results: The mean CVS for all OPAs was 4.4 (range 3.5-5.0). Fifty-seven (78%) OPAs had a CVS ≥4.0 leaving 16 (22%) below the pre-established threshold for content validity. Trainees assigned a higher importance to individual OPAs (mean CVS 4.6) compared to faculty (mean 4.4, p = 0.016), but the effect size was small (η2 = 0.10). There was no demonstrated effect of length of education experience on perceived importance of neurology OPAs (p = 0.23). Two sample resident assessment formats were developed, one using neurology OPAs alone and another using a combination of neurology OPAs and the Neurology Milestones. Conclusions: This study provides neurology training programs with content validity evidence for items to include in resident assessments, and sample assessment formats that directly translate to the Neurology Milestones. Length of education experience has little effect on perceptions of neurology OPA importance.

AB - Objective: Beginning in 2014, US neurology residency programs were required to report each trainee's educational progression within 29 neurology Milestone competency domains. Trainee assessment systems will need to be adapted to inform these requirements. The primary aims of this study were to validate neurology resident assessment content using observable practice activities (OPAs) and to develop assessment formats easily translated to the Neurology Milestones. Methods: A modified Delphi technique was used to establish consensus perceptions of importance of 73 neurology OPAs among neurology educators and trainees at 3 neurology residency programs. A content validity score (CVS) was derived for each neurology OPA, with scores ≥4.0 determined in advance to indicate sufficient content validity. Results: The mean CVS for all OPAs was 4.4 (range 3.5-5.0). Fifty-seven (78%) OPAs had a CVS ≥4.0 leaving 16 (22%) below the pre-established threshold for content validity. Trainees assigned a higher importance to individual OPAs (mean CVS 4.6) compared to faculty (mean 4.4, p = 0.016), but the effect size was small (η2 = 0.10). There was no demonstrated effect of length of education experience on perceived importance of neurology OPAs (p = 0.23). Two sample resident assessment formats were developed, one using neurology OPAs alone and another using a combination of neurology OPAs and the Neurology Milestones. Conclusions: This study provides neurology training programs with content validity evidence for items to include in resident assessments, and sample assessment formats that directly translate to the Neurology Milestones. Length of education experience has little effect on perceptions of neurology OPA importance.

UR - http://www.scopus.com/inward/record.url?scp=84930372396&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84930372396&partnerID=8YFLogxK

U2 - 10.1212/WNL.0000000000001641

DO - 10.1212/WNL.0000000000001641

M3 - Article

C2 - 25934848

AN - SCOPUS:84930372396

VL - 84

SP - 2279

EP - 2283

JO - Neurology

JF - Neurology

SN - 0028-3878

IS - 22

ER -