TY - JOUR
T1 - Validity and feasibility of the EMG direct observation tool (EMG-DOT)
AU - Leep Hunderfund, Andrea N.
AU - Rubin, Devon I.
AU - Laughlin, Ruple S.
AU - Sorenson, Eric J.
AU - Watson, James C.
AU - Jones, Lyell K.
AU - Juul, Dorthea
AU - Park, Yoon Soo
N1 - Publisher Copyright:
© 2016 American Academy of Neurology.
PY - 2016/4/26
Y1 - 2016/4/26
N2 - Objective: To develop a new workplace-based EMG direct observation tool (EMG-DOT) and gather validity evidence supporting its use for assessing electrodiagnostic skills among postgraduate medical trainees. Methods: The EMG-DOT was developed by experts using an iterative process. Validity evidence from content, response process, internal structure, relations to other variables, and consequences of testing was collected during the 2013-2014 academic year. Results: Of 3,412 studies performed by trainees during the study period, 299 (9%) were assessed using the EMG-DOT. Of these, 203 (68%) involved a physician rater and 96 (32%) involved a technician rater. The 14-item EMG-DOT had excellent internal-consistency reliability (Cronbach α 0.94). Correlations between individual items and criterion-referenced global ratings of performance ranged from 0.36 to 0.72 (all p < 0.001). Mean total scores increased from 70% to 80% over 4 months of the EMG rotation (p < 0.001) despite a corresponding significant increase in case complexity (0.21-0.74 on a 3-point rating scale; p < 0.001). Trainees reported that the observational assessment exercise improved their knowledge or skills in 82% of encounters (188/230) and that feedback generated by the EMG-DOT improved the quality of care provided to patients in 58% (133/230). Trainees were "satisfied" or "very satisfied" with the observational assessment exercise in 96% of encounters (234/243). Conclusions: This study provides validity evidence supporting the use of EMG-DOT scores to assess electrodiagnostic skills of residents and fellows. The EMG-DOT can be used to inform milestone-based assessments of trainee performance in neurology, child neurology, physical medicine and rehabilitation, neuromuscular, and clinical neurophysiology training programs.
AB - Objective: To develop a new workplace-based EMG direct observation tool (EMG-DOT) and gather validity evidence supporting its use for assessing electrodiagnostic skills among postgraduate medical trainees. Methods: The EMG-DOT was developed by experts using an iterative process. Validity evidence from content, response process, internal structure, relations to other variables, and consequences of testing was collected during the 2013-2014 academic year. Results: Of 3,412 studies performed by trainees during the study period, 299 (9%) were assessed using the EMG-DOT. Of these, 203 (68%) involved a physician rater and 96 (32%) involved a technician rater. The 14-item EMG-DOT had excellent internal-consistency reliability (Cronbach α 0.94). Correlations between individual items and criterion-referenced global ratings of performance ranged from 0.36 to 0.72 (all p < 0.001). Mean total scores increased from 70% to 80% over 4 months of the EMG rotation (p < 0.001) despite a corresponding significant increase in case complexity (0.21-0.74 on a 3-point rating scale; p < 0.001). Trainees reported that the observational assessment exercise improved their knowledge or skills in 82% of encounters (188/230) and that feedback generated by the EMG-DOT improved the quality of care provided to patients in 58% (133/230). Trainees were "satisfied" or "very satisfied" with the observational assessment exercise in 96% of encounters (234/243). Conclusions: This study provides validity evidence supporting the use of EMG-DOT scores to assess electrodiagnostic skills of residents and fellows. The EMG-DOT can be used to inform milestone-based assessments of trainee performance in neurology, child neurology, physical medicine and rehabilitation, neuromuscular, and clinical neurophysiology training programs.
UR - http://www.scopus.com/inward/record.url?scp=84964846649&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84964846649&partnerID=8YFLogxK
U2 - 10.1212/WNL.0000000000002609
DO - 10.1212/WNL.0000000000002609
M3 - Article
C2 - 27029634
AN - SCOPUS:84964846649
SN - 0028-3878
VL - 86
SP - 1627
EP - 1634
JO - Neurology
JF - Neurology
IS - 17
ER -