TY - JOUR
T1 - Response Process Validity Evidence for Video Commentary Assessment in Surgery
T2 - A Qualitative Study
AU - Yeh, Vicky J.H.
AU - Mukhtar, Fareeda
AU - Yudkowsky, Rachel
AU - Baloul, Mohamed S.
AU - Farley, David R.
AU - Cook, David A.
N1 - Funding Information:
The authors thank Courtney Backstrom, BS, Abhishek Chandra, BA, Rafael de Azevedo, MD, Anne Kamphausen, MD, Aashish Rajesh, MBBS, Mariela Rivera, MD, and Nizamuddin Shaikh, MBBS for assisting in data collection.
Publisher Copyright:
© 2022
PY - 2022/9/1
Y1 - 2022/9/1
N2 - OBJECTIVES: Well-developed mental representations of a task are fundamental to proficient performance. ‘Video Commentary’ (VC) is a novel assessment intended to measure mental representations of surgical tasks that would reflect an important aspect of task proficiency. Whether examinees’ actual response processes align with this intent remains unknown. As part of ongoing validation of the assessment, we sought to understand examinees’ response processes in VC. DESIGN: Grounded theory qualitative study. In 2019, residents were interviewed about their understanding of and approach to VC. Using grounded theory, we created a theoretical model explaining relationships among factors that influence residents' response processes and performance. Residents' perceived purpose of VC was also explored using Likert-type questions. SETTING: Academic surgical residency program. PARTICIPANTS: Forty-eight surgical residents (PGY-1 to PGY-5). RESULTS: Analysis of narrative comments indicated that residents' perceived purposes of VC generally align with the educator's intent. Resident response processes are influenced by test characteristics, residents' perception and understanding of VC, and residents' personal characteristics. Four strategies seem to guide how residents respond, namely a focus on speed, points, logic, and relevance. Quantitative results indicated residents believe VC scores reflect their ability to speak quickly, ability to think quickly, and knowledge of anatomy (mean = 5.0, 4.5, and 4.4 respectively [1 = strongly disagree, 6 = strongly agree]). PGY-1 and PGY-2 residents tend to focus on naming facts whereas PGY-4 and PGY-5 residents focus on providing comprehensive descriptions. CONCLUSIONS: Residents generally have an accurate understanding of the purpose of VC. However, their use of different approaches could represent a threat to validity. The response strategies of speed, points, logic, and relevance may inform other clinical skills assessments.
AB - OBJECTIVES: Well-developed mental representations of a task are fundamental to proficient performance. ‘Video Commentary’ (VC) is a novel assessment intended to measure mental representations of surgical tasks that would reflect an important aspect of task proficiency. Whether examinees’ actual response processes align with this intent remains unknown. As part of ongoing validation of the assessment, we sought to understand examinees’ response processes in VC. DESIGN: Grounded theory qualitative study. In 2019, residents were interviewed about their understanding of and approach to VC. Using grounded theory, we created a theoretical model explaining relationships among factors that influence residents' response processes and performance. Residents' perceived purpose of VC was also explored using Likert-type questions. SETTING: Academic surgical residency program. PARTICIPANTS: Forty-eight surgical residents (PGY-1 to PGY-5). RESULTS: Analysis of narrative comments indicated that residents' perceived purposes of VC generally align with the educator's intent. Resident response processes are influenced by test characteristics, residents' perception and understanding of VC, and residents' personal characteristics. Four strategies seem to guide how residents respond, namely a focus on speed, points, logic, and relevance. Quantitative results indicated residents believe VC scores reflect their ability to speak quickly, ability to think quickly, and knowledge of anatomy (mean = 5.0, 4.5, and 4.4 respectively [1 = strongly disagree, 6 = strongly agree]). PGY-1 and PGY-2 residents tend to focus on naming facts whereas PGY-4 and PGY-5 residents focus on providing comprehensive descriptions. CONCLUSIONS: Residents generally have an accurate understanding of the purpose of VC. However, their use of different approaches could represent a threat to validity. The response strategies of speed, points, logic, and relevance may inform other clinical skills assessments.
KW - Educational Measurement
KW - Mental Schema
KW - Response Process
KW - Surgery Education
KW - Think Aloud
KW - Validation Study
UR - http://www.scopus.com/inward/record.url?scp=85131790087&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85131790087&partnerID=8YFLogxK
U2 - 10.1016/j.jsurg.2022.05.006
DO - 10.1016/j.jsurg.2022.05.006
M3 - Article
C2 - 35688704
AN - SCOPUS:85131790087
SN - 1931-7204
VL - 79
SP - 1270
EP - 1281
JO - Journal of Surgical Education
JF - Journal of Surgical Education
IS - 5
ER -