TY - JOUR
T1 - Group assessments of resident physicians improve reliability and decrease halo error
AU - Thomas, Matthew R.
AU - Beckman, Thomas J.
AU - Mauck, Karen F.
AU - Cha, Stephen S.
AU - Thomas, Kris G.
N1 - Funding Information:
Acknowledgement: This project was supported in part through an Educational Innovations grant, Mayo Clinic Rochester.
PY - 2011/7
Y1 - 2011/7
N2 - Background: Individual faculty assessments of resident competency are complicated by inconsistent application of standards, lack of reliability, and the "halo" effect. Objective: We determined whether the addition of faculty group assessments of residents in an ambulatory clinic, compared with individual faculty-of-resident assessments alone, have better reliability and reduced halo effects. Design: This prospective, longitudinal study was performed in the outpatient continuity clinics of a large internal medicine residency program. Main measures: Faculty-on-resident and group faculty-on-resident assessment scores were used for comparison. Key results: Overall mean scores were significantly higher for group than individual assessments (3.92±0.51 vs. 3.83±0.38, p=0.0001). Overall inter-rater reliability increased when combining group and individual assessments compared to individual assessments alone (intraclass correlation coefficient, 95% CI=0.828, 0.785-0.866 vs. 0.749, 0.686-0.804). Inter-item correlations were less for group (0.49) than individual (0.68) assessments. Conclusions: This study demonstrates improved inter-rater reliability and reduced range restriction (halo effect) of resident assessment across multiple performance domains by adding the group assessmentmethod to traditional individual faculty-on-resident assessment. This feasible model could help graduate medical education programs achieve more reliable and discriminating resident assessments.
AB - Background: Individual faculty assessments of resident competency are complicated by inconsistent application of standards, lack of reliability, and the "halo" effect. Objective: We determined whether the addition of faculty group assessments of residents in an ambulatory clinic, compared with individual faculty-of-resident assessments alone, have better reliability and reduced halo effects. Design: This prospective, longitudinal study was performed in the outpatient continuity clinics of a large internal medicine residency program. Main measures: Faculty-on-resident and group faculty-on-resident assessment scores were used for comparison. Key results: Overall mean scores were significantly higher for group than individual assessments (3.92±0.51 vs. 3.83±0.38, p=0.0001). Overall inter-rater reliability increased when combining group and individual assessments compared to individual assessments alone (intraclass correlation coefficient, 95% CI=0.828, 0.785-0.866 vs. 0.749, 0.686-0.804). Inter-item correlations were less for group (0.49) than individual (0.68) assessments. Conclusions: This study demonstrates improved inter-rater reliability and reduced range restriction (halo effect) of resident assessment across multiple performance domains by adding the group assessmentmethod to traditional individual faculty-on-resident assessment. This feasible model could help graduate medical education programs achieve more reliable and discriminating resident assessments.
KW - Educational measurement
KW - Formal evaluation sessions
KW - Graduate medical education
KW - Professional competence
UR - http://www.scopus.com/inward/record.url?scp=80051544847&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=80051544847&partnerID=8YFLogxK
U2 - 10.1007/s11606-011-1670-4
DO - 10.1007/s11606-011-1670-4
M3 - Article
C2 - 21369769
AN - SCOPUS:80051544847
SN - 0884-8734
VL - 26
SP - 759
EP - 764
JO - Journal of general internal medicine
JF - Journal of general internal medicine
IS - 7
ER -