Internal structure of mini-CEX scores for internal medicine residents: Factor analysis and generalizability

David Allan Cook, Thomas J. Beckman, Jayawant Mandrekar, V. Shane Pankratz

Research output: Contribution to journalArticle

42 Citations (Scopus)

Abstract

The mini-CEX is widely used to rate directly observed resident-patient encounters. Although several studies have explored the reliability of mini-CEX scores, the dimensionality of mini-CEX scores is incompletely understood. Objective: Explore the dimensionality of mini-CEX scores through factor analysis and generalizability analysis. Design: Factor analytic and generalizability study using retrospective data. Participants: Eighty five physician preceptors and 264 internal medicine residents (postgraduate years 1-3). Methods: Preceptors used the six-item mini-CEX to rate directly observed resident-patient encounters in internal medicine resident continuity clinics. We analyzed mini-CEX scores accrued over 4 years using repeated measures analysis of variance to generate a correlation matrix adjusted for multiple observations on individual residents, and then performed factor analysis on this adjusted correlation matrix. We also performed generalizability analyses. Results: Eighty-five preceptors rated 264 residents in 1,414 resident-patient encounters. Common factor analysis of these scores after adjustment for repeated measures revealed a single-factor solution. Cronbach's alpha for this single factor (i. e. all six mini-CEX items) was ≥0. 86. Sensitivity analyses using principal components and other method variations revealed a similar factor structure. Generalizability studies revealed a reproducibility coefficient of 0.23 (0.70 for 10 raters or encounters). Conclusions: The mini-CEX appears to measure a single global dimension of clinical competence. If educators desire to measure discrete clinical skills, alternative assessment methods may be required. Our approach to factor analysis overcomes the limitation of repeated observations on subjects without discarding data, and may be useful to other researchers attempting factor analysis of datasets in which individuals contribute multiple observations.

Original languageEnglish (US)
Pages (from-to)633-645
Number of pages13
JournalAdvances in Health Sciences Education
Volume15
Issue number5
DOIs
StatePublished - Dec 2010

Fingerprint

Internal Medicine
Statistical Factor Analysis
factor analysis
medicine
resident
Clinical Competence
Principal Component Analysis
Analysis of Variance
Retrospective Studies
Research Personnel
analysis of variance
Physicians
continuity
physician
educator

Keywords

  • Assessment
  • Clinical competence
  • Educational measurement
  • Factor analysis
  • Internal consistency
  • Medical education
  • Psychometrics
  • Reproducibility of results

ASJC Scopus subject areas

  • Medicine(all)
  • Education

Cite this

Internal structure of mini-CEX scores for internal medicine residents : Factor analysis and generalizability. / Cook, David Allan; Beckman, Thomas J.; Mandrekar, Jayawant; Pankratz, V. Shane.

In: Advances in Health Sciences Education, Vol. 15, No. 5, 12.2010, p. 633-645.

Research output: Contribution to journalArticle

@article{60e04f51f14d4645ad862f7a4fce4566,
title = "Internal structure of mini-CEX scores for internal medicine residents: Factor analysis and generalizability",
abstract = "The mini-CEX is widely used to rate directly observed resident-patient encounters. Although several studies have explored the reliability of mini-CEX scores, the dimensionality of mini-CEX scores is incompletely understood. Objective: Explore the dimensionality of mini-CEX scores through factor analysis and generalizability analysis. Design: Factor analytic and generalizability study using retrospective data. Participants: Eighty five physician preceptors and 264 internal medicine residents (postgraduate years 1-3). Methods: Preceptors used the six-item mini-CEX to rate directly observed resident-patient encounters in internal medicine resident continuity clinics. We analyzed mini-CEX scores accrued over 4 years using repeated measures analysis of variance to generate a correlation matrix adjusted for multiple observations on individual residents, and then performed factor analysis on this adjusted correlation matrix. We also performed generalizability analyses. Results: Eighty-five preceptors rated 264 residents in 1,414 resident-patient encounters. Common factor analysis of these scores after adjustment for repeated measures revealed a single-factor solution. Cronbach's alpha for this single factor (i. e. all six mini-CEX items) was ≥0. 86. Sensitivity analyses using principal components and other method variations revealed a similar factor structure. Generalizability studies revealed a reproducibility coefficient of 0.23 (0.70 for 10 raters or encounters). Conclusions: The mini-CEX appears to measure a single global dimension of clinical competence. If educators desire to measure discrete clinical skills, alternative assessment methods may be required. Our approach to factor analysis overcomes the limitation of repeated observations on subjects without discarding data, and may be useful to other researchers attempting factor analysis of datasets in which individuals contribute multiple observations.",
keywords = "Assessment, Clinical competence, Educational measurement, Factor analysis, Internal consistency, Medical education, Psychometrics, Reproducibility of results",
author = "Cook, {David Allan} and Beckman, {Thomas J.} and Jayawant Mandrekar and Pankratz, {V. Shane}",
year = "2010",
month = "12",
doi = "10.1007/s10459-010-9224-9",
language = "English (US)",
volume = "15",
pages = "633--645",
journal = "Advances in Health Sciences Education",
issn = "1382-4996",
publisher = "Springer Netherlands",
number = "5",

}

TY - JOUR

T1 - Internal structure of mini-CEX scores for internal medicine residents

T2 - Factor analysis and generalizability

AU - Cook, David Allan

AU - Beckman, Thomas J.

AU - Mandrekar, Jayawant

AU - Pankratz, V. Shane

PY - 2010/12

Y1 - 2010/12

N2 - The mini-CEX is widely used to rate directly observed resident-patient encounters. Although several studies have explored the reliability of mini-CEX scores, the dimensionality of mini-CEX scores is incompletely understood. Objective: Explore the dimensionality of mini-CEX scores through factor analysis and generalizability analysis. Design: Factor analytic and generalizability study using retrospective data. Participants: Eighty five physician preceptors and 264 internal medicine residents (postgraduate years 1-3). Methods: Preceptors used the six-item mini-CEX to rate directly observed resident-patient encounters in internal medicine resident continuity clinics. We analyzed mini-CEX scores accrued over 4 years using repeated measures analysis of variance to generate a correlation matrix adjusted for multiple observations on individual residents, and then performed factor analysis on this adjusted correlation matrix. We also performed generalizability analyses. Results: Eighty-five preceptors rated 264 residents in 1,414 resident-patient encounters. Common factor analysis of these scores after adjustment for repeated measures revealed a single-factor solution. Cronbach's alpha for this single factor (i. e. all six mini-CEX items) was ≥0. 86. Sensitivity analyses using principal components and other method variations revealed a similar factor structure. Generalizability studies revealed a reproducibility coefficient of 0.23 (0.70 for 10 raters or encounters). Conclusions: The mini-CEX appears to measure a single global dimension of clinical competence. If educators desire to measure discrete clinical skills, alternative assessment methods may be required. Our approach to factor analysis overcomes the limitation of repeated observations on subjects without discarding data, and may be useful to other researchers attempting factor analysis of datasets in which individuals contribute multiple observations.

AB - The mini-CEX is widely used to rate directly observed resident-patient encounters. Although several studies have explored the reliability of mini-CEX scores, the dimensionality of mini-CEX scores is incompletely understood. Objective: Explore the dimensionality of mini-CEX scores through factor analysis and generalizability analysis. Design: Factor analytic and generalizability study using retrospective data. Participants: Eighty five physician preceptors and 264 internal medicine residents (postgraduate years 1-3). Methods: Preceptors used the six-item mini-CEX to rate directly observed resident-patient encounters in internal medicine resident continuity clinics. We analyzed mini-CEX scores accrued over 4 years using repeated measures analysis of variance to generate a correlation matrix adjusted for multiple observations on individual residents, and then performed factor analysis on this adjusted correlation matrix. We also performed generalizability analyses. Results: Eighty-five preceptors rated 264 residents in 1,414 resident-patient encounters. Common factor analysis of these scores after adjustment for repeated measures revealed a single-factor solution. Cronbach's alpha for this single factor (i. e. all six mini-CEX items) was ≥0. 86. Sensitivity analyses using principal components and other method variations revealed a similar factor structure. Generalizability studies revealed a reproducibility coefficient of 0.23 (0.70 for 10 raters or encounters). Conclusions: The mini-CEX appears to measure a single global dimension of clinical competence. If educators desire to measure discrete clinical skills, alternative assessment methods may be required. Our approach to factor analysis overcomes the limitation of repeated observations on subjects without discarding data, and may be useful to other researchers attempting factor analysis of datasets in which individuals contribute multiple observations.

KW - Assessment

KW - Clinical competence

KW - Educational measurement

KW - Factor analysis

KW - Internal consistency

KW - Medical education

KW - Psychometrics

KW - Reproducibility of results

UR - http://www.scopus.com/inward/record.url?scp=78649908982&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=78649908982&partnerID=8YFLogxK

U2 - 10.1007/s10459-010-9224-9

DO - 10.1007/s10459-010-9224-9

M3 - Article

C2 - 21120648

AN - SCOPUS:78649908982

VL - 15

SP - 633

EP - 645

JO - Advances in Health Sciences Education

JF - Advances in Health Sciences Education

SN - 1382-4996

IS - 5

ER -