When Assessment Data Are Words: Validity Evidence for Qualitative Educational Assessments

David Allan Cook, Ayelet Kuper, Rose Hatala, Shiphra Ginsburg

Research output: Contribution to journalArticle

31 Citations (Scopus)

Abstract

Quantitative scores fail to capture all important features of learner performance. This awareness has led to increased use of qualitative data when assessing health professionals. Yet the use of qualitative assessments is hampered by incomplete understanding of their role in forming judgments, and lack of consensus in how to appraise the rigor of judgments therein derived. The authors articulate the role of qualitative assessment as part of a comprehensive program of assessment, and translate the concept of validity to apply to judgments arising from qualitative assessments. They first identify standards for rigor in qualitative research, and then use two contemporary assessment validity frameworks to reorganize these standards for application to qualitative assessment.Standards for rigor in qualitative research include responsiveness, reflexivity, purposive sampling, thick description, triangulation, transparency, and transferability. These standards can be reframed using Messick’s five sources of validity evidence (content, response process, internal structure, relationships with other variables, and consequences) and Kane’s four inferences in validation (scoring, generalization, extrapolation, and implications). Evidence can be collected and evaluated for each evidence source or inference. The authors illustrate this approach using published research on learning portfolios.The authors advocate a “methods-neutral” approach to assessment, in which a clearly stated purpose determines the nature of and approach to data collection and analysis. Increased use of qualitative assessments will necessitate more rigorous judgments of the defensibility (validity) of inferences and decisions. Evidence should be strategically sought to inform a coherent validity argument.

Original languageEnglish (US)
JournalAcademic Medicine
DOIs
StateAccepted/In press - Apr 5 2016

Fingerprint

Educational Measurement
Qualitative Research
evidence
qualitative research
Consensus
Learning
triangulation
Health
reflexivity
Research
health professionals
transparency
data analysis

ASJC Scopus subject areas

  • Medicine(all)
  • Education

Cite this

When Assessment Data Are Words : Validity Evidence for Qualitative Educational Assessments. / Cook, David Allan; Kuper, Ayelet; Hatala, Rose; Ginsburg, Shiphra.

In: Academic Medicine, 05.04.2016.

Research output: Contribution to journalArticle

@article{a4bca12446b24d7c92bc2ab007217158,
title = "When Assessment Data Are Words: Validity Evidence for Qualitative Educational Assessments",
abstract = "Quantitative scores fail to capture all important features of learner performance. This awareness has led to increased use of qualitative data when assessing health professionals. Yet the use of qualitative assessments is hampered by incomplete understanding of their role in forming judgments, and lack of consensus in how to appraise the rigor of judgments therein derived. The authors articulate the role of qualitative assessment as part of a comprehensive program of assessment, and translate the concept of validity to apply to judgments arising from qualitative assessments. They first identify standards for rigor in qualitative research, and then use two contemporary assessment validity frameworks to reorganize these standards for application to qualitative assessment.Standards for rigor in qualitative research include responsiveness, reflexivity, purposive sampling, thick description, triangulation, transparency, and transferability. These standards can be reframed using Messick’s five sources of validity evidence (content, response process, internal structure, relationships with other variables, and consequences) and Kane’s four inferences in validation (scoring, generalization, extrapolation, and implications). Evidence can be collected and evaluated for each evidence source or inference. The authors illustrate this approach using published research on learning portfolios.The authors advocate a “methods-neutral” approach to assessment, in which a clearly stated purpose determines the nature of and approach to data collection and analysis. Increased use of qualitative assessments will necessitate more rigorous judgments of the defensibility (validity) of inferences and decisions. Evidence should be strategically sought to inform a coherent validity argument.",
author = "Cook, {David Allan} and Ayelet Kuper and Rose Hatala and Shiphra Ginsburg",
year = "2016",
month = "4",
day = "5",
doi = "10.1097/ACM.0000000000001175",
language = "English (US)",
journal = "Academic Medicine",
issn = "1040-2446",
publisher = "Lippincott Williams and Wilkins",

}

TY - JOUR

T1 - When Assessment Data Are Words

T2 - Validity Evidence for Qualitative Educational Assessments

AU - Cook, David Allan

AU - Kuper, Ayelet

AU - Hatala, Rose

AU - Ginsburg, Shiphra

PY - 2016/4/5

Y1 - 2016/4/5

N2 - Quantitative scores fail to capture all important features of learner performance. This awareness has led to increased use of qualitative data when assessing health professionals. Yet the use of qualitative assessments is hampered by incomplete understanding of their role in forming judgments, and lack of consensus in how to appraise the rigor of judgments therein derived. The authors articulate the role of qualitative assessment as part of a comprehensive program of assessment, and translate the concept of validity to apply to judgments arising from qualitative assessments. They first identify standards for rigor in qualitative research, and then use two contemporary assessment validity frameworks to reorganize these standards for application to qualitative assessment.Standards for rigor in qualitative research include responsiveness, reflexivity, purposive sampling, thick description, triangulation, transparency, and transferability. These standards can be reframed using Messick’s five sources of validity evidence (content, response process, internal structure, relationships with other variables, and consequences) and Kane’s four inferences in validation (scoring, generalization, extrapolation, and implications). Evidence can be collected and evaluated for each evidence source or inference. The authors illustrate this approach using published research on learning portfolios.The authors advocate a “methods-neutral” approach to assessment, in which a clearly stated purpose determines the nature of and approach to data collection and analysis. Increased use of qualitative assessments will necessitate more rigorous judgments of the defensibility (validity) of inferences and decisions. Evidence should be strategically sought to inform a coherent validity argument.

AB - Quantitative scores fail to capture all important features of learner performance. This awareness has led to increased use of qualitative data when assessing health professionals. Yet the use of qualitative assessments is hampered by incomplete understanding of their role in forming judgments, and lack of consensus in how to appraise the rigor of judgments therein derived. The authors articulate the role of qualitative assessment as part of a comprehensive program of assessment, and translate the concept of validity to apply to judgments arising from qualitative assessments. They first identify standards for rigor in qualitative research, and then use two contemporary assessment validity frameworks to reorganize these standards for application to qualitative assessment.Standards for rigor in qualitative research include responsiveness, reflexivity, purposive sampling, thick description, triangulation, transparency, and transferability. These standards can be reframed using Messick’s five sources of validity evidence (content, response process, internal structure, relationships with other variables, and consequences) and Kane’s four inferences in validation (scoring, generalization, extrapolation, and implications). Evidence can be collected and evaluated for each evidence source or inference. The authors illustrate this approach using published research on learning portfolios.The authors advocate a “methods-neutral” approach to assessment, in which a clearly stated purpose determines the nature of and approach to data collection and analysis. Increased use of qualitative assessments will necessitate more rigorous judgments of the defensibility (validity) of inferences and decisions. Evidence should be strategically sought to inform a coherent validity argument.

UR - http://www.scopus.com/inward/record.url?scp=84962338979&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84962338979&partnerID=8YFLogxK

U2 - 10.1097/ACM.0000000000001175

DO - 10.1097/ACM.0000000000001175

M3 - Article

AN - SCOPUS:84962338979

JO - Academic Medicine

JF - Academic Medicine

SN - 1040-2446

ER -