Consequences validity evidence: Evaluating the impact of educational assessments

David Allan Cook, Matthew Lineberry

Research output: Contribution to journalArticle

38 Citations (Scopus)

Abstract

Because tests that do not alter management (i.e., influence decisions and actions) should not be performed, data on the consequences of assessment constitute a critical source of validity evidence. Consequences validity evidence is challenging for many educators to understand, perhaps because it has no counterpart in the older framework of content, criterion, and construct validity. The authors' purpose is to explain consequences validity evidence and propose a framework for organizing its collection and interpretation. Both clinical and educational assessments can be viewed as interventions. The act of administering or taking a test, the interpretation of scores, and the ensuing decisions and actions influence those being assessed (e.g., patients or students) and other people and systems (e.g., physicians, teachers, hospitals, schools). Consequences validity evidence examines such impacts of assessments. Despite its importance, consequences evidence is reported infrequently in health professions education (range 5%-20% of studies in recent systematic reviews) and is typically limited in scope and rigor. Consequences validity evidence can derive from evaluations of the impact on examinees, educators, schools, or the end target of practice (e.g., patients or health care systems); and the downstream impact of classifications (e.g., different score cut points and labels). Impact can result from the uses of scores or from the assessment activity itself, and can be intended or unintended and beneficial or harmful. Both quantitative and qualitative research methods are useful. The type, quantity, and rigor of consequences evidence required will vary depending on the assessment and the claims for its use.

Original languageEnglish (US)
Pages (from-to)785-795
Number of pages11
JournalAcademic Medicine
Volume91
Issue number6
DOIs
StatePublished - Jun 1 2016

Fingerprint

Educational Measurement
Health Occupations
Qualitative Research
Health Education
Patient Care
Students
Delivery of Health Care
Physicians
evidence
educator
interpretation
quantitative research
construct validity
School Teachers
patient care
qualitative method
school
research method
qualitative research
profession

ASJC Scopus subject areas

  • Medicine(all)
  • Education

Cite this

Consequences validity evidence : Evaluating the impact of educational assessments. / Cook, David Allan; Lineberry, Matthew.

In: Academic Medicine, Vol. 91, No. 6, 01.06.2016, p. 785-795.

Research output: Contribution to journalArticle

@article{d9359861cd1b44648df86b9ec4f9fa24,
title = "Consequences validity evidence: Evaluating the impact of educational assessments",
abstract = "Because tests that do not alter management (i.e., influence decisions and actions) should not be performed, data on the consequences of assessment constitute a critical source of validity evidence. Consequences validity evidence is challenging for many educators to understand, perhaps because it has no counterpart in the older framework of content, criterion, and construct validity. The authors' purpose is to explain consequences validity evidence and propose a framework for organizing its collection and interpretation. Both clinical and educational assessments can be viewed as interventions. The act of administering or taking a test, the interpretation of scores, and the ensuing decisions and actions influence those being assessed (e.g., patients or students) and other people and systems (e.g., physicians, teachers, hospitals, schools). Consequences validity evidence examines such impacts of assessments. Despite its importance, consequences evidence is reported infrequently in health professions education (range 5{\%}-20{\%} of studies in recent systematic reviews) and is typically limited in scope and rigor. Consequences validity evidence can derive from evaluations of the impact on examinees, educators, schools, or the end target of practice (e.g., patients or health care systems); and the downstream impact of classifications (e.g., different score cut points and labels). Impact can result from the uses of scores or from the assessment activity itself, and can be intended or unintended and beneficial or harmful. Both quantitative and qualitative research methods are useful. The type, quantity, and rigor of consequences evidence required will vary depending on the assessment and the claims for its use.",
author = "Cook, {David Allan} and Matthew Lineberry",
year = "2016",
month = "6",
day = "1",
doi = "10.1097/ACM.0000000000001114",
language = "English (US)",
volume = "91",
pages = "785--795",
journal = "Academic Medicine",
issn = "1040-2446",
publisher = "Lippincott Williams and Wilkins",
number = "6",

}

TY - JOUR

T1 - Consequences validity evidence

T2 - Evaluating the impact of educational assessments

AU - Cook, David Allan

AU - Lineberry, Matthew

PY - 2016/6/1

Y1 - 2016/6/1

N2 - Because tests that do not alter management (i.e., influence decisions and actions) should not be performed, data on the consequences of assessment constitute a critical source of validity evidence. Consequences validity evidence is challenging for many educators to understand, perhaps because it has no counterpart in the older framework of content, criterion, and construct validity. The authors' purpose is to explain consequences validity evidence and propose a framework for organizing its collection and interpretation. Both clinical and educational assessments can be viewed as interventions. The act of administering or taking a test, the interpretation of scores, and the ensuing decisions and actions influence those being assessed (e.g., patients or students) and other people and systems (e.g., physicians, teachers, hospitals, schools). Consequences validity evidence examines such impacts of assessments. Despite its importance, consequences evidence is reported infrequently in health professions education (range 5%-20% of studies in recent systematic reviews) and is typically limited in scope and rigor. Consequences validity evidence can derive from evaluations of the impact on examinees, educators, schools, or the end target of practice (e.g., patients or health care systems); and the downstream impact of classifications (e.g., different score cut points and labels). Impact can result from the uses of scores or from the assessment activity itself, and can be intended or unintended and beneficial or harmful. Both quantitative and qualitative research methods are useful. The type, quantity, and rigor of consequences evidence required will vary depending on the assessment and the claims for its use.

AB - Because tests that do not alter management (i.e., influence decisions and actions) should not be performed, data on the consequences of assessment constitute a critical source of validity evidence. Consequences validity evidence is challenging for many educators to understand, perhaps because it has no counterpart in the older framework of content, criterion, and construct validity. The authors' purpose is to explain consequences validity evidence and propose a framework for organizing its collection and interpretation. Both clinical and educational assessments can be viewed as interventions. The act of administering or taking a test, the interpretation of scores, and the ensuing decisions and actions influence those being assessed (e.g., patients or students) and other people and systems (e.g., physicians, teachers, hospitals, schools). Consequences validity evidence examines such impacts of assessments. Despite its importance, consequences evidence is reported infrequently in health professions education (range 5%-20% of studies in recent systematic reviews) and is typically limited in scope and rigor. Consequences validity evidence can derive from evaluations of the impact on examinees, educators, schools, or the end target of practice (e.g., patients or health care systems); and the downstream impact of classifications (e.g., different score cut points and labels). Impact can result from the uses of scores or from the assessment activity itself, and can be intended or unintended and beneficial or harmful. Both quantitative and qualitative research methods are useful. The type, quantity, and rigor of consequences evidence required will vary depending on the assessment and the claims for its use.

UR - http://www.scopus.com/inward/record.url?scp=84956954652&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84956954652&partnerID=8YFLogxK

U2 - 10.1097/ACM.0000000000001114

DO - 10.1097/ACM.0000000000001114

M3 - Article

C2 - 26839945

AN - SCOPUS:84956954652

VL - 91

SP - 785

EP - 795

JO - Academic Medicine

JF - Academic Medicine

SN - 1040-2446

IS - 6

ER -