TY - JOUR
T1 - Consequences validity evidence
T2 - Evaluating the impact of educational assessments
AU - Cook, David A.
AU - Lineberry, Matthew
N1 - Publisher Copyright:
© 2016 by the Association of American Medical Colleges.
PY - 2016/6/1
Y1 - 2016/6/1
N2 - Because tests that do not alter management (i.e., influence decisions and actions) should not be performed, data on the consequences of assessment constitute a critical source of validity evidence. Consequences validity evidence is challenging for many educators to understand, perhaps because it has no counterpart in the older framework of content, criterion, and construct validity. The authors' purpose is to explain consequences validity evidence and propose a framework for organizing its collection and interpretation. Both clinical and educational assessments can be viewed as interventions. The act of administering or taking a test, the interpretation of scores, and the ensuing decisions and actions influence those being assessed (e.g., patients or students) and other people and systems (e.g., physicians, teachers, hospitals, schools). Consequences validity evidence examines such impacts of assessments. Despite its importance, consequences evidence is reported infrequently in health professions education (range 5%-20% of studies in recent systematic reviews) and is typically limited in scope and rigor. Consequences validity evidence can derive from evaluations of the impact on examinees, educators, schools, or the end target of practice (e.g., patients or health care systems); and the downstream impact of classifications (e.g., different score cut points and labels). Impact can result from the uses of scores or from the assessment activity itself, and can be intended or unintended and beneficial or harmful. Both quantitative and qualitative research methods are useful. The type, quantity, and rigor of consequences evidence required will vary depending on the assessment and the claims for its use.
AB - Because tests that do not alter management (i.e., influence decisions and actions) should not be performed, data on the consequences of assessment constitute a critical source of validity evidence. Consequences validity evidence is challenging for many educators to understand, perhaps because it has no counterpart in the older framework of content, criterion, and construct validity. The authors' purpose is to explain consequences validity evidence and propose a framework for organizing its collection and interpretation. Both clinical and educational assessments can be viewed as interventions. The act of administering or taking a test, the interpretation of scores, and the ensuing decisions and actions influence those being assessed (e.g., patients or students) and other people and systems (e.g., physicians, teachers, hospitals, schools). Consequences validity evidence examines such impacts of assessments. Despite its importance, consequences evidence is reported infrequently in health professions education (range 5%-20% of studies in recent systematic reviews) and is typically limited in scope and rigor. Consequences validity evidence can derive from evaluations of the impact on examinees, educators, schools, or the end target of practice (e.g., patients or health care systems); and the downstream impact of classifications (e.g., different score cut points and labels). Impact can result from the uses of scores or from the assessment activity itself, and can be intended or unintended and beneficial or harmful. Both quantitative and qualitative research methods are useful. The type, quantity, and rigor of consequences evidence required will vary depending on the assessment and the claims for its use.
UR - http://www.scopus.com/inward/record.url?scp=84956954652&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84956954652&partnerID=8YFLogxK
U2 - 10.1097/ACM.0000000000001114
DO - 10.1097/ACM.0000000000001114
M3 - Article
C2 - 26839945
AN - SCOPUS:84956954652
SN - 1040-2446
VL - 91
SP - 785
EP - 795
JO - Academic Medicine
JF - Academic Medicine
IS - 6
ER -