Much ado about differences: why expert-novice comparisons add little to the validity argument

Research output: Contribution to journalArticle

31 Citations (Scopus)

Abstract

One approach to validating assessment scores involves evaluating the ability of scores to discriminate among groups who differ in a specific characteristic, such as training status (in education) or disease state (in clinical applications). Such known-groups comparison studies provide validity evidence of “relationships with other variables.” The typical education research study might compare scores between staff physicians and postgraduate trainees with the hypothesis that those with more advanced training (the “experts”) will have higher scores than those less advanced (the “novices”). However, such comparisons are too nonspecific to support clear conclusions, and expert-novice comparisons (and known-groups comparisons in general) thus contribute little to the validity argument. The major flaw is the problem of confounding: there are multiple plausible explanations for any observed between-group differences. The absence of hypothesized differences would suggest a serious flaw in the validity argument, but the confirmation of such differences adds little. As such, accurate known-groups discrimination may be necessary, but will never be sufficient, to support the validity of scores. This article elaborates on this and other problems with the known-groups comparison that limit its utility as a source of validity evidence.

Original languageEnglish (US)
Pages (from-to)829-834
Number of pages6
JournalAdvances in Health Sciences Education
Volume20
Issue number3
DOIs
StatePublished - Sep 27 2014

Fingerprint

expert
Education
Group
Physicians
Research
trainee
evidence
education
discrimination
physician
staff
Disease
ability

Keywords

  • Assessment
  • Data collection
  • Data interpretation, statistical
  • Evaluation
  • Medical education
  • Reliability
  • Validation Studies

ASJC Scopus subject areas

  • Medicine(all)
  • Education

Cite this

Much ado about differences : why expert-novice comparisons add little to the validity argument. / Cook, David Allan.

In: Advances in Health Sciences Education, Vol. 20, No. 3, 27.09.2014, p. 829-834.

Research output: Contribution to journalArticle

@article{b97fc05b633a4d3d90fb0debc1ce9b35,
title = "Much ado about differences: why expert-novice comparisons add little to the validity argument",
abstract = "One approach to validating assessment scores involves evaluating the ability of scores to discriminate among groups who differ in a specific characteristic, such as training status (in education) or disease state (in clinical applications). Such known-groups comparison studies provide validity evidence of “relationships with other variables.” The typical education research study might compare scores between staff physicians and postgraduate trainees with the hypothesis that those with more advanced training (the “experts”) will have higher scores than those less advanced (the “novices”). However, such comparisons are too nonspecific to support clear conclusions, and expert-novice comparisons (and known-groups comparisons in general) thus contribute little to the validity argument. The major flaw is the problem of confounding: there are multiple plausible explanations for any observed between-group differences. The absence of hypothesized differences would suggest a serious flaw in the validity argument, but the confirmation of such differences adds little. As such, accurate known-groups discrimination may be necessary, but will never be sufficient, to support the validity of scores. This article elaborates on this and other problems with the known-groups comparison that limit its utility as a source of validity evidence.",
keywords = "Assessment, Data collection, Data interpretation, statistical, Evaluation, Medical education, Reliability, Validation Studies",
author = "Cook, {David Allan}",
year = "2014",
month = "9",
day = "27",
doi = "10.1007/s10459-014-9551-3",
language = "English (US)",
volume = "20",
pages = "829--834",
journal = "Advances in Health Sciences Education",
issn = "1382-4996",
publisher = "Springer Netherlands",
number = "3",

}

TY - JOUR

T1 - Much ado about differences

T2 - why expert-novice comparisons add little to the validity argument

AU - Cook, David Allan

PY - 2014/9/27

Y1 - 2014/9/27

N2 - One approach to validating assessment scores involves evaluating the ability of scores to discriminate among groups who differ in a specific characteristic, such as training status (in education) or disease state (in clinical applications). Such known-groups comparison studies provide validity evidence of “relationships with other variables.” The typical education research study might compare scores between staff physicians and postgraduate trainees with the hypothesis that those with more advanced training (the “experts”) will have higher scores than those less advanced (the “novices”). However, such comparisons are too nonspecific to support clear conclusions, and expert-novice comparisons (and known-groups comparisons in general) thus contribute little to the validity argument. The major flaw is the problem of confounding: there are multiple plausible explanations for any observed between-group differences. The absence of hypothesized differences would suggest a serious flaw in the validity argument, but the confirmation of such differences adds little. As such, accurate known-groups discrimination may be necessary, but will never be sufficient, to support the validity of scores. This article elaborates on this and other problems with the known-groups comparison that limit its utility as a source of validity evidence.

AB - One approach to validating assessment scores involves evaluating the ability of scores to discriminate among groups who differ in a specific characteristic, such as training status (in education) or disease state (in clinical applications). Such known-groups comparison studies provide validity evidence of “relationships with other variables.” The typical education research study might compare scores between staff physicians and postgraduate trainees with the hypothesis that those with more advanced training (the “experts”) will have higher scores than those less advanced (the “novices”). However, such comparisons are too nonspecific to support clear conclusions, and expert-novice comparisons (and known-groups comparisons in general) thus contribute little to the validity argument. The major flaw is the problem of confounding: there are multiple plausible explanations for any observed between-group differences. The absence of hypothesized differences would suggest a serious flaw in the validity argument, but the confirmation of such differences adds little. As such, accurate known-groups discrimination may be necessary, but will never be sufficient, to support the validity of scores. This article elaborates on this and other problems with the known-groups comparison that limit its utility as a source of validity evidence.

KW - Assessment

KW - Data collection

KW - Data interpretation, statistical

KW - Evaluation

KW - Medical education

KW - Reliability

KW - Validation Studies

UR - http://www.scopus.com/inward/record.url?scp=84937635245&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84937635245&partnerID=8YFLogxK

U2 - 10.1007/s10459-014-9551-3

DO - 10.1007/s10459-014-9551-3

M3 - Article

C2 - 25260974

AN - SCOPUS:84937635245

VL - 20

SP - 829

EP - 834

JO - Advances in Health Sciences Education

JF - Advances in Health Sciences Education

SN - 1382-4996

IS - 3

ER -