Script concordance testing: A review of published validity evidence

Stuart Lubarsky, Bernard Charlin, David Allan Cook, Colin Chalk, Cees P M van der Vleuten

Research output: Contribution to journalArticle

72 Citations (Scopus)

Abstract

Context: Script concordance test (SCT) scores are intended to reflect respondents' competence in interpreting clinical data under conditions of uncertainty. The validity of inferences based on SCT scores has not been rigorously established. Objectives This study was conducted in order to develop a structured validity argument for the interpretation of test scores derived through use of the script concordance method. Methods: We searched the PubMed, EMBASE and PsycINFO databases for articles pertaining to script concordance testing. We then reviewed these articles to evaluate the construct validity of the script concordance method, following an established approach for analysing validity data from five categories: content; response process; internal structure; relations to other variables, and consequences. Results: Content evidence derives from clear guidelines for the creation of authentic, ill-defined scenarios. High internal consistency reliability supports the internal structure of SCT scores. As might be expected, SCT scores correlate poorly with assessments of pure factual knowledge, in which correlations for more advanced learners are lower. The validity of SCT scores is weakly supported by evidence pertaining to examinee response processes and educational consequences. Conclusions: Published research generally supports the use of SCT to assess the interpretation of clinical data under conditions of uncertainty, although specifics of the validity argument vary and require verification in different contexts and for particular SCTs. Our review identifies potential areas of further validity inquiry in all five categories of evidence. In particular, future SCT research might explore the impact of the script concordance method on teaching and learning, and examine how SCTs integrate with other assessment methods within comprehensive assessment programmes.

Original languageEnglish (US)
Pages (from-to)329-338
Number of pages10
JournalMedical Education
Volume45
Issue number4
DOIs
StatePublished - Apr 2011

Fingerprint

Uncertainty
evidence
Research
PubMed
Mental Competency
Teaching
Learning
Databases
Guidelines
uncertainty
interpretation
construct validity
scenario
learning
Surveys and Questionnaires

ASJC Scopus subject areas

  • Medicine(all)
  • Education

Cite this

Script concordance testing : A review of published validity evidence. / Lubarsky, Stuart; Charlin, Bernard; Cook, David Allan; Chalk, Colin; van der Vleuten, Cees P M.

In: Medical Education, Vol. 45, No. 4, 04.2011, p. 329-338.

Research output: Contribution to journalArticle

Lubarsky, S, Charlin, B, Cook, DA, Chalk, C & van der Vleuten, CPM 2011, 'Script concordance testing: A review of published validity evidence', Medical Education, vol. 45, no. 4, pp. 329-338. https://doi.org/10.1111/j.1365-2923.2010.03863.x
Lubarsky, Stuart ; Charlin, Bernard ; Cook, David Allan ; Chalk, Colin ; van der Vleuten, Cees P M. / Script concordance testing : A review of published validity evidence. In: Medical Education. 2011 ; Vol. 45, No. 4. pp. 329-338.
@article{ea63ccadd543455da0314aab15d91b74,
title = "Script concordance testing: A review of published validity evidence",
abstract = "Context: Script concordance test (SCT) scores are intended to reflect respondents' competence in interpreting clinical data under conditions of uncertainty. The validity of inferences based on SCT scores has not been rigorously established. Objectives This study was conducted in order to develop a structured validity argument for the interpretation of test scores derived through use of the script concordance method. Methods: We searched the PubMed, EMBASE and PsycINFO databases for articles pertaining to script concordance testing. We then reviewed these articles to evaluate the construct validity of the script concordance method, following an established approach for analysing validity data from five categories: content; response process; internal structure; relations to other variables, and consequences. Results: Content evidence derives from clear guidelines for the creation of authentic, ill-defined scenarios. High internal consistency reliability supports the internal structure of SCT scores. As might be expected, SCT scores correlate poorly with assessments of pure factual knowledge, in which correlations for more advanced learners are lower. The validity of SCT scores is weakly supported by evidence pertaining to examinee response processes and educational consequences. Conclusions: Published research generally supports the use of SCT to assess the interpretation of clinical data under conditions of uncertainty, although specifics of the validity argument vary and require verification in different contexts and for particular SCTs. Our review identifies potential areas of further validity inquiry in all five categories of evidence. In particular, future SCT research might explore the impact of the script concordance method on teaching and learning, and examine how SCTs integrate with other assessment methods within comprehensive assessment programmes.",
author = "Stuart Lubarsky and Bernard Charlin and Cook, {David Allan} and Colin Chalk and {van der Vleuten}, {Cees P M}",
year = "2011",
month = "4",
doi = "10.1111/j.1365-2923.2010.03863.x",
language = "English (US)",
volume = "45",
pages = "329--338",
journal = "Medical Education",
issn = "0308-0110",
publisher = "Wiley-Blackwell",
number = "4",

}

TY - JOUR

T1 - Script concordance testing

T2 - A review of published validity evidence

AU - Lubarsky, Stuart

AU - Charlin, Bernard

AU - Cook, David Allan

AU - Chalk, Colin

AU - van der Vleuten, Cees P M

PY - 2011/4

Y1 - 2011/4

N2 - Context: Script concordance test (SCT) scores are intended to reflect respondents' competence in interpreting clinical data under conditions of uncertainty. The validity of inferences based on SCT scores has not been rigorously established. Objectives This study was conducted in order to develop a structured validity argument for the interpretation of test scores derived through use of the script concordance method. Methods: We searched the PubMed, EMBASE and PsycINFO databases for articles pertaining to script concordance testing. We then reviewed these articles to evaluate the construct validity of the script concordance method, following an established approach for analysing validity data from five categories: content; response process; internal structure; relations to other variables, and consequences. Results: Content evidence derives from clear guidelines for the creation of authentic, ill-defined scenarios. High internal consistency reliability supports the internal structure of SCT scores. As might be expected, SCT scores correlate poorly with assessments of pure factual knowledge, in which correlations for more advanced learners are lower. The validity of SCT scores is weakly supported by evidence pertaining to examinee response processes and educational consequences. Conclusions: Published research generally supports the use of SCT to assess the interpretation of clinical data under conditions of uncertainty, although specifics of the validity argument vary and require verification in different contexts and for particular SCTs. Our review identifies potential areas of further validity inquiry in all five categories of evidence. In particular, future SCT research might explore the impact of the script concordance method on teaching and learning, and examine how SCTs integrate with other assessment methods within comprehensive assessment programmes.

AB - Context: Script concordance test (SCT) scores are intended to reflect respondents' competence in interpreting clinical data under conditions of uncertainty. The validity of inferences based on SCT scores has not been rigorously established. Objectives This study was conducted in order to develop a structured validity argument for the interpretation of test scores derived through use of the script concordance method. Methods: We searched the PubMed, EMBASE and PsycINFO databases for articles pertaining to script concordance testing. We then reviewed these articles to evaluate the construct validity of the script concordance method, following an established approach for analysing validity data from five categories: content; response process; internal structure; relations to other variables, and consequences. Results: Content evidence derives from clear guidelines for the creation of authentic, ill-defined scenarios. High internal consistency reliability supports the internal structure of SCT scores. As might be expected, SCT scores correlate poorly with assessments of pure factual knowledge, in which correlations for more advanced learners are lower. The validity of SCT scores is weakly supported by evidence pertaining to examinee response processes and educational consequences. Conclusions: Published research generally supports the use of SCT to assess the interpretation of clinical data under conditions of uncertainty, although specifics of the validity argument vary and require verification in different contexts and for particular SCTs. Our review identifies potential areas of further validity inquiry in all five categories of evidence. In particular, future SCT research might explore the impact of the script concordance method on teaching and learning, and examine how SCTs integrate with other assessment methods within comprehensive assessment programmes.

UR - http://www.scopus.com/inward/record.url?scp=79952657744&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=79952657744&partnerID=8YFLogxK

U2 - 10.1111/j.1365-2923.2010.03863.x

DO - 10.1111/j.1365-2923.2010.03863.x

M3 - Article

C2 - 21401680

AN - SCOPUS:79952657744

VL - 45

SP - 329

EP - 338

JO - Medical Education

JF - Medical Education

SN - 0308-0110

IS - 4

ER -