Determining reliability of clinical assessment scores in real time

Thomas J. Beckman, Jayawant Mandrekar, Gregory J. Engstler, Robert D. Ficalora

Research output: Contribution to journalArticle

8 Citations (Scopus)

Abstract

Background: Assessment score reliability is usually based on a single analysis. However, reliability is an essential component of validity and assessment validation and revision is a never-ending cycle. For ongoing assessments over extended time frames, real-time reliability computations may alert users to possible changes in the learning environment that are revealed by variations in reliability over time. Purpose: To develop software that calculates the reliability of clinical assessments in real time. Methods: Over 2,400 assessment forms were analyzed. We developed software that calculates reliability in real time. Software accuracy was verified by comparing data from our software with a standard method. Factor analysis determined scale dimensionality. Results: Correlation between our software and a standard method was excellent (ICC for kappas = 0.97; Cronbach's alphas differed by < 0.03). Cron-bach's alpha ranged from 0.94 to 0.97 and weighted kappa ranged from 0.08 to 0.40. Factor analysis confirmed 3 teaching domains. Conclusions: We describe an accurate method for calculating reliability in real time. The benefit of real time computation is that it provides a mechanism for detecting possible changes (related to curriculum, teachers, and students) in the learning environment indicated by changes in reliability over time. This technique will enable investigators to monitor and detect changes in the reliability of assessment scores and, with future study, isolate aspects of the learning environment that impact on reliability.

Original languageEnglish (US)
Pages (from-to)188-194
Number of pages7
JournalTeaching and Learning in Medicine
Volume21
Issue number3
DOIs
StatePublished - Jul 2009

Fingerprint

Software
Learning
learning environment
Statistical Factor Analysis
factor analysis
time
Curriculum
Teaching
Research Personnel
Students
software
curriculum
teacher
student

ASJC Scopus subject areas

  • Medicine(all)
  • Education

Cite this

Determining reliability of clinical assessment scores in real time. / Beckman, Thomas J.; Mandrekar, Jayawant; Engstler, Gregory J.; Ficalora, Robert D.

In: Teaching and Learning in Medicine, Vol. 21, No. 3, 07.2009, p. 188-194.

Research output: Contribution to journalArticle

Beckman, Thomas J. ; Mandrekar, Jayawant ; Engstler, Gregory J. ; Ficalora, Robert D. / Determining reliability of clinical assessment scores in real time. In: Teaching and Learning in Medicine. 2009 ; Vol. 21, No. 3. pp. 188-194.
@article{676b58d390f24be8a20622602efa6849,
title = "Determining reliability of clinical assessment scores in real time",
abstract = "Background: Assessment score reliability is usually based on a single analysis. However, reliability is an essential component of validity and assessment validation and revision is a never-ending cycle. For ongoing assessments over extended time frames, real-time reliability computations may alert users to possible changes in the learning environment that are revealed by variations in reliability over time. Purpose: To develop software that calculates the reliability of clinical assessments in real time. Methods: Over 2,400 assessment forms were analyzed. We developed software that calculates reliability in real time. Software accuracy was verified by comparing data from our software with a standard method. Factor analysis determined scale dimensionality. Results: Correlation between our software and a standard method was excellent (ICC for kappas = 0.97; Cronbach's alphas differed by < 0.03). Cron-bach's alpha ranged from 0.94 to 0.97 and weighted kappa ranged from 0.08 to 0.40. Factor analysis confirmed 3 teaching domains. Conclusions: We describe an accurate method for calculating reliability in real time. The benefit of real time computation is that it provides a mechanism for detecting possible changes (related to curriculum, teachers, and students) in the learning environment indicated by changes in reliability over time. This technique will enable investigators to monitor and detect changes in the reliability of assessment scores and, with future study, isolate aspects of the learning environment that impact on reliability.",
author = "Beckman, {Thomas J.} and Jayawant Mandrekar and Engstler, {Gregory J.} and Ficalora, {Robert D.}",
year = "2009",
month = "7",
doi = "10.1080/10401330903014137",
language = "English (US)",
volume = "21",
pages = "188--194",
journal = "Teaching and Learning in Medicine",
issn = "1040-1334",
publisher = "Routledge",
number = "3",

}

TY - JOUR

T1 - Determining reliability of clinical assessment scores in real time

AU - Beckman, Thomas J.

AU - Mandrekar, Jayawant

AU - Engstler, Gregory J.

AU - Ficalora, Robert D.

PY - 2009/7

Y1 - 2009/7

N2 - Background: Assessment score reliability is usually based on a single analysis. However, reliability is an essential component of validity and assessment validation and revision is a never-ending cycle. For ongoing assessments over extended time frames, real-time reliability computations may alert users to possible changes in the learning environment that are revealed by variations in reliability over time. Purpose: To develop software that calculates the reliability of clinical assessments in real time. Methods: Over 2,400 assessment forms were analyzed. We developed software that calculates reliability in real time. Software accuracy was verified by comparing data from our software with a standard method. Factor analysis determined scale dimensionality. Results: Correlation between our software and a standard method was excellent (ICC for kappas = 0.97; Cronbach's alphas differed by < 0.03). Cron-bach's alpha ranged from 0.94 to 0.97 and weighted kappa ranged from 0.08 to 0.40. Factor analysis confirmed 3 teaching domains. Conclusions: We describe an accurate method for calculating reliability in real time. The benefit of real time computation is that it provides a mechanism for detecting possible changes (related to curriculum, teachers, and students) in the learning environment indicated by changes in reliability over time. This technique will enable investigators to monitor and detect changes in the reliability of assessment scores and, with future study, isolate aspects of the learning environment that impact on reliability.

AB - Background: Assessment score reliability is usually based on a single analysis. However, reliability is an essential component of validity and assessment validation and revision is a never-ending cycle. For ongoing assessments over extended time frames, real-time reliability computations may alert users to possible changes in the learning environment that are revealed by variations in reliability over time. Purpose: To develop software that calculates the reliability of clinical assessments in real time. Methods: Over 2,400 assessment forms were analyzed. We developed software that calculates reliability in real time. Software accuracy was verified by comparing data from our software with a standard method. Factor analysis determined scale dimensionality. Results: Correlation between our software and a standard method was excellent (ICC for kappas = 0.97; Cronbach's alphas differed by < 0.03). Cron-bach's alpha ranged from 0.94 to 0.97 and weighted kappa ranged from 0.08 to 0.40. Factor analysis confirmed 3 teaching domains. Conclusions: We describe an accurate method for calculating reliability in real time. The benefit of real time computation is that it provides a mechanism for detecting possible changes (related to curriculum, teachers, and students) in the learning environment indicated by changes in reliability over time. This technique will enable investigators to monitor and detect changes in the reliability of assessment scores and, with future study, isolate aspects of the learning environment that impact on reliability.

UR - http://www.scopus.com/inward/record.url?scp=70449591712&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=70449591712&partnerID=8YFLogxK

U2 - 10.1080/10401330903014137

DO - 10.1080/10401330903014137

M3 - Article

VL - 21

SP - 188

EP - 194

JO - Teaching and Learning in Medicine

JF - Teaching and Learning in Medicine

SN - 1040-1334

IS - 3

ER -