Level of intrateam agreement on candidate scores in a structured interview process

James W. Youdas, Connie L. Bogard, Vera Jean Suman

Research output: Contribution to journalArticle

6 Citations (Scopus)

Abstract

This study examined the intrateam reliability of a structured interview process used to help select students to a professional education program in physical therapy. Two hundred and twenty academically qualified applicants were interviewed by one of six two-member teams using a standardized format. Each member of a team independently rated the applicant on each of four performance skills using a 5-point ordinal scale; the maximal total score possible was 16 points. A weighted kappa (Kw) statistic was used to estimate intrateam agreement for both 1993 and 1994 on each performance skill and overall interview score. We observed wide variation in intrateam Kw scores for each of the four performance skills between the 1993 and 1994 admission cycles. According to Kw values for overall score agreement, five of the six interview teams either remained the same or increased at least one level between 1993 and 1994. Intrateam variability may be reduced by having team members review specific criteria used to rate a skill before the interview process commences each year.

Original languageEnglish (US)
Pages (from-to)303-305
Number of pages3
JournalJournal of Allied Health
Volume25
Issue number4
StatePublished - Sep 1996
Externally publishedYes

Fingerprint

Interviews
Professional Education
Students
Therapeutics

ASJC Scopus subject areas

  • Public Health, Environmental and Occupational Health

Cite this

Level of intrateam agreement on candidate scores in a structured interview process. / Youdas, James W.; Bogard, Connie L.; Suman, Vera Jean.

In: Journal of Allied Health, Vol. 25, No. 4, 09.1996, p. 303-305.

Research output: Contribution to journalArticle

@article{2cc150dbf8ff4b3895ab0cf092dd04b2,
title = "Level of intrateam agreement on candidate scores in a structured interview process",
abstract = "This study examined the intrateam reliability of a structured interview process used to help select students to a professional education program in physical therapy. Two hundred and twenty academically qualified applicants were interviewed by one of six two-member teams using a standardized format. Each member of a team independently rated the applicant on each of four performance skills using a 5-point ordinal scale; the maximal total score possible was 16 points. A weighted kappa (Kw) statistic was used to estimate intrateam agreement for both 1993 and 1994 on each performance skill and overall interview score. We observed wide variation in intrateam Kw scores for each of the four performance skills between the 1993 and 1994 admission cycles. According to Kw values for overall score agreement, five of the six interview teams either remained the same or increased at least one level between 1993 and 1994. Intrateam variability may be reduced by having team members review specific criteria used to rate a skill before the interview process commences each year.",
author = "Youdas, {James W.} and Bogard, {Connie L.} and Suman, {Vera Jean}",
year = "1996",
month = "9",
language = "English (US)",
volume = "25",
pages = "303--305",
journal = "Journal of Allied Health",
issn = "0090-7421",
publisher = "Science and Medicine Inc.",
number = "4",

}

TY - JOUR

T1 - Level of intrateam agreement on candidate scores in a structured interview process

AU - Youdas, James W.

AU - Bogard, Connie L.

AU - Suman, Vera Jean

PY - 1996/9

Y1 - 1996/9

N2 - This study examined the intrateam reliability of a structured interview process used to help select students to a professional education program in physical therapy. Two hundred and twenty academically qualified applicants were interviewed by one of six two-member teams using a standardized format. Each member of a team independently rated the applicant on each of four performance skills using a 5-point ordinal scale; the maximal total score possible was 16 points. A weighted kappa (Kw) statistic was used to estimate intrateam agreement for both 1993 and 1994 on each performance skill and overall interview score. We observed wide variation in intrateam Kw scores for each of the four performance skills between the 1993 and 1994 admission cycles. According to Kw values for overall score agreement, five of the six interview teams either remained the same or increased at least one level between 1993 and 1994. Intrateam variability may be reduced by having team members review specific criteria used to rate a skill before the interview process commences each year.

AB - This study examined the intrateam reliability of a structured interview process used to help select students to a professional education program in physical therapy. Two hundred and twenty academically qualified applicants were interviewed by one of six two-member teams using a standardized format. Each member of a team independently rated the applicant on each of four performance skills using a 5-point ordinal scale; the maximal total score possible was 16 points. A weighted kappa (Kw) statistic was used to estimate intrateam agreement for both 1993 and 1994 on each performance skill and overall interview score. We observed wide variation in intrateam Kw scores for each of the four performance skills between the 1993 and 1994 admission cycles. According to Kw values for overall score agreement, five of the six interview teams either remained the same or increased at least one level between 1993 and 1994. Intrateam variability may be reduced by having team members review specific criteria used to rate a skill before the interview process commences each year.

UR - http://www.scopus.com/inward/record.url?scp=0030225314&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0030225314&partnerID=8YFLogxK

M3 - Article

VL - 25

SP - 303

EP - 305

JO - Journal of Allied Health

JF - Journal of Allied Health

SN - 0090-7421

IS - 4

ER -