A unified approach for assessing agreement for continuous and categorical data

Lawrence Lin, A. S. Hedayat, Wenting Wu

Research output: Contribution to journalArticle

65 Citations (Scopus)

Abstract

This paper proposes several Concordance Correlation Coefficient (CCC) indices to measure the agreement among k raters, with each rater having multiple (m) readings from each of the n subjects for continuous and categorical data. In addition, for normal data, this paper also proposes the coverage probability (CP) and total deviation index (TDI). Those indices are used to measure intra, inter and total agreement among all raters. Intra-rater indices are used to measure the agreement among the multiple readings from the same rater. Inter-rater indices are used to measure the agreement among different raters based on the average of multiple readings. Total-rater indices are used to measure the agreement among different raters based on individual readings. In addition to the agreement, the paper also assess intra, inter, and total precision and accuracy. Through a two-way mixed model, all CCC, precision and accuracy, TDI, and CP indices are expressed as functions of variance components, and GEE method is used to obtain the estimates and perform inferences for all the functions of variance components. Each of previous proposed approaches for assessing agreement becomes one of the special case of the proposed approach. For continuous data, when m approaches ∞, the proposed estimates reduce to the agreement indices proposed by Barnhart et al. (2005). When m = 1, the proposed estimate reduces to the ICC proposed by Carrasco and Jover (2003). When m = 1, the proposed estimate also reduces to the OCCC proposed by Lin (1989), King and Chinchilli (2001a) and Barnhart et al. (2002). When m = 1 and k = 2, the proposed estimate reduces to the original CCC proposed by Lin (1989). For categorical data, when k = 2 and m = 1, the proposed estimate and its associated inference reduce to the kappa for binary data and weighted kappa with squared weight for ordinal data.

Original languageEnglish (US)
Pages (from-to)629-652
Number of pages24
JournalJournal of Biopharmaceutical Statistics
Volume17
Issue number4
DOIs
StatePublished - Jul 2007

Fingerprint

Nominal or categorical data
Reading
Concordance
Correlation coefficient
Estimate
Components of Variance
Coverage Probability
Deviation
Weights and Measures
Ordinal Data
Binary Data
Mixed Model

Keywords

  • Accuracy
  • CCC
  • CP
  • ICC
  • Inter-agreement
  • Intra-agreement
  • Kappa
  • MSD
  • Precision
  • TDI
  • Total-agreement

ASJC Scopus subject areas

  • Pharmacology (medical)
  • Pharmacology, Toxicology and Pharmaceutics(all)

Cite this

A unified approach for assessing agreement for continuous and categorical data. / Lin, Lawrence; Hedayat, A. S.; Wu, Wenting.

In: Journal of Biopharmaceutical Statistics, Vol. 17, No. 4, 07.2007, p. 629-652.

Research output: Contribution to journalArticle

Lin, Lawrence ; Hedayat, A. S. ; Wu, Wenting. / A unified approach for assessing agreement for continuous and categorical data. In: Journal of Biopharmaceutical Statistics. 2007 ; Vol. 17, No. 4. pp. 629-652.
@article{217f3546eeae4868b216d4c4addfc4cc,
title = "A unified approach for assessing agreement for continuous and categorical data",
abstract = "This paper proposes several Concordance Correlation Coefficient (CCC) indices to measure the agreement among k raters, with each rater having multiple (m) readings from each of the n subjects for continuous and categorical data. In addition, for normal data, this paper also proposes the coverage probability (CP) and total deviation index (TDI). Those indices are used to measure intra, inter and total agreement among all raters. Intra-rater indices are used to measure the agreement among the multiple readings from the same rater. Inter-rater indices are used to measure the agreement among different raters based on the average of multiple readings. Total-rater indices are used to measure the agreement among different raters based on individual readings. In addition to the agreement, the paper also assess intra, inter, and total precision and accuracy. Through a two-way mixed model, all CCC, precision and accuracy, TDI, and CP indices are expressed as functions of variance components, and GEE method is used to obtain the estimates and perform inferences for all the functions of variance components. Each of previous proposed approaches for assessing agreement becomes one of the special case of the proposed approach. For continuous data, when m approaches ∞, the proposed estimates reduce to the agreement indices proposed by Barnhart et al. (2005). When m = 1, the proposed estimate reduces to the ICC proposed by Carrasco and Jover (2003). When m = 1, the proposed estimate also reduces to the OCCC proposed by Lin (1989), King and Chinchilli (2001a) and Barnhart et al. (2002). When m = 1 and k = 2, the proposed estimate reduces to the original CCC proposed by Lin (1989). For categorical data, when k = 2 and m = 1, the proposed estimate and its associated inference reduce to the kappa for binary data and weighted kappa with squared weight for ordinal data.",
keywords = "Accuracy, CCC, CP, ICC, Inter-agreement, Intra-agreement, Kappa, MSD, Precision, TDI, Total-agreement",
author = "Lawrence Lin and Hedayat, {A. S.} and Wenting Wu",
year = "2007",
month = "7",
doi = "10.1080/10543400701376498",
language = "English (US)",
volume = "17",
pages = "629--652",
journal = "Journal of Biopharmaceutical Statistics",
issn = "1054-3406",
publisher = "Taylor and Francis Ltd.",
number = "4",

}

TY - JOUR

T1 - A unified approach for assessing agreement for continuous and categorical data

AU - Lin, Lawrence

AU - Hedayat, A. S.

AU - Wu, Wenting

PY - 2007/7

Y1 - 2007/7

N2 - This paper proposes several Concordance Correlation Coefficient (CCC) indices to measure the agreement among k raters, with each rater having multiple (m) readings from each of the n subjects for continuous and categorical data. In addition, for normal data, this paper also proposes the coverage probability (CP) and total deviation index (TDI). Those indices are used to measure intra, inter and total agreement among all raters. Intra-rater indices are used to measure the agreement among the multiple readings from the same rater. Inter-rater indices are used to measure the agreement among different raters based on the average of multiple readings. Total-rater indices are used to measure the agreement among different raters based on individual readings. In addition to the agreement, the paper also assess intra, inter, and total precision and accuracy. Through a two-way mixed model, all CCC, precision and accuracy, TDI, and CP indices are expressed as functions of variance components, and GEE method is used to obtain the estimates and perform inferences for all the functions of variance components. Each of previous proposed approaches for assessing agreement becomes one of the special case of the proposed approach. For continuous data, when m approaches ∞, the proposed estimates reduce to the agreement indices proposed by Barnhart et al. (2005). When m = 1, the proposed estimate reduces to the ICC proposed by Carrasco and Jover (2003). When m = 1, the proposed estimate also reduces to the OCCC proposed by Lin (1989), King and Chinchilli (2001a) and Barnhart et al. (2002). When m = 1 and k = 2, the proposed estimate reduces to the original CCC proposed by Lin (1989). For categorical data, when k = 2 and m = 1, the proposed estimate and its associated inference reduce to the kappa for binary data and weighted kappa with squared weight for ordinal data.

AB - This paper proposes several Concordance Correlation Coefficient (CCC) indices to measure the agreement among k raters, with each rater having multiple (m) readings from each of the n subjects for continuous and categorical data. In addition, for normal data, this paper also proposes the coverage probability (CP) and total deviation index (TDI). Those indices are used to measure intra, inter and total agreement among all raters. Intra-rater indices are used to measure the agreement among the multiple readings from the same rater. Inter-rater indices are used to measure the agreement among different raters based on the average of multiple readings. Total-rater indices are used to measure the agreement among different raters based on individual readings. In addition to the agreement, the paper also assess intra, inter, and total precision and accuracy. Through a two-way mixed model, all CCC, precision and accuracy, TDI, and CP indices are expressed as functions of variance components, and GEE method is used to obtain the estimates and perform inferences for all the functions of variance components. Each of previous proposed approaches for assessing agreement becomes one of the special case of the proposed approach. For continuous data, when m approaches ∞, the proposed estimates reduce to the agreement indices proposed by Barnhart et al. (2005). When m = 1, the proposed estimate reduces to the ICC proposed by Carrasco and Jover (2003). When m = 1, the proposed estimate also reduces to the OCCC proposed by Lin (1989), King and Chinchilli (2001a) and Barnhart et al. (2002). When m = 1 and k = 2, the proposed estimate reduces to the original CCC proposed by Lin (1989). For categorical data, when k = 2 and m = 1, the proposed estimate and its associated inference reduce to the kappa for binary data and weighted kappa with squared weight for ordinal data.

KW - Accuracy

KW - CCC

KW - CP

KW - ICC

KW - Inter-agreement

KW - Intra-agreement

KW - Kappa

KW - MSD

KW - Precision

KW - TDI

KW - Total-agreement

UR - http://www.scopus.com/inward/record.url?scp=34347396549&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=34347396549&partnerID=8YFLogxK

U2 - 10.1080/10543400701376498

DO - 10.1080/10543400701376498

M3 - Article

VL - 17

SP - 629

EP - 652

JO - Journal of Biopharmaceutical Statistics

JF - Journal of Biopharmaceutical Statistics

SN - 1054-3406

IS - 4

ER -