A quantitative assessment of a methodology for collaborative specification and evaluation of clinical guidelines

Erez Shalom, Yuval Shahar, Meirav Taieb-Maimon, Guy Bar, Avi Yarkoni, Ohad Young, Susana B. Martins, Laszlo Vaszar, Mary K. Goldstein, Yair Liel, Akiva Leibowitz, Tal Marom, Eitan Lunenfeld

Research output: Contribution to journalArticle

33 Citations (Scopus)

Abstract

We introduce a three-phase, nine-step methodology for specification of clinical guidelines (GLs) by expert physicians, clinical editors, and knowledge engineers and for quantitative evaluation of the specification's quality. We applied this methodology to a particular framework for incremental GL structuring (mark-up) and to GLs in three clinical domains. A gold-standard mark-up was created, including 196 plans and subplans, and 326 instances of ontological knowledge roles (KRs). A completeness measure of the acquired knowledge revealed that 97% of the plans and 91% of the KR instances of the GLs were recreated by the clinical editors. A correctness measure often revealed high variability within clinical editor pairs structuring each GL, but for all GLs and clinical editors the specification quality was significantly higher than random (p < 0.01). Procedural KRs were more difficult to mark-up than declarative KRs. We conclude that given an ontology-specific consensus, clinical editors with mark-up training can structure GL knowledge with high completeness, whereas the main demand for correct structuring is training in the ontology's semantics.

Original languageEnglish (US)
Pages (from-to)889-903
Number of pages15
JournalJournal of Biomedical Informatics
Volume41
Issue number6
DOIs
StatePublished - Dec 1 2008
Externally publishedYes

Fingerprint

Guidelines
Specifications
Ontology
Gold
Semantics
Engineers
Physicians

Keywords

  • Clinical decision support systems
  • Clinical guidelines
  • Completeness
  • Correctness
  • Evaluation
  • Knowledge acquisition
  • Knowledge bases
  • Mark-up
  • Ontologies
  • Structuring

ASJC Scopus subject areas

  • Computer Science Applications
  • Health Informatics

Cite this

A quantitative assessment of a methodology for collaborative specification and evaluation of clinical guidelines. / Shalom, Erez; Shahar, Yuval; Taieb-Maimon, Meirav; Bar, Guy; Yarkoni, Avi; Young, Ohad; Martins, Susana B.; Vaszar, Laszlo; Goldstein, Mary K.; Liel, Yair; Leibowitz, Akiva; Marom, Tal; Lunenfeld, Eitan.

In: Journal of Biomedical Informatics, Vol. 41, No. 6, 01.12.2008, p. 889-903.

Research output: Contribution to journalArticle

Shalom, E, Shahar, Y, Taieb-Maimon, M, Bar, G, Yarkoni, A, Young, O, Martins, SB, Vaszar, L, Goldstein, MK, Liel, Y, Leibowitz, A, Marom, T & Lunenfeld, E 2008, 'A quantitative assessment of a methodology for collaborative specification and evaluation of clinical guidelines', Journal of Biomedical Informatics, vol. 41, no. 6, pp. 889-903. https://doi.org/10.1016/j.jbi.2008.04.009
Shalom, Erez ; Shahar, Yuval ; Taieb-Maimon, Meirav ; Bar, Guy ; Yarkoni, Avi ; Young, Ohad ; Martins, Susana B. ; Vaszar, Laszlo ; Goldstein, Mary K. ; Liel, Yair ; Leibowitz, Akiva ; Marom, Tal ; Lunenfeld, Eitan. / A quantitative assessment of a methodology for collaborative specification and evaluation of clinical guidelines. In: Journal of Biomedical Informatics. 2008 ; Vol. 41, No. 6. pp. 889-903.
@article{a1b11b32f1414f609b19ec3668020dcb,
title = "A quantitative assessment of a methodology for collaborative specification and evaluation of clinical guidelines",
abstract = "We introduce a three-phase, nine-step methodology for specification of clinical guidelines (GLs) by expert physicians, clinical editors, and knowledge engineers and for quantitative evaluation of the specification's quality. We applied this methodology to a particular framework for incremental GL structuring (mark-up) and to GLs in three clinical domains. A gold-standard mark-up was created, including 196 plans and subplans, and 326 instances of ontological knowledge roles (KRs). A completeness measure of the acquired knowledge revealed that 97{\%} of the plans and 91{\%} of the KR instances of the GLs were recreated by the clinical editors. A correctness measure often revealed high variability within clinical editor pairs structuring each GL, but for all GLs and clinical editors the specification quality was significantly higher than random (p < 0.01). Procedural KRs were more difficult to mark-up than declarative KRs. We conclude that given an ontology-specific consensus, clinical editors with mark-up training can structure GL knowledge with high completeness, whereas the main demand for correct structuring is training in the ontology's semantics.",
keywords = "Clinical decision support systems, Clinical guidelines, Completeness, Correctness, Evaluation, Knowledge acquisition, Knowledge bases, Mark-up, Ontologies, Structuring",
author = "Erez Shalom and Yuval Shahar and Meirav Taieb-Maimon and Guy Bar and Avi Yarkoni and Ohad Young and Martins, {Susana B.} and Laszlo Vaszar and Goldstein, {Mary K.} and Yair Liel and Akiva Leibowitz and Tal Marom and Eitan Lunenfeld",
year = "2008",
month = "12",
day = "1",
doi = "10.1016/j.jbi.2008.04.009",
language = "English (US)",
volume = "41",
pages = "889--903",
journal = "Journal of Biomedical Informatics",
issn = "1532-0464",
publisher = "Academic Press Inc.",
number = "6",

}

TY - JOUR

T1 - A quantitative assessment of a methodology for collaborative specification and evaluation of clinical guidelines

AU - Shalom, Erez

AU - Shahar, Yuval

AU - Taieb-Maimon, Meirav

AU - Bar, Guy

AU - Yarkoni, Avi

AU - Young, Ohad

AU - Martins, Susana B.

AU - Vaszar, Laszlo

AU - Goldstein, Mary K.

AU - Liel, Yair

AU - Leibowitz, Akiva

AU - Marom, Tal

AU - Lunenfeld, Eitan

PY - 2008/12/1

Y1 - 2008/12/1

N2 - We introduce a three-phase, nine-step methodology for specification of clinical guidelines (GLs) by expert physicians, clinical editors, and knowledge engineers and for quantitative evaluation of the specification's quality. We applied this methodology to a particular framework for incremental GL structuring (mark-up) and to GLs in three clinical domains. A gold-standard mark-up was created, including 196 plans and subplans, and 326 instances of ontological knowledge roles (KRs). A completeness measure of the acquired knowledge revealed that 97% of the plans and 91% of the KR instances of the GLs were recreated by the clinical editors. A correctness measure often revealed high variability within clinical editor pairs structuring each GL, but for all GLs and clinical editors the specification quality was significantly higher than random (p < 0.01). Procedural KRs were more difficult to mark-up than declarative KRs. We conclude that given an ontology-specific consensus, clinical editors with mark-up training can structure GL knowledge with high completeness, whereas the main demand for correct structuring is training in the ontology's semantics.

AB - We introduce a three-phase, nine-step methodology for specification of clinical guidelines (GLs) by expert physicians, clinical editors, and knowledge engineers and for quantitative evaluation of the specification's quality. We applied this methodology to a particular framework for incremental GL structuring (mark-up) and to GLs in three clinical domains. A gold-standard mark-up was created, including 196 plans and subplans, and 326 instances of ontological knowledge roles (KRs). A completeness measure of the acquired knowledge revealed that 97% of the plans and 91% of the KR instances of the GLs were recreated by the clinical editors. A correctness measure often revealed high variability within clinical editor pairs structuring each GL, but for all GLs and clinical editors the specification quality was significantly higher than random (p < 0.01). Procedural KRs were more difficult to mark-up than declarative KRs. We conclude that given an ontology-specific consensus, clinical editors with mark-up training can structure GL knowledge with high completeness, whereas the main demand for correct structuring is training in the ontology's semantics.

KW - Clinical decision support systems

KW - Clinical guidelines

KW - Completeness

KW - Correctness

KW - Evaluation

KW - Knowledge acquisition

KW - Knowledge bases

KW - Mark-up

KW - Ontologies

KW - Structuring

UR - http://www.scopus.com/inward/record.url?scp=55549125176&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=55549125176&partnerID=8YFLogxK

U2 - 10.1016/j.jbi.2008.04.009

DO - 10.1016/j.jbi.2008.04.009

M3 - Article

VL - 41

SP - 889

EP - 903

JO - Journal of Biomedical Informatics

JF - Journal of Biomedical Informatics

SN - 1532-0464

IS - 6

ER -