Making work visible for electronic phenotype implementation: Lessons learned from the eMERGE network

Ning Shang, Cong Liu, Luke V. Rasmussen, Casey N. Ta, Robert J. Caroll, Barbara Benoit, Todd Lingren, Ozan Dikilitas, Frank D. Mentch, David S. Carrell, Wei Qi Wei, Yuan Luo, Vivian S. Gainer, Iftikhar J. Kullo, Jennifer A. Pacheco, Hakon Hakonarson, Theresa L. Walunas, Joshua C. Denny, Ken Wiley, Shawn N. MurphyGeorge Hripcsak, Chunhua Weng

Research output: Contribution to journalArticle

Abstract

Background: Implementation of phenotype algorithms requires phenotype engineers to interpret human-readable algorithms and translate the description (text and flowcharts) into computable phenotypes – a process that can be labor intensive and error prone. To address the critical need for reducing the implementation efforts, it is important to develop portable algorithms. Methods: We conducted a retrospective analysis of phenotype algorithms developed in the Electronic Medical Records and Genomics (eMERGE) network and identified common customization tasks required for implementation. A novel scoring system was developed to quantify portability from three aspects: Knowledge conversion, clause Interpretation, and Programming (KIP). Tasks were grouped into twenty representative categories. Experienced phenotype engineers were asked to estimate the average time spent on each category and evaluate time saving enabled by a common data model (CDM), specifically the Observational Medical Outcomes Partnership (OMOP) model, for each category. Results: A total of 485 distinct clauses (phenotype criteria) were identified from 55 phenotype algorithms, corresponding to 1153 customization tasks. In addition to 25 non-phenotype-specific tasks, 46 tasks are related to interpretation, 613 tasks are related to knowledge conversion, and 469 tasks are related to programming. A score between 0 and 2 (0 for easy, 1 for moderate, and 2 for difficult portability) is assigned for each aspect, yielding a total KIP score range of 0 to 6. The average clause-wise KIP score to reflect portability is 1.37 ± 1.38. Specifically, the average knowledge (K) score is 0.64 ± 0.66, interpretation (I) score is 0.33 ± 0.55, and programming (P) score is 0.40 ± 0.64. 5% of the categories can be completed within one hour (median). 70% of the categories take from days to months to complete. The OMOP model can assist with vocabulary mapping tasks. Conclusion: This study presents firsthand knowledge of the substantial implementation efforts in phenotyping and introduces a novel metric (KIP) to measure portability of phenotype algorithms for quantifying such efforts across the eMERGE Network. Phenotype developers are encouraged to analyze and optimize the portability in regards to knowledge, interpretation and programming. CDMs can be used to improve the portability for some ‘knowledge-oriented’ tasks.

Original languageEnglish (US)
Article number103293
JournalJournal of Biomedical Informatics
Volume99
DOIs
StatePublished - Nov 2019

Fingerprint

Electronic medical equipment
Electronic Health Records
Genomics
Phenotype
Engineers
Software Design
Data structures
Vocabulary
Personnel

Keywords

  • Electronic health records
  • Phenotyping
  • Portability

ASJC Scopus subject areas

  • Computer Science Applications
  • Health Informatics

Cite this

Making work visible for electronic phenotype implementation : Lessons learned from the eMERGE network. / Shang, Ning; Liu, Cong; Rasmussen, Luke V.; Ta, Casey N.; Caroll, Robert J.; Benoit, Barbara; Lingren, Todd; Dikilitas, Ozan; Mentch, Frank D.; Carrell, David S.; Wei, Wei Qi; Luo, Yuan; Gainer, Vivian S.; Kullo, Iftikhar J.; Pacheco, Jennifer A.; Hakonarson, Hakon; Walunas, Theresa L.; Denny, Joshua C.; Wiley, Ken; Murphy, Shawn N.; Hripcsak, George; Weng, Chunhua.

In: Journal of Biomedical Informatics, Vol. 99, 103293, 11.2019.

Research output: Contribution to journalArticle

Shang, N, Liu, C, Rasmussen, LV, Ta, CN, Caroll, RJ, Benoit, B, Lingren, T, Dikilitas, O, Mentch, FD, Carrell, DS, Wei, WQ, Luo, Y, Gainer, VS, Kullo, IJ, Pacheco, JA, Hakonarson, H, Walunas, TL, Denny, JC, Wiley, K, Murphy, SN, Hripcsak, G & Weng, C 2019, 'Making work visible for electronic phenotype implementation: Lessons learned from the eMERGE network', Journal of Biomedical Informatics, vol. 99, 103293. https://doi.org/10.1016/j.jbi.2019.103293
Shang, Ning ; Liu, Cong ; Rasmussen, Luke V. ; Ta, Casey N. ; Caroll, Robert J. ; Benoit, Barbara ; Lingren, Todd ; Dikilitas, Ozan ; Mentch, Frank D. ; Carrell, David S. ; Wei, Wei Qi ; Luo, Yuan ; Gainer, Vivian S. ; Kullo, Iftikhar J. ; Pacheco, Jennifer A. ; Hakonarson, Hakon ; Walunas, Theresa L. ; Denny, Joshua C. ; Wiley, Ken ; Murphy, Shawn N. ; Hripcsak, George ; Weng, Chunhua. / Making work visible for electronic phenotype implementation : Lessons learned from the eMERGE network. In: Journal of Biomedical Informatics. 2019 ; Vol. 99.
@article{03d31275a22141859c664781aaff4e3b,
title = "Making work visible for electronic phenotype implementation: Lessons learned from the eMERGE network",
abstract = "Background: Implementation of phenotype algorithms requires phenotype engineers to interpret human-readable algorithms and translate the description (text and flowcharts) into computable phenotypes – a process that can be labor intensive and error prone. To address the critical need for reducing the implementation efforts, it is important to develop portable algorithms. Methods: We conducted a retrospective analysis of phenotype algorithms developed in the Electronic Medical Records and Genomics (eMERGE) network and identified common customization tasks required for implementation. A novel scoring system was developed to quantify portability from three aspects: Knowledge conversion, clause Interpretation, and Programming (KIP). Tasks were grouped into twenty representative categories. Experienced phenotype engineers were asked to estimate the average time spent on each category and evaluate time saving enabled by a common data model (CDM), specifically the Observational Medical Outcomes Partnership (OMOP) model, for each category. Results: A total of 485 distinct clauses (phenotype criteria) were identified from 55 phenotype algorithms, corresponding to 1153 customization tasks. In addition to 25 non-phenotype-specific tasks, 46 tasks are related to interpretation, 613 tasks are related to knowledge conversion, and 469 tasks are related to programming. A score between 0 and 2 (0 for easy, 1 for moderate, and 2 for difficult portability) is assigned for each aspect, yielding a total KIP score range of 0 to 6. The average clause-wise KIP score to reflect portability is 1.37 ± 1.38. Specifically, the average knowledge (K) score is 0.64 ± 0.66, interpretation (I) score is 0.33 ± 0.55, and programming (P) score is 0.40 ± 0.64. 5{\%} of the categories can be completed within one hour (median). 70{\%} of the categories take from days to months to complete. The OMOP model can assist with vocabulary mapping tasks. Conclusion: This study presents firsthand knowledge of the substantial implementation efforts in phenotyping and introduces a novel metric (KIP) to measure portability of phenotype algorithms for quantifying such efforts across the eMERGE Network. Phenotype developers are encouraged to analyze and optimize the portability in regards to knowledge, interpretation and programming. CDMs can be used to improve the portability for some ‘knowledge-oriented’ tasks.",
keywords = "Electronic health records, Phenotyping, Portability",
author = "Ning Shang and Cong Liu and Rasmussen, {Luke V.} and Ta, {Casey N.} and Caroll, {Robert J.} and Barbara Benoit and Todd Lingren and Ozan Dikilitas and Mentch, {Frank D.} and Carrell, {David S.} and Wei, {Wei Qi} and Yuan Luo and Gainer, {Vivian S.} and Kullo, {Iftikhar J.} and Pacheco, {Jennifer A.} and Hakon Hakonarson and Walunas, {Theresa L.} and Denny, {Joshua C.} and Ken Wiley and Murphy, {Shawn N.} and George Hripcsak and Chunhua Weng",
year = "2019",
month = "11",
doi = "10.1016/j.jbi.2019.103293",
language = "English (US)",
volume = "99",
journal = "Journal of Biomedical Informatics",
issn = "1532-0464",
publisher = "Academic Press Inc.",

}

TY - JOUR

T1 - Making work visible for electronic phenotype implementation

T2 - Lessons learned from the eMERGE network

AU - Shang, Ning

AU - Liu, Cong

AU - Rasmussen, Luke V.

AU - Ta, Casey N.

AU - Caroll, Robert J.

AU - Benoit, Barbara

AU - Lingren, Todd

AU - Dikilitas, Ozan

AU - Mentch, Frank D.

AU - Carrell, David S.

AU - Wei, Wei Qi

AU - Luo, Yuan

AU - Gainer, Vivian S.

AU - Kullo, Iftikhar J.

AU - Pacheco, Jennifer A.

AU - Hakonarson, Hakon

AU - Walunas, Theresa L.

AU - Denny, Joshua C.

AU - Wiley, Ken

AU - Murphy, Shawn N.

AU - Hripcsak, George

AU - Weng, Chunhua

PY - 2019/11

Y1 - 2019/11

N2 - Background: Implementation of phenotype algorithms requires phenotype engineers to interpret human-readable algorithms and translate the description (text and flowcharts) into computable phenotypes – a process that can be labor intensive and error prone. To address the critical need for reducing the implementation efforts, it is important to develop portable algorithms. Methods: We conducted a retrospective analysis of phenotype algorithms developed in the Electronic Medical Records and Genomics (eMERGE) network and identified common customization tasks required for implementation. A novel scoring system was developed to quantify portability from three aspects: Knowledge conversion, clause Interpretation, and Programming (KIP). Tasks were grouped into twenty representative categories. Experienced phenotype engineers were asked to estimate the average time spent on each category and evaluate time saving enabled by a common data model (CDM), specifically the Observational Medical Outcomes Partnership (OMOP) model, for each category. Results: A total of 485 distinct clauses (phenotype criteria) were identified from 55 phenotype algorithms, corresponding to 1153 customization tasks. In addition to 25 non-phenotype-specific tasks, 46 tasks are related to interpretation, 613 tasks are related to knowledge conversion, and 469 tasks are related to programming. A score between 0 and 2 (0 for easy, 1 for moderate, and 2 for difficult portability) is assigned for each aspect, yielding a total KIP score range of 0 to 6. The average clause-wise KIP score to reflect portability is 1.37 ± 1.38. Specifically, the average knowledge (K) score is 0.64 ± 0.66, interpretation (I) score is 0.33 ± 0.55, and programming (P) score is 0.40 ± 0.64. 5% of the categories can be completed within one hour (median). 70% of the categories take from days to months to complete. The OMOP model can assist with vocabulary mapping tasks. Conclusion: This study presents firsthand knowledge of the substantial implementation efforts in phenotyping and introduces a novel metric (KIP) to measure portability of phenotype algorithms for quantifying such efforts across the eMERGE Network. Phenotype developers are encouraged to analyze and optimize the portability in regards to knowledge, interpretation and programming. CDMs can be used to improve the portability for some ‘knowledge-oriented’ tasks.

AB - Background: Implementation of phenotype algorithms requires phenotype engineers to interpret human-readable algorithms and translate the description (text and flowcharts) into computable phenotypes – a process that can be labor intensive and error prone. To address the critical need for reducing the implementation efforts, it is important to develop portable algorithms. Methods: We conducted a retrospective analysis of phenotype algorithms developed in the Electronic Medical Records and Genomics (eMERGE) network and identified common customization tasks required for implementation. A novel scoring system was developed to quantify portability from three aspects: Knowledge conversion, clause Interpretation, and Programming (KIP). Tasks were grouped into twenty representative categories. Experienced phenotype engineers were asked to estimate the average time spent on each category and evaluate time saving enabled by a common data model (CDM), specifically the Observational Medical Outcomes Partnership (OMOP) model, for each category. Results: A total of 485 distinct clauses (phenotype criteria) were identified from 55 phenotype algorithms, corresponding to 1153 customization tasks. In addition to 25 non-phenotype-specific tasks, 46 tasks are related to interpretation, 613 tasks are related to knowledge conversion, and 469 tasks are related to programming. A score between 0 and 2 (0 for easy, 1 for moderate, and 2 for difficult portability) is assigned for each aspect, yielding a total KIP score range of 0 to 6. The average clause-wise KIP score to reflect portability is 1.37 ± 1.38. Specifically, the average knowledge (K) score is 0.64 ± 0.66, interpretation (I) score is 0.33 ± 0.55, and programming (P) score is 0.40 ± 0.64. 5% of the categories can be completed within one hour (median). 70% of the categories take from days to months to complete. The OMOP model can assist with vocabulary mapping tasks. Conclusion: This study presents firsthand knowledge of the substantial implementation efforts in phenotyping and introduces a novel metric (KIP) to measure portability of phenotype algorithms for quantifying such efforts across the eMERGE Network. Phenotype developers are encouraged to analyze and optimize the portability in regards to knowledge, interpretation and programming. CDMs can be used to improve the portability for some ‘knowledge-oriented’ tasks.

KW - Electronic health records

KW - Phenotyping

KW - Portability

UR - http://www.scopus.com/inward/record.url?scp=85072694762&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85072694762&partnerID=8YFLogxK

U2 - 10.1016/j.jbi.2019.103293

DO - 10.1016/j.jbi.2019.103293

M3 - Article

C2 - 31542521

AN - SCOPUS:85072694762

VL - 99

JO - Journal of Biomedical Informatics

JF - Journal of Biomedical Informatics

SN - 1532-0464

M1 - 103293

ER -