Towards assigning references using semantic, journal and citation relevance

Clinical knowledge systems (CKS) facilitate healthcare practitioners to make the best decisions at the point of care by having access to most recent knowledge. However, because of the overabundance of clinical research articles, it is time consuming to add new content manually to any CKS such as UpToDate and to ensure that it is consistent with evidence. For this reason, many CKS such as Mayo Clinic's AskMayoExpert use expert-based (and not evidence-based) knowledge learned from peer-to-peer communication. The main motivation of this work is to explore methods for assigning references automatically to expert-written content. The system employs information retrieval (IR) techniques to retrieve related sentences from MEDLINE abstracts as evidence candidates and then uses a machine learning model trained with physician's evaluation score on certain journals for re-ranking. Finally, the system utilizes citation counts of individual articles for further refining the ranking. A preliminary evaluation using 59 UpToDate sentences and respective citations as gold standard showed that the median ranking improved 11 folds after adding the journal and citation relevance metrics on the top of baseline that only uses the semantic relevance metric. The system seems promising and ready for trials in real use-case scenarios with experts.