Effects of Information Masking in the Task-Specific Finetuning of a Transformers-Based Clinical Question-Answering Framework

Sungrim Moon, Huan He, Jungwei W. Fan

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Transformers-based language models have achieved impressive performance in biomedical question-answering (QA). Our previous work led to surmise that such models could leverage frequent literal question-answer pairs to get the correct answers, casting doubt on true intelligence and transferability. Therefore, we conducted experiments by masking the anchor concept in the question and context documents during the fine-tuning stage of BERT for a reading comprehension QA task on clinical notes. The perturbation involved randomly replacing 0%, 10%, 20%, 30%, and 100% of the concept occurrences into a dummy string. We found the 100% masking harshly penalized the overall accuracy by about 0.10 versus 0% masking. However, the accuracy improved about 0.01 to 0.02 at 20% masking - and the benefit was able to transfer when tested on a different corpus. We also found the masking preferably enhanced the accuracy for question-answer pairs of the top 20%-40% frequent in the train set. The results suggested that transformers-based QA systems may benefit from moderate masking during fine-tuning, likely by forcing the model to learn abstract context patterns rather than relying on specific surface terms or relations. The beneficial effect skewed toward a specific non-top frequency tier could reflect a more general phenomenon in machine learning where such enhancement techniques are most effective for cases that sit around the make-or-fail border.

Original languageEnglish (US)
Title of host publicationProceedings - 2022 IEEE 10th International Conference on Healthcare Informatics, ICHI 2022
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages36-41
Number of pages6
ISBN (Electronic)9781665468459
DOIs
StatePublished - 2022
Event10th IEEE International Conference on Healthcare Informatics, ICHI 2022 - Rochester, United States
Duration: Jun 11 2022Jun 14 2022

Publication series

NameProceedings - 2022 IEEE 10th International Conference on Healthcare Informatics, ICHI 2022

Conference

Conference10th IEEE International Conference on Healthcare Informatics, ICHI 2022
Country/TerritoryUnited States
CityRochester
Period6/11/226/14/22

Keywords

  • Deep Learning
  • Electronic Health Records
  • Natural Language Processing
  • Question Answering
  • Supervised Machine Learning

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Science Applications
  • Information Systems and Management
  • Safety, Risk, Reliability and Quality
  • Health Informatics

Fingerprint

Dive into the research topics of 'Effects of Information Masking in the Task-Specific Finetuning of a Transformers-Based Clinical Question-Answering Framework'. Together they form a unique fingerprint.

Cite this