29 research outputs found
Medical Knowledge-enriched Textual Entailment Framework
One of the cardinal tasks in achieving robust medical question answering systems is textual entailment. The existing approaches make use of an ensemble of pre-trained language models or data augmentation, often to clock higher numbers on the validation metrics. However, two major shortcomings impede higher success in identifying entailment: (1) understanding the focus/intent of the question and (2) ability to utilize the real-world background knowledge to capture the context beyond the sentence. In this paper, we present a novel Medical Knowledge-Enriched Textual Entailment framework that allows the model to acquire a semantic and global representation of the input medical text with the help of a relevant domain-specific knowledge graph. We evaluate our framework on the benchmark MEDIQA-RQE dataset and manifest that the use of knowledge enriched dual-encoding mechanism help in achieving an absolute improvement of 8.27% over SOTA language models. We have made the source code available here
IITP at MEDIQA 2019: Systems Report for Natural Language Inference, Question Entailment and Question Answering
This paper presents the experiments accomplished as a part of our
participation in the MEDIQA challenge, an (Abacha et al., 2019) shared task. We
participated in all the three tasks defined in this particular shared task. The
tasks are viz. i. Natural Language Inference (NLI) ii. Recognizing Question
Entailment(RQE) and their application in medical Question Answering (QA). We
submitted runs using multiple deep learning based systems (runs) for each of
these three tasks. We submitted five system results in each of the NLI and RQE
tasks, and four system results for the QA task. The systems yield encouraging
results in all three tasks. The highest performance obtained in NLI, RQE and QA
tasks are 81.8%, 53.2%, and 71.7%, respectively
Surf at MEDIQA 2019: Improving Performance of Natural Language Inference in the Clinical Domain by Adopting Pre-trained Language Model
While deep learning techniques have shown promising results in many natural
language processing (NLP) tasks, it has not been widely applied to the clinical
domain. The lack of large datasets and the pervasive use of domain-specific
language (i.e. abbreviations and acronyms) in the clinical domain causes slower
progress in NLP tasks than that of the general NLP tasks. To fill this gap, we
employ word/subword-level based models that adopt large-scale data-driven
methods such as pre-trained language models and transfer learning in analyzing
text for the clinical domain. Empirical results demonstrate the superiority of
the proposed methods by achieving 90.6% accuracy in medical domain natural
language inference task. Furthermore, we inspect the independent strengths of
the proposed approaches in quantitative and qualitative manners. This analysis
will help researchers to select necessary components in building models for the
medical domain.Comment: 9 pages, Accepted to ACL 2019 workshop on BioNL
UW-BHI at MEDIQA 2019: An Analysis of Representation Methods for Medical Natural Language Inference
Recent advances in distributed language modeling have led to large
performance increases on a variety of natural language processing (NLP) tasks.
However, it is not well understood how these methods may be augmented by
knowledge-based approaches. This paper compares the performance and internal
representation of an Enhanced Sequential Inference Model (ESIM) between three
experimental conditions based on the representation method: Bidirectional
Encoder Representations from Transformers (BERT), Embeddings of Semantic
Predications (ESP), or Cui2Vec. The methods were evaluated on the Medical
Natural Language Inference (MedNLI) subtask of the MEDIQA 2019 shared task.
This task relied heavily on semantic understanding and thus served as a
suitable evaluation set for the comparison of these representation methods
Probing Pre-Trained Language Models for Disease Knowledge
Pre-trained language models such as ClinicalBERT have achieved impressive
results on tasks such as medical Natural Language Inference. At first glance,
this may suggest that these models are able to perform medical reasoning tasks,
such as mapping symptoms to diseases. However, we find that standard benchmarks
such as MedNLI contain relatively few examples that require such forms of
reasoning. To better understand the medical reasoning capabilities of existing
language models, in this paper we introduce DisKnE, a new benchmark for Disease
Knowledge Evaluation. To construct this benchmark, we annotated each positive
MedNLI example with the types of medical reasoning that are needed. We then
created negative examples by corrupting these positive examples in an
adversarial way. Furthermore, we define training-test splits per disease,
ensuring that no knowledge about test diseases can be learned from the training
data, and we canonicalize the formulation of the hypotheses to avoid the
presence of artefacts. This leads to a number of binary classification
problems, one for each type of reasoning and each disease. When analysing
pre-trained models for the clinical/biomedical domain on the proposed
benchmark, we find that their performance drops considerably.Comment: Accepted by ACL 2021 Finding