305 research outputs found
LUKE-Graph: A Transformer-based Approach with Gated Relational Graph Attention for Cloze-style Reading Comprehension
Incorporating prior knowledge can improve existing pre-training models in
cloze-style machine reading and has become a new trend in recent studies.
Notably, most of the existing models have integrated external knowledge graphs
(KG) and transformer-based models, such as BERT into a unified data structure.
However, selecting the most relevant ambiguous entities in KG and extracting
the best subgraph remains a challenge. In this paper, we propose the
LUKE-Graph, a model that builds a heterogeneous graph based on the intuitive
relationships between entities in a document without using any external KG. We
then use a Relational Graph Attention (RGAT) network to fuse the graph's
reasoning information and the contextual representation encoded by the
pre-trained LUKE model. In this way, we can take advantage of LUKE, to derive
an entity-aware representation; and a graph model - to exploit relation-aware
representation. Moreover, we propose Gated-RGAT by augmenting RGAT with a
gating mechanism that regulates the question information for the graph
convolution operation. This is very similar to human reasoning processing
because they always choose the best entity candidate based on the question
information. Experimental results demonstrate that the LUKE-Graph achieves
state-of-the-art performance on the ReCoRD dataset with commonsense reasoning.Comment: submitted for neurocomputing journa
REM-Net: Recursive Erasure Memory Network for Commonsense Evidence Refinement
When answering a question, people often draw upon their rich world knowledge
in addition to the particular context. While recent works retrieve supporting
facts/evidence from commonsense knowledge bases to supply additional
information to each question, there is still ample opportunity to advance it on
the quality of the evidence. It is crucial since the quality of the evidence is
the key to answering commonsense questions, and even determines the upper bound
on the QA systems performance. In this paper, we propose a recursive erasure
memory network (REM-Net) to cope with the quality improvement of evidence. To
address this, REM-Net is equipped with a module to refine the evidence by
recursively erasing the low-quality evidence that does not explain the question
answering. Besides, instead of retrieving evidence from existing knowledge
bases, REM-Net leverages a pre-trained generative model to generate candidate
evidence customized for the question. We conduct experiments on two commonsense
question answering datasets, WIQA and CosmosQA. The results demonstrate the
performance of REM-Net and show that the refined evidence is explainable.Comment: Accepted by AAAI 202
- …