1,785 research outputs found
Integrating a Heterogeneous Graph with Entity-aware Self-attention using Relative Position Labels for Reading Comprehension Model
Despite the significant progress made by transformer models in machine
reading comprehension tasks, they still fall short in handling complex
reasoning tasks due to the absence of explicit knowledge in the input sequence.
To address this limitation, many recent works have proposed injecting external
knowledge into the model. However, selecting relevant external knowledge,
ensuring its availability, and requiring additional processing steps remain
challenging. In this paper, we introduce a novel attention pattern that
integrates reasoning knowledge derived from a heterogeneous graph into the
transformer architecture without relying on external knowledge. The proposed
attention pattern comprises three key elements: global-local attention for word
tokens, graph attention for entity tokens that exhibit strong attention towards
tokens connected in the graph as opposed to those unconnected, and the
consideration of the type of relationship between each entity token and word
token. This results in optimized attention between the two if a relationship
exists. The pattern is coupled with special relative position labels, allowing
it to integrate with LUKE's entity-aware self-attention mechanism. The
experimental findings corroborate that our model outperforms both the
cutting-edge LUKE-Graph and the baseline LUKE model on the ReCoRD dataset that
focuses on commonsense reasoning.Comment: submitted for Knowledge-Based Systems Journa
Modular Approach to Machine Reading Comprehension: Mixture of Task-Aware Experts
In this work we present a Mixture of Task-Aware Experts Network for Machine
Reading Comprehension on a relatively small dataset. We particularly focus on
the issue of common-sense learning, enforcing the common ground knowledge by
specifically training different expert networks to capture different kinds of
relationships between each passage, question and choice triplet. Moreover, we
take inspi ration on the recent advancements of multitask and transfer learning
by training each network a relevant focused task. By making the
mixture-of-networks aware of a specific goal by enforcing a task and a
relationship, we achieve state-of-the-art results and reduce over-fitting
LUKE-Graph: A Transformer-based Approach with Gated Relational Graph Attention for Cloze-style Reading Comprehension
Incorporating prior knowledge can improve existing pre-training models in
cloze-style machine reading and has become a new trend in recent studies.
Notably, most of the existing models have integrated external knowledge graphs
(KG) and transformer-based models, such as BERT into a unified data structure.
However, selecting the most relevant ambiguous entities in KG and extracting
the best subgraph remains a challenge. In this paper, we propose the
LUKE-Graph, a model that builds a heterogeneous graph based on the intuitive
relationships between entities in a document without using any external KG. We
then use a Relational Graph Attention (RGAT) network to fuse the graph's
reasoning information and the contextual representation encoded by the
pre-trained LUKE model. In this way, we can take advantage of LUKE, to derive
an entity-aware representation; and a graph model - to exploit relation-aware
representation. Moreover, we propose Gated-RGAT by augmenting RGAT with a
gating mechanism that regulates the question information for the graph
convolution operation. This is very similar to human reasoning processing
because they always choose the best entity candidate based on the question
information. Experimental results demonstrate that the LUKE-Graph achieves
state-of-the-art performance on the ReCoRD dataset with commonsense reasoning.Comment: submitted for neurocomputing journa
- …