379 research outputs found
Hierarchical Interaction Networks with Rethinking Mechanism for Document-level Sentiment Analysis
Document-level Sentiment Analysis (DSA) is more challenging due to vague
semantic links and complicate sentiment information. Recent works have been
devoted to leveraging text summarization and have achieved promising results.
However, these summarization-based methods did not take full advantage of the
summary including ignoring the inherent interactions between the summary and
document. As a result, they limited the representation to express major points
in the document, which is highly indicative of the key sentiment. In this
paper, we study how to effectively generate a discriminative representation
with explicit subject patterns and sentiment contexts for DSA. A Hierarchical
Interaction Networks (HIN) is proposed to explore bidirectional interactions
between the summary and document at multiple granularities and learn
subject-oriented document representations for sentiment classification.
Furthermore, we design a Sentiment-based Rethinking mechanism (SR) by refining
the HIN with sentiment label information to learn a more sentiment-aware
document representation. We extensively evaluate our proposed models on three
public datasets. The experimental results consistently demonstrate the
effectiveness of our proposed models and show that HIN-SR outperforms various
state-of-the-art methods.Comment: 17 pages, accepted by ECML-PKDD 202
Tensorized Self-Attention: Efficiently Modeling Pairwise and Global Dependencies Together
Neural networks equipped with self-attention have parallelizable computation,
light-weight structure, and the ability to capture both long-range and local
dependencies. Further, their expressive power and performance can be boosted by
using a vector to measure pairwise dependency, but this requires to expand the
alignment matrix to a tensor, which results in memory and computation
bottlenecks. In this paper, we propose a novel attention mechanism called
"Multi-mask Tensorized Self-Attention" (MTSA), which is as fast and as
memory-efficient as a CNN, but significantly outperforms previous
CNN-/RNN-/attention-based models. MTSA 1) captures both pairwise (token2token)
and global (source2token) dependencies by a novel compatibility function
composed of dot-product and additive attentions, 2) uses a tensor to represent
the feature-wise alignment scores for better expressive power but only requires
parallelizable matrix multiplications, and 3) combines multi-head with
multi-dimensional attentions, and applies a distinct positional mask to each
head (subspace), so the memory and computation can be distributed to multiple
heads, each with sequential information encoded independently. The experiments
show that a CNN/RNN-free model based on MTSA achieves state-of-the-art or
competitive performance on nine NLP benchmarks with compelling memory- and
time-efficiency
Neural Natural Language Processing for Long Texts: A Survey of the State-of-the-Art
The adoption of Deep Neural Networks (DNNs) has greatly benefited Natural
Language Processing (NLP) during the past decade. However, the demands of long
document analysis are quite different from those of shorter texts, while the
ever increasing size of documents uploaded on-line renders automated
understanding of long texts a critical area of research. This article has two
goals: a) it overviews the relevant neural building blocks, thus serving as a
short tutorial, and b) it surveys the state-of-the-art in long document NLP,
mainly focusing on two central tasks: document classification and document
summarization. Sentiment analysis for long texts is also covered, since it is
typically treated as a particular case of document classification.
Additionally, this article discusses the main challenges, issues and current
solutions related to long document NLP. Finally, the relevant, publicly
available, annotated datasets are presented, in order to facilitate further
research.Comment: 53 pages, 2 figures, 171 citation
- …