1,246 research outputs found
On the Robustness of Aspect-based Sentiment Analysis: Rethinking Model, Data, and Training
Aspect-based sentiment analysis (ABSA) aims at automatically inferring the
specific sentiment polarities toward certain aspects of products or services
behind the social media texts or reviews, which has been a fundamental
application to the real-world society. Since the early 2010s, ABSA has achieved
extraordinarily high accuracy with various deep neural models. However,
existing ABSA models with strong in-house performances may fail to generalize
to some challenging cases where the contexts are variable, i.e., low robustness
to real-world environments. In this study, we propose to enhance the ABSA
robustness by systematically rethinking the bottlenecks from all possible
angles, including model, data, and training. First, we strengthen the current
best-robust syntax-aware models by further incorporating the rich external
syntactic dependencies and the labels with aspect simultaneously with a
universal-syntax graph convolutional network. In the corpus perspective, we
propose to automatically induce high-quality synthetic training data with
various types, allowing models to learn sufficient inductive bias for better
robustness. Last, we based on the rich pseudo data perform adversarial training
to enhance the resistance to the context perturbation and meanwhile employ
contrastive learning to reinforce the representations of instances with
contrastive sentiments. Extensive robustness evaluations are conducted. The
results demonstrate that our enhanced syntax-aware model achieves better
robustness performances than all the state-of-the-art baselines. By
additionally incorporating our synthetic corpus, the robust testing results are
pushed with around 10% accuracy, which are then further improved by installing
the advanced training strategies. In-depth analyses are presented for revealing
the factors influencing the ABSA robustness.Comment: Accepted in ACM Transactions on Information System
Reducing Spurious Correlations for Aspect-Based Sentiment Analysis with Variational Information Bottleneck and Contrastive Learning
Deep learning techniques have dominated the literature on aspect-based
sentiment analysis (ABSA), yielding state-of-the-art results. However, these
deep models generally suffer from spurious correlation problems between input
features and output labels, which creates significant barriers to robustness
and generalization capability. In this paper, we propose a novel Contrastive
Variational Information Bottleneck framework (called CVIB) to reduce spurious
correlations for ABSA. The proposed CVIB framework is composed of an original
network and a self-pruned network, and these two networks are optimized
simultaneously via contrastive learning. Concretely, we employ the Variational
Information Bottleneck (VIB) principle to learn an informative and compressed
network (self-pruned network) from the original network, which discards the
superfluous patterns or spurious correlations between input features and
prediction labels. Then, self-pruning contrastive learning is devised to pull
together semantically similar positive pairs and push away dissimilar pairs,
where the representations of the anchor learned by the original and self-pruned
networks respectively are regarded as a positive pair while the representations
of two different sentences within a mini-batch are treated as a negative pair.
To verify the effectiveness of our CVIB method, we conduct extensive
experiments on five benchmark ABSA datasets and the experimental results show
that our approach achieves better performance than the strong competitors in
terms of overall prediction performance, robustness, and generalization
State-of-the-art generalisation research in NLP: a taxonomy and review
The ability to generalise well is one of the primary desiderata of natural
language processing (NLP). Yet, what `good generalisation' entails and how it
should be evaluated is not well understood, nor are there any common standards
to evaluate it. In this paper, we aim to lay the ground-work to improve both of
these issues. We present a taxonomy for characterising and understanding
generalisation research in NLP, we use that taxonomy to present a comprehensive
map of published generalisation studies, and we make recommendations for which
areas might deserve attention in the future. Our taxonomy is based on an
extensive literature review of generalisation research, and contains five axes
along which studies can differ: their main motivation, the type of
generalisation they aim to solve, the type of data shift they consider, the
source by which this data shift is obtained, and the locus of the shift within
the modelling pipeline. We use our taxonomy to classify over 400 previous
papers that test generalisation, for a total of more than 600 individual
experiments. Considering the results of this review, we present an in-depth
analysis of the current state of generalisation research in NLP, and make
recommendations for the future. Along with this paper, we release a webpage
where the results of our review can be dynamically explored, and which we
intend to up-date as new NLP generalisation studies are published. With this
work, we aim to make steps towards making state-of-the-art generalisation
testing the new status quo in NLP.Comment: 35 pages of content + 53 pages of reference
Towards Debiasing NLU Models from Unknown Biases
NLU models often exploit biases to achieve high dataset-specific performance
without properly learning the intended task. Recently proposed debiasing
methods are shown to be effective in mitigating this tendency. However, these
methods rely on a major assumption that the types of bias should be
\emph{known} a-priori, which limits their application to many NLU tasks and
datasets. In this work, we present the first step to bridge this gap by
introducing a self-debiasing framework that prevents models from mainly
utilizing biases without knowing them in advance. The proposed framework is
general and complementary to the existing debiasing methods. We show that it
allows these existing methods to retain the improvement on the challenge
datasets (i.e., sets of examples designed to expose models' reliance on biases)
without specifically targeting certain biases. Furthermore, the evaluation
suggests that applying the framework results in improved overall robustness.Comment: Accepted at EMNLP 202
Context-aware Adversarial Attack on Named Entity Recognition
In recent years, large pre-trained language models (PLMs) have achieved
remarkable performance on many natural language processing benchmarks. Despite
their success, prior studies have shown that PLMs are vulnerable to attacks
from adversarial examples. In this work, we focus on the named entity
recognition task and study context-aware adversarial attack methods to examine
the model's robustness. Specifically, we propose perturbing the most
informative words for recognizing entities to create adversarial examples and
investigate different candidate replacement methods to generate natural and
plausible adversarial examples. Experiments and analyses show that our methods
are more effective in deceiving the model into making wrong predictions than
strong baselines
Large Language Models for Software Engineering: A Systematic Literature Review
Large Language Models (LLMs) have significantly impacted numerous domains,
notably including Software Engineering (SE). Nevertheless, a well-rounded
understanding of the application, effects, and possible limitations of LLMs
within SE is still in its early stages. To bridge this gap, our systematic
literature review takes a deep dive into the intersection of LLMs and SE, with
a particular focus on understanding how LLMs can be exploited in SE to optimize
processes and outcomes. Through a comprehensive review approach, we collect and
analyze a total of 229 research papers from 2017 to 2023 to answer four key
research questions (RQs). In RQ1, we categorize and provide a comparative
analysis of different LLMs that have been employed in SE tasks, laying out
their distinctive features and uses. For RQ2, we detail the methods involved in
data collection, preprocessing, and application in this realm, shedding light
on the critical role of robust, well-curated datasets for successful LLM
implementation. RQ3 allows us to examine the specific SE tasks where LLMs have
shown remarkable success, illuminating their practical contributions to the
field. Finally, RQ4 investigates the strategies employed to optimize and
evaluate the performance of LLMs in SE, as well as the common techniques
related to prompt optimization. Armed with insights drawn from addressing the
aforementioned RQs, we sketch a picture of the current state-of-the-art,
pinpointing trends, identifying gaps in existing research, and flagging
promising areas for future study
- …