2,078 research outputs found
Utilizing Priming to Identify Optimal Class Ordering to Alleviate Catastrophic Forgetting
In order for artificial neural networks to begin accurately mimicking
biological ones, they must be able to adapt to new exigencies without
forgetting what they have learned from previous training. Lifelong learning
approaches to artificial neural networks attempt to strive towards this goal,
yet have not progressed far enough to be realistically deployed for natural
language processing tasks. The proverbial roadblock of catastrophic forgetting
still gate-keeps researchers from an adequate lifelong learning model. While
efforts are being made to quell catastrophic forgetting, there is a lack of
research that looks into the importance of class ordering when training on new
classes for incremental learning. This is surprising as the ordering of
"classes" that humans learn is heavily monitored and incredibly important.
While heuristics to develop an ideal class order have been researched, this
paper examines class ordering as it relates to priming as a scheme for
incremental class learning. By examining the connections between various
methods of priming found in humans and how those are mimicked yet remain
unexplained in life-long machine learning, this paper provides a better
understanding of the similarities between our biological systems and the
synthetic systems while simultaneously improving current practices to combat
catastrophic forgetting. Through the merging of psychological priming practices
with class ordering, this paper is able to identify a generalizable method for
class ordering in NLP incremental learning tasks that consistently outperforms
random class ordering.Comment: Accepted to IEEE International Conference on Semantic Computing
(ICSC) 202
From Word to Sense Embeddings: A Survey on Vector Representations of Meaning
Over the past years, distributed semantic representations have proved to be
effective and flexible keepers of prior knowledge to be integrated into
downstream applications. This survey focuses on the representation of meaning.
We start from the theoretical background behind word vector space models and
highlight one of their major limitations: the meaning conflation deficiency,
which arises from representing a word with all its possible meanings as a
single vector. Then, we explain how this deficiency can be addressed through a
transition from the word level to the more fine-grained level of word senses
(in its broader acceptation) as a method for modelling unambiguous lexical
meaning. We present a comprehensive overview of the wide range of techniques in
the two main branches of sense representation, i.e., unsupervised and
knowledge-based. Finally, this survey covers the main evaluation procedures and
applications for this type of representation, and provides an analysis of four
of its important aspects: interpretability, sense granularity, adaptability to
different domains and compositionality.Comment: 46 pages, 8 figures. Published in Journal of Artificial Intelligence
Researc
Continual Learning of Natural Language Processing Tasks: A Survey
Continual learning (CL) is an emerging learning paradigm that aims to emulate
the human capability of learning and accumulating knowledge continually without
forgetting the previously learned knowledge and also transferring the knowledge
to new tasks to learn them better. This survey presents a comprehensive review
of the recent progress of CL in the NLP field. It covers (1) all CL settings
with a taxonomy of existing techniques. Besides dealing with forgetting, it
also focuses on (2) knowledge transfer, which is of particular importance to
NLP. Both (1) and (2) are not mentioned in the existing survey. Finally, a list
of future directions is also discussed
- …