Attention in Natural Language Processing

Attention is an increasingly popular mechanism used in a wide range of neural architectures. The mechanism itself has been realized in a variety of formats. However, because of the fast-paced advances in this domain, a systematic overview of attention is still missing. In this article, we define a unified model for attention architectures in natural language processing, with a focus on those designed to work with vector representations of the textual data. We propose a taxonomy of attention models according to four dimensions: the representation of the input, the compatibility function, the distribution function, and the multiplicity of the input and/or output. We present the examples of how prior information can be exploited in attention models and discuss ongoing research efforts and open challenges in the area, providing the first extensive categorization of the vast body of literature in this exciting domain

ATP: A holistic attention integrated approach to enhance ABSA

Aspect based sentiment analysis (ABSA) deals with the identification of the sentiment polarity of a review sentence towards a given aspect. Deep Learning sequential models like RNN, LSTM, and GRU are current state-of-the-art methods for inferring the sentiment polarity. These methods work well to capture the contextual relationship between the words of a review sentence. However, these methods are insignificant in capturing long-term dependencies. Attention mechanism plays a significant role by focusing only on the most crucial part of the sentence. In the case of ABSA, aspect position plays a vital role. Words near to aspect contribute more while determining the sentiment towards the aspect. Therefore, we propose a method that captures the position based information using dependency parsing tree and helps attention mechanism. Using this type of position information over a simple word-distance-based position enhances the deep learning model's performance. We performed the experiments on SemEval'14 dataset to demonstrate the effect of dependency parsing relation-based attention for ABSA

Analisis Sentimen Berbasis Aspek dengan Deep Learning Ditinjau dari Sudut Pandang Filsafat Ilmu

Pesatnya pertumbuhan internet dan semakin populernya aplikasi media sosial memungkinkan orang untuk mengekspresikan opini dan pengalaman tentang sesuatu kepada public secara terbuka. Hal tersebut dapat dimanfaatkan dan dianalisis untuk mengeksplorasi customer behaviour (perilaku pengguna), memprediksi kebutuhan pengguna dan memahami opininya. Analisis sentimen berbasis aspek (aspect-based sentiment analysis) membuat analisis dan investigasi untuk mengidentifikasi polaritas sentimen pada aspek spesifik secara tepat. Deep learning untuk analisis sentimen berbasis aspek saat ini telah menunjukkan kinerja yang cukup menjanjikan karena efisiensinya dalam ekstraksi fitur otomatis dan kemampuannya untuk menangkap fitur sintaksis dan semantik teks tanpa perlu rekayasa fitur tingkat tinggi. Menurut Thomas Kuhn, ilmu pengetahuan tidak bersifat kumulatif, tetapi revolusioner dan berkembang secara historis. Ilmu pengetahuan tidak terlepas dari paradigma. Tulisan ini bertujuan untuk memberikan ulasan tentang penggunaan deep learning untuk analisis sentimen berbasis aspek dan tinjauannya menurut pandangan filsafat ilmu.The rapid growth of the internet and social media application make possiblity for people to express their opinion and experinces about something publicly. It can be utilized and analysed to explore the user behaviour, predict their demand and understand their opinion. Aspect-based sentiment analysis makes an analysis and investigation identify sentiment polarity on specific aspects precisely. Currently, deep learning for aspect-based sentiment analysis has shown a promising performance due to their efficiency of automatic feature extraction and their ability to capture both syntactic and semantic features of text without requirements for high-level feature engineering. According to Thomas Kuhn, the development of science is not cummulative but revolusionary and has a historical story. Science can not be separated from paradigm. The aim of this paper is for describing the used of deep learning for aspect-based sentiment analysis and its review from the philosophy of science

Sentiment Analysis Based on Deep Learning: A Comparative Study

The study of public opinion can provide us with valuable information. The analysis of sentiment on social networks, such as Twitter or Facebook, has become a powerful means of learning about the users' opinions and has a wide range of applications. However, the efficiency and accuracy of sentiment analysis is being hindered by the challenges encountered in natural language processing (NLP). In recent years, it has been demonstrated that deep learning models are a promising solution to the challenges of NLP. This paper reviews the latest studies that have employed deep learning to solve sentiment analysis problems, such as sentiment polarity. Models using term frequency-inverse document frequency (TF-IDF) and word embedding have been applied to a series of datasets. Finally, a comparative study has been conducted on the experimental results obtained for the different models and input feature

TDAM: a topic-dependent attention model for sentiment analysis

We propose a topic-dependent attention model for sentiment classification and topic extraction. Our model assumes that a global topic embedding is shared across documents and employs an attention mechanism to derive local topic embedding for words and sentences. These are subsequently incorporated in a modified Gated Recurrent Unit (GRU) for sentiment classification and extraction of topics bearing different sentiment polarities. Those topics emerge from the words' local topic embeddings learned by the internal attention of the GRU cells in the context of a multi-task learning framework. In this paper, we present the hierarchical architecture, the new GRU unit and the experiments conducted on users' reviews which demonstrate classification performance on a par with the state-of-the-art methodologies for sentiment classification and topic coherence outperforming the current approaches for supervised topic extraction. In addition, our model is able to extract coherent aspect-sentiment clusters despite using no aspect-level annotations for training