5,854 research outputs found
Targeted aspect based multimodal sentiment analysis:an attention capsule extraction and multi-head fusion network
Multimodal sentiment analysis has currently identified its significance in a
variety of domains. For the purpose of sentiment analysis, different aspects of
distinguishing modalities, which correspond to one target, are processed and
analyzed. In this work, we propose the targeted aspect-based multimodal
sentiment analysis (TABMSA) for the first time. Furthermore, an attention
capsule extraction and multi-head fusion network (EF-Net) on the task of TABMSA
is devised. The multi-head attention (MHA) based network and the ResNet-152 are
employed to deal with texts and images, respectively. The integration of MHA
and capsule network aims to capture the interaction among the multimodal
inputs. In addition to the targeted aspect, the information from the context
and the image is also incorporated for sentiment delivered. We evaluate the
proposed model on two manually annotated datasets. the experimental results
demonstrate the effectiveness of our proposed model for this new task
Multimodal Sentiment Analysis Based on Deep Learning: Recent Progress
Multimodal sentiment analysis is an important research topic in the field of NLP, aiming to analyze speakers\u27 sentiment tendencies through features extracted from textual, visual, and acoustic modalities. Its main methods are based on machine learning and deep learning. Machine learning-based methods rely heavily on labeled data. But deep learning-based methods can overcome this shortcoming and capture the in-depth semantic information and modal characteristics of the data, as well as the interactive information between multimodal data. In this paper, we survey the deep learning-based methods, including fusion of text and image and fusion of text, image, audio, and video. Specifically, we discuss the main problems of these methods and the future directions. Finally, we review the work of multimodal sentiment analysis in conversation
Hierachical Delta-Attention Method for Multimodal Fusion
In vision and linguistics; the main input modalities are facial expressions,
speech patterns, and the words uttered. The issue with analysis of any one mode
of expression (Visual, Verbal or Vocal) is that lot of contextual information
can get lost. This asks researchers to inspect multiple modalities to get a
thorough understanding of the cross-modal dependencies and temporal context of
the situation to analyze the expression. This work attempts at preserving the
long-range dependencies within and across different modalities, which would be
bottle-necked by the use of recurrent networks and adds the concept of
delta-attention to focus on local differences per modality to capture the
idiosyncrasy of different people. We explore a cross-attention fusion technique
to get the global view of the emotion expressed through these
delta-self-attended modalities, in order to fuse all the local nuances and
global context together. The addition of attention is new to the multi-modal
fusion field and currently being scrutinized for on what stage the attention
mechanism should be used, this work achieves competitive accuracy for overall
and per-class classification which is close to the current state-of-the-art
with almost half number of parameters
A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis
Understanding expressed sentiment and emotions are two crucial factors in
human multimodal language. This paper describes a Transformer-based
joint-encoding (TBJE) for the task of Emotion Recognition and Sentiment
Analysis. In addition to use the Transformer architecture, our approach relies
on a modular co-attention and a glimpse layer to jointly encode one or more
modalities. The proposed solution has also been submitted to the ACL20: Second
Grand-Challenge on Multimodal Language to be evaluated on the CMU-MOSEI
dataset. The code to replicate the presented experiments is open-source:
https://github.com/jbdel/MOSEI_UMONS.Comment: Winner of the ACL20: Second Grand-Challenge on Multimodal Languag
- …