367 research outputs found
Weakly supervised coupled networks for visual sentiment analysis
Automatic assessment of sentiment from visual content
has gained considerable attention with the increasing tendency
of expressing opinions on-line. In this paper, we solve
the problem of visual sentiment analysis using the high-level
abstraction in the recognition process. Existing methods
based on convolutional neural networks learn sentiment
representations from the holistic image appearance. However,
different image regions can have a different influence
on the intended expression. This paper presents a weakly
supervised coupled convolutional network with two branches
to leverage the localized information. The first branch
detects a sentiment specific soft map by training a fully convolutional
network with the cross spatial pooling strategy,
which only requires image-level labels, thereby significantly
reducing the annotation burden. The second branch utilizes
both the holistic and localized information by coupling
the sentiment map with deep features for robust classification.
We integrate the sentiment detection and classification
branches into a unified deep framework and optimize
the network in an end-to-end manner. Extensive experiments
on six benchmark datasets demonstrate that the
proposed method performs favorably against the state-ofthe-
art methods for visual sentiment analysis
Automatic Understanding of Image and Video Advertisements
There is more to images than their objective physical content: for example,
advertisements are created to persuade a viewer to take a certain action. We
propose the novel problem of automatic advertisement understanding. To enable
research on this problem, we create two datasets: an image dataset of 64,832
image ads, and a video dataset of 3,477 ads. Our data contains rich annotations
encompassing the topic and sentiment of the ads, questions and answers
describing what actions the viewer is prompted to take and the reasoning that
the ad presents to persuade the viewer ("What should I do according to this ad,
and why should I do it?"), and symbolic references ads make (e.g. a dove
symbolizes peace). We also analyze the most common persuasive strategies ads
use, and the capabilities that computer vision systems should have to
understand these strategies. We present baseline classification results for
several prediction tasks, including automatically answering questions about the
messages of the ads.Comment: To appear in CVPR 2017; data available on
http://cs.pitt.edu/~kovashka/ad
PDANet: Polarity-consistent Deep Attention Network for Fine-grained Visual Emotion Regression
Existing methods on visual emotion analysis mainly focus on coarse-grained
emotion classification, i.e. assigning an image with a dominant discrete
emotion category. However, these methods cannot well reflect the complexity and
subtlety of emotions. In this paper, we study the fine-grained regression
problem of visual emotions based on convolutional neural networks (CNNs).
Specifically, we develop a Polarity-consistent Deep Attention Network (PDANet),
a novel network architecture that integrates attention into a CNN with an
emotion polarity constraint. First, we propose to incorporate both spatial and
channel-wise attentions into a CNN for visual emotion regression, which jointly
considers the local spatial connectivity patterns along each channel and the
interdependency between different channels. Second, we design a novel
regression loss, i.e. polarity-consistent regression (PCR) loss, based on the
weakly supervised emotion polarity to guide the attention generation. By
optimizing the PCR loss, PDANet can generate a polarity preserved attention map
and thus improve the emotion regression performance. Extensive experiments are
conducted on the IAPS, NAPS, and EMOTIC datasets, and the results demonstrate
that the proposed PDANet outperforms the state-of-the-art approaches by a large
margin for fine-grained visual emotion regression. Our source code is released
at: https://github.com/ZizhouJia/PDANet.Comment: Accepted by ACM Multimedia 201
Affective Image Content Analysis: Two Decades Review and New Perspectives
Images can convey rich semantics and induce various emotions in viewers.
Recently, with the rapid advancement of emotional intelligence and the
explosive growth of visual data, extensive research efforts have been dedicated
to affective image content analysis (AICA). In this survey, we will
comprehensively review the development of AICA in the recent two decades,
especially focusing on the state-of-the-art methods with respect to three main
challenges -- the affective gap, perception subjectivity, and label noise and
absence. We begin with an introduction to the key emotion representation models
that have been widely employed in AICA and description of available datasets
for performing evaluation with quantitative comparison of label noise and
dataset bias. We then summarize and compare the representative approaches on
(1) emotion feature extraction, including both handcrafted and deep features,
(2) learning methods on dominant emotion recognition, personalized emotion
prediction, emotion distribution learning, and learning from noisy data or few
labels, and (3) AICA based applications. Finally, we discuss some challenges
and promising research directions in the future, such as image content and
context understanding, group emotion clustering, and viewer-image interaction.Comment: Accepted by IEEE TPAM
Visual sentiment prediction based on automatic discovery of affective regions
Automatic assessment of sentiment from visual
content has gained considerable attention with the increasing
tendency of expressing opinions via images and videos online.
This paper investigates the problem of visual sentiment analysis,
which involves a high-level abstraction in the recognition process.
While most of the current methods focus on improving holistic
representations, we aim to utilize the local information, which is
inspired by the observation that both the whole image and local
regions convey significant sentiment information. We propose
a framework to leverage affective regions, where we first use
an off-the-shelf objectness tool to generate the candidates, and
employ a candidate selection method to remove redundant and
noisy proposals. Then a convolutional neural network (CNN) is
connected with each candidate to compute the sentiment scores,
and the affective regions are automatically discovered, taking the
objectness score as well as the sentiment score into consideration.
Finally, the CNN outputs from local regions are aggregated with
the whole images to produce the final predictions. Our framework
only requires image-level labels, thereby significantly reducing
the annotation burden otherwise required for training. This is
especially important for sentiment analysis as sentiment can be
abstract, and labeling affective regions is too subjective and
labor-consuming. Extensive experiments show that the proposed
algorithm outperforms the state-of-the-art approaches on eight
popular benchmark datasets
A survey on deep learning in image polarity detection: Balancing generalization performances and computational costs
Deep convolutional neural networks (CNNs) provide an effective tool to extract complex information from images. In the area of image polarity detection, CNNs are customarily utilized in combination with transfer learning techniques to tackle a major problem: the unavailability of large sets of labeled data. Thus, polarity predictors in general exploit a pre-trained CNN as the feature extractor that in turn feeds a classification unit. While the latter unit is trained from scratch, the pre-trained CNN is subject to fine-tuning. As a result, the specific CNN architecture employed as the feature extractor strongly affects the overall performance of the model. This paper analyses state-of-the-art literature on image polarity detection and identifies the most reliable CNN architectures. Moreover, the paper provides an experimental protocol that should allow assessing the role played by the baseline architecture in the polarity detection task. Performance is evaluated in terms of both generalization abilities and computational complexity. The latter attribute becomes critical as polarity predictors, in the era of social networks, might need to be updated within hours or even minutes. In this regard, the paper gives practical hints on the advantages and disadvantages of the examined architectures both in terms of generalization and computational cost
Big data analytics:Computational intelligence techniques and application areas
Big Data has significant impact in developing functional smart cities and supporting modern societies. In this paper, we investigate the importance of Big Data in modern life and economy, and discuss challenges arising from Big Data utilization. Different computational intelligence techniques have been considered as tools for Big Data analytics. We also explore the powerful combination of Big Data and Computational Intelligence (CI) and identify a number of areas, where novel applications in real world smart city problems can be developed by utilizing these powerful tools and techniques. We present a case study for intelligent transportation in the context of a smart city, and a novel data modelling methodology based on a biologically inspired universal generative modelling approach called Hierarchical Spatial-Temporal State Machine (HSTSM). We further discuss various implications of policy, protection, valuation and commercialization related to Big Data, its applications and deployment
- …