Detecting Sarcasm in Multimodal Social Platforms

Sarcasm is a peculiar form of sentiment expression, where the surface sentiment differs from the implied sentiment. The detection of sarcasm in social media platforms has been applied in the past mainly to textual utterances where lexical indicators (such as interjections and intensifiers), linguistic markers, and contextual information (such as user profiles, or past conversations) were used to detect the sarcastic tone. However, modern social media platforms allow to create multimodal messages where audiovisual content is integrated with the text, making the analysis of a mode in isolation partial. In our work, we first study the relationship between the textual and visual aspects in multimodal posts from three major social media platforms, i.e., Instagram, Tumblr and Twitter, and we run a crowdsourcing task to quantify the extent to which images are perceived as necessary by human annotators. Moreover, we propose two different computational frameworks to detect sarcasm that integrate the textual and visual modalities. The first approach exploits visual semantics trained on an external dataset, and concatenates the semantics features with state-of-the-art textual features. The second method adapts a visual neural network initialized with parameters trained on ImageNet to multimodal sarcastic posts. Results show the positive effect of combining modalities for the detection of sarcasm across platforms and methods.Comment: 10 pages, 3 figures, final version published in the Proceedings of ACM Multimedia 201

Neurophysiological Profile of Antismoking Campaigns

Over the past few decades, antismoking public service announcements (PSAs) have been used by governments to promote healthy behaviours in citizens, for instance, against drinking before the drive and against smoke. Effectiveness of such PSAs has been suggested especially for young persons. By now, PSAs efficacy is still mainly assessed through traditional methods (questionnaires and metrics) and could be performed only after the PSAs broadcasting, leading to waste of economic resources and time in the case of Ineffective PSAs. One possible countermeasure to such ineffective use of PSAs could be promoted by the evaluation of the cerebral reaction to the PSA of particular segments of population (e.g., old, young, and heavy smokers). In addition, it is crucial to gather such cerebral activity in front of PSAs that have been assessed to be effective against smoke (Effective PSAs), comparing results to the cerebral reactions to PSAs that have been certified to be not effective (Ineffective PSAs). &e eventual differences between the cerebral responses toward the two PSA groups will provide crucial information about the possible outcome of new PSAs before to its broadcasting. &is study focused on adult population, by investigating the cerebral reaction to the vision of different PSA images, which have already been shown to be Effective and Ineffective for the promotion of an antismoking behaviour. Results showed how variables as gender and smoking habits can influence the perception of PSA images, and how different communication styles of the antismoking campaigns could facilitate the comprehension of PSA’s message and then enhance the related impac

6 Seconds of Sound and Vision: Creativity in Micro-Videos

The notion of creativity, as opposed to related concepts such as beauty or interestingness, has not been studied from the perspective of automatic analysis of multimedia content. Meanwhile, short online videos shared on social media platforms, or micro-videos, have arisen as a new medium for creative expression. In this paper we study creative micro-videos in an effort to understand the features that make a video creative, and to address the problem of automatic detection of creative content. Defining creative videos as those that are novel and have aesthetic value, we conduct a crowdsourcing experiment to create a dataset of over 3,800 micro-videos labelled as creative and non-creative. We propose a set of computational features that we map to the components of our definition of creativity, and conduct an analysis to determine which of these features correlate most with creative video. Finally, we evaluate a supervised approach to automatically detect creative video, with promising results, showing that it is necessary to model both aesthetic value and novelty to achieve optimal classification accuracy.Comment: 8 pages, 1 figures, conference IEEE CVPR 201

Exploring narrative presentation for large multimodal lifelog collections through card sorting

Using lifelogging tools, personal digital artifacts are collected continuously and passively throughout each day. The wealth of information such an archive contains on our life history provides novel opportunities for the creation of digital life narratives. However, the complexity, volume and multimodal nature of such collections create barriers to achieving this. Nine participants engaged in a card-sorting activity designed to explore practices of content reduction and presentation for narrative composition. We found the visual modalities to be most fluent in communicating experience with other modalities serving to support them and that the users employed the salient themes of the story to organise, arrange and facilitate filtering of the content

Uncertainties in the Algorithmic Image

The incorporation of algorithmic procedures into the automation of image production has been gradual, but has reached critical mass over the past century, especially with the advent of photography, the introduction of digital computers and the use of artificial intelligence (AI) and machine learning (ML). Due to the increasingly significant influence algorithmic processes have on visual media, there has been an expansion of the possibilities as to how images may behave, and a consequent struggle to define them. This algorithmic turnhighlights inner tensions within existing notions of the image, namely raising questions regarding the autonomy of machines, author- and viewer- ship, and the veracity of representations. In this sense, algorithmic images hover uncertainly between human and machine as producers and interpreters of visual information, between representational and non-representational, and between visible surface and the processes behind it. This paper gives an introduction to fundamental internal discrepancies which arise within algorithmically produced images, examined through a selection of relevant artistic examples. Focusing on the theme of uncertainty, this investigation considers how algorithmic images contain aspects which conflict with the certitude of computation, and how this contributes to a difficulty in defining images

Robust Image Sentiment Analysis Using Progressively Trained and Domain Transferred Deep Networks

Sentiment analysis of online user generated content is important for many social media analytics tasks. Researchers have largely relied on textual sentiment analysis to develop systems to predict political elections, measure economic indicators, and so on. Recently, social media users are increasingly using images and videos to express their opinions and share their experiences. Sentiment analysis of such large scale visual content can help better extract user sentiments toward events or topics, such as those in image tweets, so that prediction of sentiment from visual content is complementary to textual sentiment analysis. Motivated by the needs in leveraging large scale yet noisy training data to solve the extremely challenging problem of image sentiment analysis, we employ Convolutional Neural Networks (CNN). We first design a suitable CNN architecture for image sentiment analysis. We obtain half a million training samples by using a baseline sentiment algorithm to label Flickr images. To make use of such noisy machine labeled data, we employ a progressive strategy to fine-tune the deep network. Furthermore, we improve the performance on Twitter images by inducing domain transfer with a small number of manually labeled Twitter images. We have conducted extensive experiments on manually labeled Twitter images. The results show that the proposed CNN can achieve better performance in image sentiment analysis than competing algorithms.Comment: 9 pages, 5 figures, AAAI 201