Multimodal Content Analysis for Effective Advertisements on YouTube

The rapid advances in e-commerce and Web 2.0 technologies have greatly increased the impact of commercial advertisements on the general public. As a key enabling technology, a multitude of recommender systems exists which analyzes user features and browsing patterns to recommend appealing advertisements to users. In this work, we seek to study the characteristics or attributes that characterize an effective advertisement and recommend a useful set of features to aid the designing and production processes of commercial advertisements. We analyze the temporal patterns from multimedia content of advertisement videos including auditory, visual and textual components, and study their individual roles and synergies in the success of an advertisement. The objective of this work is then to measure the effectiveness of an advertisement, and to recommend a useful set of features to advertisement designers to make it more successful and approachable to users. Our proposed framework employs the signal processing technique of cross modality feature learning where data streams from different components are employed to train separate neural network models and are then fused together to learn a shared representation. Subsequently, a neural network model trained on this joint feature embedding representation is utilized as a classifier to predict advertisement effectiveness. We validate our approach using subjective ratings from a dedicated user study, the sentiment strength of online viewer comments, and a viewer opinion metric of the ratio of the Likes and Views received by each advertisement from an online platform.Comment: 11 pages, 5 figures, ICDM 201

Visual Affect Around the World: A Large-scale Multilingual Visual Sentiment Ontology

Every culture and language is unique. Our work expressly focuses on the uniqueness of culture and language in relation to human affect, specifically sentiment and emotion semantics, and how they manifest in social multimedia. We develop sets of sentiment- and emotion-polarized visual concepts by adapting semantic structures called adjective-noun pairs, originally introduced by Borth et al. (2013), but in a multilingual context. We propose a new language-dependent method for automatic discovery of these adjective-noun constructs. We show how this pipeline can be applied on a social multimedia platform for the creation of a large-scale multilingual visual sentiment concept ontology (MVSO). Unlike the flat structure in Borth et al. (2013), our unified ontology is organized hierarchically by multilingual clusters of visually detectable nouns and subclusters of emotionally biased versions of these nouns. In addition, we present an image-based prediction task to show how generalizable language-specific models are in a multilingual context. A new, publicly available dataset of >15.6K sentiment-biased visual concepts across 12 languages with language-specific detector banks, >7.36M images and their metadata is also released.Comment: 11 pages, to appear at ACM MM'1

Affective Image Content Analysis: Two Decades Review and New Perspectives

Images can convey rich semantics and induce various emotions in viewers. Recently, with the rapid advancement of emotional intelligence and the explosive growth of visual data, extensive research efforts have been dedicated to affective image content analysis (AICA). In this survey, we will comprehensively review the development of AICA in the recent two decades, especially focusing on the state-of-the-art methods with respect to three main challenges -- the affective gap, perception subjectivity, and label noise and absence. We begin with an introduction to the key emotion representation models that have been widely employed in AICA and description of available datasets for performing evaluation with quantitative comparison of label noise and dataset bias. We then summarize and compare the representative approaches on (1) emotion feature extraction, including both handcrafted and deep features, (2) learning methods on dominant emotion recognition, personalized emotion prediction, emotion distribution learning, and learning from noisy data or few labels, and (3) AICA based applications. Finally, we discuss some challenges and promising research directions in the future, such as image content and context understanding, group emotion clustering, and viewer-image interaction.Comment: Accepted by IEEE TPAM

Destination advertising in a smarter way: A machine learning model for DMOs’ photo selection

Photos are important carriers for destination image communication. Currently, effectively and efficiently selecting appropriate photos for destination promotion remains a major challenge for DMOs, and has caused the discrepancy between projected and received images. In the process of photo selection, contents that can best evoke viewers’ potential motives should be highly considered. This project proposes and implements a machine learning based model to assist DMOs with photo content selection. This protocol can rank candidate photos describing a specific theme from viewers’ perspective. In our study, over 190,000 Flickr photos of New York City were analyzed to demonstrate the effectiveness of our approach. The results indicate that the proposed method can facilitate the selection of destination photos and address the well-known gap between the projected image and the received image

What movie will I watch today? The role of online review ratings, reviewers’ comments, and user’s gratification style

Browsing online ratings and viewers’ comments is an integral part of the experience of choosing and watching a movie. Current theories have broadened the concept of entertainment beyond amusement (hedonic motives) to include experiences of meaning, value, and self-development (eudaimonic motives). With a between-subjects design, we examined the role of the reviewer’s rating (medium rating vs high rating), comments (hedonic vs. eudaimonic), and participant’s gratification style on their interest in watching a movie. Results showed that participants (N = 383) reported a higher preference for the high rating movie. Results also revealed a match between comment type and individual gratification style, with participants with hedonic motives reporting more interest in the movie with hedonic comments, and those reporting eudaimonic motives for the movie with eudaimonic comments.info:eu-repo/semantics/acceptedVersio

A Study of Danmaku Video on Attention Allocation, Social Presence, Transportation to Narrative, Cognitive Workload and Enjoyment

Danmaku video (video with overlaid comments) is a relatively new social TV format and is getting popular in China. This study conducted a 3-condition experiment to examine Danmaku video watching experience in terms of 5 aspects: attention allocation, social presence, transportation into narrative, cognitive workload and enjoyment. 61 Chinese college students from the Northeast region of US were recruited to participate the study. Result indicated out that Danmaku distracted some attention from the initial video content but fostered a feeling of joint viewing with others. The presence of Danmaku also had some effect on the enjoyment of watching videos, but did not affect cognitive workload or the degree of feeling being transported into video’s narrative