Measurement of Visual Cues and Their Effects on Online Users: An Image Mining Approach

Textual marketing communication is effective in various contexts such as print advertising, user-generated content, and social media (Diamond 1968; Ludwig et al. 2013; Nam and Kannan 2014). However, visual marketing communication studies are limited in the context of print advertising (e.g., Hagtvedt and Brasel 2017). This dissertation includes two essays to examine the visual communication effectiveness online. Essay 1 develops a conceptual framework to examine the visual-based brand perception (VBBP) and related concepts on social media. We propose that the VBBP is a co-creational process between a company and its consumers and exhibits three characteristics: i) a two-way communication that both a company and its consumers are pivotal authors of brand stories, ii) a dynamic process that the brand meaning keeps evolving, and iii) a dyadic process between a company and its consumers. In the conceptual model development, we identify six visual attributes as measures of VBBP and adopt a machine learning-based image mining technique to quantify the measures on a large scale. We empirically validate the conceptual model and find that during the co-creational process, both the company and consumer visual-based brand perception information richness (VBBP_R) increase over time. Moreover, in examining the difference between a company and its consumers, we find that there is a visual-based brand perception gap (VBBP_G) between a company and its consumers. From these findings, we advise three marketing communication strategies to help companies manage their VBBP_G. Essay 2 examines a related research question: the joint effects of visual and textual communication on crowdfunding success. Essay 2 extends Essay 1 in three ways: i) we consider both textual and visual marketing communication on another online platform, ii) beyond the concept of perception, we emphasize examining how marketing communication influences a marketing outcome: duration of crowdfunding success, iii) we investigate not only how visual and textual communication influence crowdfunding success individually but also how they influence the outcome jointly. We empirically validate visual communication is more effective than textual communication on a crowdfunding platform. Our findings support an integrated marketing communication strategy that marketers should implement using multiple communication tools in a harmonic way. We demonstrate that the synergistic effect of visual and textual communication has a positive effect on crowdfunding outcome

Econometrics meets sentiment : an overview of methodology and applications

The advent of massive amounts of textual, audio, and visual data has spurred the development of econometric methodology to transform qualitative sentiment data into quantitative sentiment variables, and to use those variables in an econometric analysis of the relationships between sentiment and other variables. We survey this emerging research field and refer to it as sentometrics, which is a portmanteau of sentiment and econometrics. We provide a synthesis of the relevant methodological approaches, illustrate with empirical results, and discuss useful software

Learning Social Image Embedding with Deep Multimodal Attention Networks

Learning social media data embedding by deep models has attracted extensive research interest as well as boomed a lot of applications, such as link prediction, classification, and cross-modal search. However, for social images which contain both link information and multimodal contents (e.g., text description, and visual content), simply employing the embedding learnt from network structure or data content results in sub-optimal social image representation. In this paper, we propose a novel social image embedding approach called Deep Multimodal Attention Networks (DMAN), which employs a deep model to jointly embed multimodal contents and link information. Specifically, to effectively capture the correlations between multimodal contents, we propose a multimodal attention network to encode the fine-granularity relation between image regions and textual words. To leverage the network structure for embedding learning, a novel Siamese-Triplet neural network is proposed to model the links among images. With the joint deep model, the learnt embedding can capture both the multimodal contents and the nonlinear network information. Extensive experiments are conducted to investigate the effectiveness of our approach in the applications of multi-label classification and cross-modal search. Compared to state-of-the-art image embeddings, our proposed DMAN achieves significant improvement in the tasks of multi-label classification and cross-modal search

Multimodal Classification of Urban Micro-Events

In this paper we seek methods to effectively detect urban micro-events. Urban micro-events are events which occur in cities, have limited geographical coverage and typically affect only a small group of citizens. Because of their scale these are difficult to identify in most data sources. However, by using citizen sensing to gather data, detecting them becomes feasible. The data gathered by citizen sensing is often multimodal and, as a consequence, the information required to detect urban micro-events is distributed over multiple modalities. This makes it essential to have a classifier capable of combining them. In this paper we explore several methods of creating such a classifier, including early, late, hybrid fusion and representation learning using multimodal graphs. We evaluate performance on a real world dataset obtained from a live citizen reporting system. We show that a multimodal approach yields higher performance than unimodal alternatives. Furthermore, we demonstrate that our hybrid combination of early and late fusion with multimodal embeddings performs best in classification of urban micro-events

Temporal Cross-Media Retrieval with Soft-Smoothing

Multimedia information have strong temporal correlations that shape the way modalities co-occur over time. In this paper we study the dynamic nature of multimedia and social-media information, where the temporal dimension emerges as a strong source of evidence for learning the temporal correlations across visual and textual modalities. So far, cross-media retrieval models, explored the correlations between different modalities (e.g. text and image) to learn a common subspace, in which semantically similar instances lie in the same neighbourhood. Building on such knowledge, we propose a novel temporal cross-media neural architecture, that departs from standard cross-media methods, by explicitly accounting for the temporal dimension through temporal subspace learning. The model is softly-constrained with temporal and inter-modality constraints that guide the new subspace learning task by favouring temporal correlations between semantically similar and temporally close instances. Experiments on three distinct datasets show that accounting for time turns out to be important for cross-media retrieval. Namely, the proposed method outperforms a set of baselines on the task of temporal cross-media retrieval, demonstrating its effectiveness for performing temporal subspace learning.Comment: To appear in ACM MM 201