Zapping index: Using smile to measure advertisement zapping likelihood

In marketing and advertising research, 'zapping' is defined as the action when a viewer stops watching a commercial. Researchers analyze users' behavior in order to prevent zapping which helps advertisers to design effective commercials. Since emotions can be used to engage consumers, in this paper, we leverage automated facial expression analysis to understand consumers' zapping behavior. Firstly, we provide an accurate moment-to-moment smile detection algorithm. Secondly, we formulate a binary classification problem (zapping/non-zapping) based on real-world scenarios, and adopt smile response as the feature to predict zapping. Thirdly, to cope with the lack of a metric in advertising evaluation, we propose a new metric called Zapping Index (ZI). ZI is a moment-to-moment measurement of a user's zapping probability. It gauges not only the reaction of a user, but also the preference of a user to commercials. Finally, extensive experiments are performed to provide insights and we make recommendations that will be useful to both advertisers and advertisement publishers

Measuring Voter's Candidate Preference Based on Affective Responses to Election Debates

In this paper we present the first analysis of facial responses to electoral debates measured automatically over the Internet. We show that significantly different responses can be detected from viewers with different political preferences and that similar expressions at significant moments can have very different meanings depending on the actions that appear subsequently. We used an Internet based framework to collect 611 naturalistic and spontaneous facial responses to five video clips from the 3rd presidential debate during the 2012 American presidential election campaign. Using this framework we were able to collect over 60% of these video responses (374 videos) within one day of the live debate and over 80% within three days. No participants were compensated for taking the survey. We present and evaluate a method for predicting independent voter preference based on automatically measured facial responses and self-reported preferences from the viewers. We predict voter preference with an average accuracy of over 73% (AUC 0.779)

Predicting Ad Liking and Purchase Intent: Large-Scale Analysis of Facial Responses to Ads

Billions of online video ads are viewed every month. We present a large-scale analysis of facial responses to video content measured over the Internet and their relationship to marketing effectiveness. We collected over 12,000 facial responses from 1,223 people to 170 ads from a range of markets and product categories. The facial responses were automatically coded frame-by-frame. Collection and coding of these 3.7 million frames would not have been feasible with traditional research methods. We show that detected expressions are sparse but that aggregate responses reveal rich emotion trajectories. By modeling the relationship between the facial responses and ad effectiveness, we show that ad liking can be predicted accurately (ROC AUC = 0.85) from webcam facial responses. Furthermore, the prediction of a change in purchase intent is possible (ROC AUC = 0.78). Ad liking is shown by eliciting expressions, particularly positive expressions. Driving purchase intent is more complex than just making viewers smile: peak positive responses that are immediately preceded by a brand appearance are more likely to be effective. The results presented here demonstrate a reliable and generalizable system for predicting ad effectiveness automatically from facial responses without a need to elicit self-report responses from the viewers. In addition we can gain insight into the structure of effective ads.MIT Media Lab ConsortiumNEC CorporationMAR

Emotion Recognition in the Wild using Deep Neural Networks and Bayesian Classifiers

Group emotion recognition in the wild is a challenging problem, due to the unstructured environments in which everyday life pictures are taken. Some of the obstacles for an effective classification are occlusions, variable lighting conditions, and image quality. In this work we present a solution based on a novel combination of deep neural networks and Bayesian classifiers. The neural network works on a bottom-up approach, analyzing emotions expressed by isolated faces. The Bayesian classifier estimates a global emotion integrating top-down features obtained through a scene descriptor. In order to validate the system we tested the framework on the dataset released for the Emotion Recognition in the Wild Challenge 2017. Our method achieved an accuracy of 64.68% on the test set, significantly outperforming the 53.62% competition baseline.Comment: accepted by the Fifth Emotion Recognition in the Wild (EmotiW) Challenge 201

Consumer Neuroscience-Based Metrics Predict Recall, Liking and Viewing Rates in Online Advertising

[EN] The purpose of the present study is to investigate whether the effectiveness of a new ad on digital channels (YouTube) can be predicted by using neural networks and neuroscience-based metrics (brain response, heart rate variability and eye tracking). Neurophysiological records from 35 participants were exposed to 8 relevant TV Super Bowl commercials. Correlations between neurophysiological-based metrics, ad recall, ad liking, the ACE metrix score and the number of views on YouTube during a year were investigated. Our findings suggest a significant correlation between neuroscience metrics and self-reported of ad effectiveness and the direct number of views on the YouTube channel. In addition, and using an artificial neural network based on neuroscience metrics, the model classifies (82.9% of average accuracy) and estimate the number of online views (mean error of 0.199). The results highlight the validity of neuromarketing-based techniques for predicting the success of advertising responses. Practitioners can consider the proposed methodology at the design stages of advertising content, thus enhancing advertising effectiveness. The study pioneers the use of neurophysiological methods in predicting advertising success in a digital context. This is the first article that has examined whether these measures could actually be used for predicting views for advertising on YouTube.This work has been supported by the Heineken Endowed Chair in Neuromarketing at the Polytechnic University of Valencia in order to research and apply new technologies and neuroscience in communication, distribution and consumption fields.Guixeres Provinciale, J.; Bigné-Alcañiz, E.; Ausin-Azofra, JM.; Alcañiz Raya, ML.; Colomer, A.; Fuentes-Hurtado, FJ.; Naranjo Ornedo, V. (2017). Consumer Neuroscience-Based Metrics Predict Recall, Liking and Viewing Rates in Online Advertising. Frontiers in Psychology. 8:1-14. https://doi.org/10.3389/fpsyg.2017.01808S1148Rajendra Acharya, U., Paul Joseph, K., Kannathal, N., Lim, C. M., & Suri, J. S. (2006). Heart rate variability: a review. Medical & Biological Engineering & Computing, 44(12), 1031-1051. doi:10.1007/s11517-006-0119-0Aftanas, L. I., Reva, N. V., Varlamov, A. A., Pavlov, S. V., & Makhnev, V. P. (2004). Analysis of Evoked EEG Synchronization and Desynchronization in Conditions of Emotional Activation in Humans: Temporal and Topographic Characteristics. Neuroscience and Behavioral Physiology, 34(8), 859-867. doi:10.1023/b:neab.0000038139.39812.ebAstolfi, L., De Vico Fallani, F., Cincotti, F., Mattia, D., Bianchi, L., Marciani, M. G., … Babiloni, F. (2008). Neural Basis for Brain Responses to TV Commercials: A High-Resolution EEG Study. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 16(6), 522-531. doi:10.1109/tnsre.2008.2009784Astolfi, L., Fallani, F. D. V., Cincotti, F., Mattia, D., Bianchi, L., Marciani, M. G., … Babiloni, F. (2009). Brain activity during the memorization of visual scenes from TV commercials: An application of high resolution EEG and steady state somatosensory evoked potentials technologies. Journal of Physiology-Paris, 103(6), 333-341. doi:10.1016/j.jphysparis.2009.07.002Baack, D. W., Wilson, R. T., & Till, B. D. (2008). Creativity and Memory Effects: Recall, Recognition, and an Exploration of Nontraditional Media. Journal of Advertising, 37(4), 85-94. doi:10.2753/joa0091-3367370407Bellman, S., Murphy, J., Treleaven-Hassard, S., O’Farrell, J., Qiu, L., & Varan, D. (2013). Using Internet Behavior to Deliver Relevant Television Commercials. Journal of Interactive Marketing, 27(2), 130-140. doi:10.1016/j.intmar.2012.12.001Bigné, E. (2016). Frontiers in research in business: Will you be in? European Journal of Management and Business Economics, 25(3), 89-90. doi:10.1016/j.redeen.2016.09.001Bigné, E., Llinares, C., & Torrecilla, C. (2016). Elapsed time on first buying triggers brand choices within a category: A virtual reality-based study. Journal of Business Research, 69(4), 1423-1427. doi:10.1016/j.jbusres.2015.10.119Blanco-Velasco, M., Weng, B., & Barner, K. E. (2008). ECG signal denoising and baseline wander correction based on the empirical mode decomposition. Computers in Biology and Medicine, 38(1), 1-13. doi:10.1016/j.compbiomed.2007.06.003Boksem, M. A. S., & Smidts, A. (2015). Brain Responses to Movie Trailers Predict Individual Preferences for Movies and Their Population-Wide Commercial Success. Journal of Marketing Research, 52(4), 482-492. doi:10.1509/jmr.13.0572Bradley, M. M., Houbova, P., Miccoli, L., Costa, V. D., & Lang, P. J. (2011). Scan patterns when viewing natural scenes: Emotion, complexity, and repetition. Psychophysiology, 48(11), 1544-1553. doi:10.1111/j.1469-8986.2011.01223.xCastiglioni, P., & Di Rienzo, M. (s. f.). On the evaluation of heart rate spectra: the Lomb periodogram. Computers in Cardiology 1996. doi:10.1109/cic.1996.542584Cha, M., Kwak, H., Rodriguez, P., Ahn, Y.-Y., & Moon, S. (2007). I tube, you tube, everybody tubes. Proceedings of the 7th ACM SIGCOMM conference on Internet measurement - IMC ’07. doi:10.1145/1298306.1298309Chen, L., Zhou, Y., & Chiu, D. M. (2014). A lifetime model of online video popularity. 2014 23rd International Conference on Computer Communication and Networks (ICCCN). doi:10.1109/icccn.2014.6911774Christoforou, C., Christou-Champi, S., Constantinidou, F., & Theodorou, M. (2015). From the eyes and the heart: a novel eye-gaze metric that predicts video preferences of a large audience. Frontiers in Psychology, 6. doi:10.3389/fpsyg.2015.00579Colomer Granero, A., Fuentes-Hurtado, F., Naranjo Ornedo, V., Guixeres Provinciale, J., Ausín, J. M., & Alcañiz Raya, M. (2016). A Comparison of Physiological Signal Analysis Techniques and Classifiers for Automatic Emotional Evaluation of Audiovisual Contents. Frontiers in Computational Neuroscience, 10. doi:10.3389/fncom.2016.00074Couwenberg, L. E., Boksem, M. A. S., Dietvorst, R. C., Worm, L., Verbeke, W. J. M. I., & Smidts, A. (2017). Neural responses to functional and experiential ad appeals: Explaining ad effectiveness. International Journal of Research in Marketing, 34(2), 355-366. doi:10.1016/j.ijresmar.2016.10.005Curry, B., & Moutinho, L. (1993). Neural Networks in Marketing: Modelling Consumer Responses to Advertising Stimuli. European Journal of Marketing, 27(7), 5-20. doi:10.1108/03090569310040325Daugherty, T., Hoffman, E., & Kennedy, K. (2016). Research in reverse: Ad testing using an inductive consumer neuroscience approach. Journal of Business Research, 69(8), 3168-3176. doi:10.1016/j.jbusres.2015.12.005Davidson, R. J. (2004). What does the prefrontal cortex «do» in affect: perspectives on frontal EEG asymmetry research. Biological Psychology, 67(1-2), 219-234. doi:10.1016/j.biopsycho.2004.03.008Deitz, G. D., Royne, M. B., Peasley, M. C., & Huang, J. «Coco». (2016). EEG-Based Measures versus Panel Ratings: Predicting Social-Media Based Behavioral Responses to Super Bowl Ads. Journal of Advertising Research, 56(2), 217. doi:10.2501/jar-2016-030Demarzo, M. M. P., Montero-Marin, J., Stein, P. K., Cebolla, A. s, Provinciale, J. G., & García-Campayo, J. (2014). Mindfulness may both moderate and mediate the effect of physical fitness on cardiovascular responses to stress: a speculative hypothesis. Frontiers in Physiology, 5. doi:10.3389/fphys.2014.00105Santos, R. D. O. J. dos, Oliveira, J. H. C. de, Rocha, J. B., & Giraldi, J. D. M. E. (2015). Eye Tracking in Neuromarketing: A Research Agenda for Marketing Studies. International Journal of Psychological Studies, 7(1). doi:10.5539/ijps.v7n1p32Elsen, M., Pieters, R., & Wedel, M. (2016). Thin Slice Impressions: How Advertising Evaluation Depends on Exposure Duration. Journal of Marketing Research, 53(4), 563-579. doi:10.1509/jmr.13.0398Feise, R. J. (2002). Do multiple outcome measures require p-value adjustment? BMC Medical Research Methodology, 2(1). doi:10.1186/1471-2288-2-8Fishman, M., Jacono, F. J., Park, S., Jamasebi, R., Thungtong, A., Loparo, K. A., & Dick, T. E. (2012). A method for analyzing temporal patterns of variability of a time series from Poincaré plots. Journal of Applied Physiology, 113(2), 297-306. doi:10.1152/japplphysiol.01377.2010Fjorback, L. O., Arendt, M., Ørnbøl, E., Fink, P., & Walach, H. (2011). Mindfulness-Based Stress Reduction and Mindfulness-Based Cognitive Therapy - a systematic review of randomized controlled trials. Acta Psychiatrica Scandinavica, 124(2), 102-119. doi:10.1111/j.1600-0447.2011.01704.xGao, J. F., Yang, Y., Lin, P., Wang, P., & Zheng, C. X. (2009). Automatic Removal of Eye-Movement and Blink Artifacts from EEG Signals. Brain Topography, 23(1), 105-114. doi:10.1007/s10548-009-0131-4Geisler, F. C. M., Vennewald, N., Kubiak, T., & Weber, H. (2010). The impact of heart rate variability on subjective well-being is mediated by emotion regulation. Personality and Individual Differences, 49(7), 723-728. doi:10.1016/j.paid.2010.06.015Goldberg, J. H., Stimson, M. J., Lewenstein, M., Scott, N., & Wichansky, A. M. (2002). Eye tracking in web search tasks. Proceedings of the symposium on Eye tracking research & applications - ETRA ’02. doi:10.1145/507072.507082Grandjean, D., Sander, D., & Scherer, K. R. (2008). Conscious emotional experience emerges as a function of multilevel, appraisal-driven response synchronization. Consciousness and Cognition, 17(2), 484-495. doi:10.1016/j.concog.2008.03.019Guerreiro, J., Rita, P., & Trigueiros, D. (2015). Attention, emotions and cause-related marketing effectiveness. European Journal of Marketing, 49(11/12), 1728-1750. doi:10.1108/ejm-09-2014-0543Ha, L. (2008). Online Advertising Research in Advertising Journals: A Review. Journal of Current Issues & Research in Advertising, 30(1), 31-48. doi:10.1080/10641734.2008.10505236Harmon-Jones, E., Gable, P. A., & Peterson, C. K. (2010). The role of asymmetric frontal cortical activity in emotion-related phenomena: A review and update. Biological Psychology, 84(3), 451-462. doi:10.1016/j.biopsycho.2009.08.010Holmqvist, K., Andrà, C., Lindström, P., Arzarello, F., Ferrara, F., Robutti, O., & Sabena, C. (2011). A method for quantifying focused versus overview behavior in AOI sequences. Behavior Research Methods, 43(4), 987-998. doi:10.3758/s13428-011-0104-xKent, R. J., & Allen, C. T. (1994). Competitive Interference Effects in Consumer Memory for Advertising: The Role of Brand Familiarity. Journal of Marketing, 58(3), 97. doi:10.2307/1252313Khushaba, R. N., Wise, C., Kodagoda, S., Louviere, J., Kahn, B. E., & Townsend, C. (2013). Consumer neuroscience: Assessing the brain response to marketing stimuli using electroencephalogram (EEG) and eye tracking. Expert Systems with Applications, 40(9), 3803-3812. doi:10.1016/j.eswa.2012.12.095Kim, K., Hayes, J. L., Avant, J. A., & Reid, L. N. (2014). Trends in Advertising Research: A Longitudinal Analysis of Leading Advertising, Marketing, and Communication Journals, 1980 to 2010. Journal of Advertising, 43(3), 296-316. doi:10.1080/00913367.2013.857620Kopton, I. M., & Kenning, P. (2014). Near-infrared spectroscopy (NIRS) as a new tool for neuroeconomic research. Frontiers in Human Neuroscience, 8. doi:10.3389/fnhum.2014.00549Kühn, S., Strelow, E., & Gallinat, J. (2016). Multiple «buy buttons» in the brain: Forecasting chocolate sales at point-of-sale based on functional brain activation using fMRI. NeuroImage, 136, 122-128. doi:10.1016/j.neuroimage.2016.05.021Lang, A., Bolls, P., Potter, R. F., & Kawahara, K. (1999). The effects of production pacing and arousing content on the information processing of television messages. Journal of Broadcasting & Electronic Media, 43(4), 451-475. doi:10.1080/08838159909364504Lee, J., & Ahn, J.-H. (2012). Attention to Banner Ads and Their Effectiveness: An Eye-Tracking Approach. International Journal of Electronic Commerce, 17(1), 119-137. doi:10.2753/jec1086-4415170105McAlister, L., Srinivasan, R., Jindal, N., & Cannella, A. A. (2016). Advertising Effectiveness: The Moderating Effect of Firm Strategy. Journal of Marketing Research, 53(2), 207-224. doi:10.1509/jmr.13.0285McDuff, D., Kaliouby, R. E., Cohn, J. F., & Picard, R. W. (2015). Predicting Ad Liking and Purchase Intent: Large-Scale Analysis of Facial Responses to Ads. IEEE Transactions on Affective Computing, 6(3), 223-235. doi:10.1109/taffc.2014.2384198Mognon, A., Jovicich, J., Bruzzone, L., & Buiatti, M. (2011). ADJUST: An automatic EEG artifact detector based on the joint use of spatial and temporal features. Psychophysiology, 48(2), 229-240. doi:10.1111/j.1469-8986.2010.01061.xBabiloni, F. (2012). Consumer Nueroscience: A New Area of Study for Biomedical Engineers. IEEE Pulse, 3(3), 21-23. doi:10.1109/mpul.2012.2189166Mould, D., Mandryk, R. L., & Li, H. (2012). Emotional response and visual attention to non-photorealistic images. Computers & Graphics, 36(6), 658-672. doi:10.1016/j.cag.2012.03.039Cartocci, G., Caratù, M., Modica, E., Maglione, A. G., Rossi, D., Cherubino, P., & Babiloni, F. (2017). Electroencephalographic, Heart Rate, and Galvanic Skin Response Assessment for an Advertising Perception Study: Application to Antismoking Public Service Announcements. Journal of Visualized Experiments, (126). doi:10.3791/55872Pan, J., & Tompkins, W. J. (1985). A Real-Time QRS Detection Algorithm. IEEE Transactions on Biomedical Engineering, BME-32(3), 230-236. doi:10.1109/tbme.1985.325532Pieters, R., Warlop, L., & Wedel, M. (2002). Breaking Through the Clutter: Benefits of Advertisement Originality and Familiarity for Brand Attention and Memory. Management Science, 48(6), 765-781. doi:10.1287/mnsc.48.6.765.192Piskorski, J., & Guzik, P. (2007). Geometry of the Poincaré plot ofRRintervals and its asymmetry in healthy adults. Physiological Measurement, 28(3), 287-300. doi:10.1088/0967-3334/28/3/005Richman, J. S., & Moorman, J. R. (2000). Physiological time-series analysis using approximate entropy and sample entropy. American Journal of Physiology-Heart and Circulatory Physiology, 278(6), H2039-H2049. doi:10.1152/ajpheart.2000.278.6.h2039Ruanguttamanun, C. (2014). Neuromarketing: I Put Myself into a fMRI Scanner and Realized that I love Louis Vuitton Ads. Procedia - Social and Behavioral Sciences, 148, 211-218. doi:10.1016/j.sbspro.2014.07.036Shehu, E., Bijmolt, T. H. A., & Clement, M. (2016). Effects of Likeability Dynamics on Consumers’ Intention to Share Online Video Advertisements. Journal of Interactive Marketing, 35, 27-43. doi:10.1016/j.intmar.2016.01.001Smith, A. N., Fischer, E., & Yongjian, C. (2012). How Does Brand-related User-generated Content Differ across YouTube, Facebook, and Twitter? Journal of Interactive Marketing, 26(2), 102-113. doi:10.1016/j.intmar.2012.01.002Strach, P., Zuber, K., Fowler, E. F., Ridout, T. N., & Searles, K. (2015). In a Different Voice? Explaining the Use of Men and Women as Voice-Over Announcers in Political Advertising. Political Communication, 32(2), 183-205. doi:10.1080/10584609.2014.914614Malik, M., Bigger, J. T., Camm, A. J., Kleiger, R. E., Malliani, A., Moss, A. J., & Schwartz, P. J. (1996). Heart rate variability: Standards of measurement, physiological interpretation, and clinical use. European Heart Journal, 17(3), 354-381. doi:10.1093/oxfordjournals.eurheartj.a014868Tomkovick, C., Yelkur, R., & Christians, L. (2001). The USA’s biggest marketing event keeps getting bigger: an in-depth look at Super Bowl advertising in the 1990s. Journal of Marketing Communications, 7(2), 89-108. doi:10.1080/13527260121725Vakratsas, D., & Ambler, T. (1999). How Advertising Works: What Do We Really Know? Journal of Marketing, 63(1), 26. doi:10.2307/1251999Valenza, G., Allegrini, P., Lanatà, A., & Scilingo, E. P. (2012). Dominant Lyapunov exponent and approximate entropy in heart rate variability during emotional visual elicitation. Frontiers in Neuroengineering, 5. doi:10.3389/fneng.2012.00003Valenza, G., Citi, L., Lanatá, A., Scilingo, E. P., & Barbieri, R. (2014). Revealing Real-Time Emotional Responses: a Personalized Assessment based on Heartbeat Dynamics. Scientific Reports, 4(1). doi:10.1038/srep04998Varan, D., Lang, A., Barwise, P., Weber, R., & Bellman, S. (2015). How Reliable Are Neuromarketers’ Measures of Advertising Effectiveness? Journal of Advertising Research, 55(2), 176-191. doi:10.2501/jar-55-2-176-191Vecchiato, G., Astolfi, L., Tabarrini, A., Salinari, S., Mattia, D., Cincotti, F., … Babiloni, F. (2010). EEG Analysis of the Brain Activity during the Observation of Commercial, Political, or Public Service Announcements. Computational Intelligence and Neuroscience, 2010, 1-7. doi:10.1155/2010/985867Vecchiato, G., Maglione, A. G., Cherubino, P., Wasikowska, B., Wawrzyniak, A., Latuszynska, A., … Babiloni, F. (2014). Neurophysiological Tools to Investigate Consumer’s Gender Differences during the Observation of TV Commercials. Computational and Mathematical Methods in Medicine, 2014, 1-12. doi:10.1155/2014/912981Vecchiato, G., Susac, A., Margeti, S., De Vico Fallani, F., Maglione, A. G., Supek, S., … Babiloni, F. (2012). High-Resolution EEG Analysis of Power Spectral Density Maps and Coherence Networks in a Proportional Reasoning Task. Brain Topography, 26(2), 303-314. doi:10.1007/s10548-012-0259-5Vecchiato, G., Toppi, J., Astolfi, L., De Vico Fallani, F., Cincotti, F., Mattia, D., … Babiloni, F. (2011). Spectral EEG frontal asymmetries correlate with the experienced pleasantness of TV commercial advertisements. Medical & Biological Engineering & Computing, 49(5), 579-583. doi:10.1007/s11517-011-0747-xVenkatraman, V., Dimoka, A., Pavlou, P. A., Vo, K., Hampton, W., Bollinger, B., … Winer, R. S. (2015). Predicting Advertising success beyond Traditional Measures: New Insights from Neurophysiological Methods and Market Response Modeling. Journal of Marketing Research, 52(4), 436-452. doi:10.1509/jmr.13.0593Wackermann, J., Lehmann, D., Michel, C. M., & Strik, W. K. (1993). Adaptive segmentation of spontaneous EEG map series into spatially defined microstates. International Journal of Psychophysiology, 14(3), 269-283. doi:10.1016/0167-8760(93)90041-mWedel, M., & Kannan, P. K. (2016). Marketing Analytics for Data-Rich Environments. Journal of Marketing, 80(6), 97-121. doi:10.1509/jm.15.0413Werkle-Bergner, M., Müller, V., Li, S.-C., & Lindenberger, U. (2006). Cortical EEG correlates of successful memory encoding: Implications for lifespan comparisons. Neuroscience & Biobehavioral Reviews, 30(6), 839-854. doi:10.1016/j.neubiorev.2006.06.009West, P. M., Brockett, P. L., & Golden, L. L. (1997). A Comparative Analysis of Neural Networks and Statistical Methods for Predicting Consumer Choice. Marketing Science, 16(4), 370-391. doi:10.1287/mksc.16.4.370Zhou, R., Khemmarat, S., Gao, L., Wan, J., & Zhang, J. (2016). How YouTube videos are discovered and its impact on video views. Multimedia Tools and Applications, 75(10), 6035-6058. doi:10.1007/s11042-015-3206-

Affect-based information retrieval

One of the main challenges Information Retrieval (IR) systems face nowadays originates from the semantic gap problem: the semantic difference between a user’s query representation and the internal representation of an information item in a collection. The gap is further widened when the user is driven by an ill-defined information need, often the result of an anomaly in his/her current state of knowledge. The formulated search queries, which are submitted to the retrieval systems to locate relevant items, produce poor results that do not address the users’ information needs. To deal with information need uncertainty IR systems have employed in the past a range of feedback techniques, which vary from explicit to implicit. The first category of feedback techniques necessitates the communication of explicit relevance judgments, in return for better query reformulations and recommendations of relevant results. However, the latter happens at the expense of users’ cognitive resources and, furthermore, introduces an additional layer of complexity to the search process. On the other hand, implicit feedback techniques make inferences on what is relevant based on observations of user search behaviour. By doing so, they disengage users from the cognitive burden of document rating and relevance assessments. However, both categories of RF techniques determine topical relevance with respect to the cognitive and situational levels of interaction, failing to acknowledge the importance of emotions in cognition and decision making. In this thesis I investigate the role of emotions in the information seeking process and develop affective feedback techniques for interactive IR. This novel feedback framework aims to aid the search process and facilitate a more natural and meaningful interaction. I develop affective models that determine topical relevance based on information gathered from various sensory channels, and enhance their performance using personalisation techniques. Furthermore, I present an operational video retrieval system that employs affective feedback to enrich user profiles and offers meaningful recommendations of unseen videos. The use of affective feedback as a surrogate for the information need is formalised as the Affective Model of Browsing. This is a cognitive model that motivates the use of evidence extracted from the psycho-somatic mobilisation that occurs during cognitive appraisal. Finally, I address some of the ethical and privacy issues that arise from the social-emotional interaction between users and computer systems. This study involves questionnaire data gathered over three user studies, from 74 participants of different educational background, ethnicity and search experience. The results show that affective feedback is a promising area of research and it can improve many aspects of the information seeking process, such as indexing, ranking and recommendation. Eventually, it may be that relevance inferences obtained from affective models will provide a more robust and personalised form of feedback, which will allow us to deal more effectively with issues such as the semantic gap

A Visual Approach to Improving the Experience of Health Information for Vulnerable Individuals

Many individuals with low health literacy (LHL) and limited English proficiency (LEP) have poor experiences consuming health information: they find it unengaging, unappealing, difficult to understand, and un-motivating. These negative experiences may blunt, or even sabotage, the desired effect of communicating health information: to increase engagement and ability to manage health. It is imperative to find solutions to improve poor experiences of health information, because such experiences heighten vulnerability to poor health outcomes. We aimed to address a gap in the health literacy literature by studying the patient experience of health information and how visualization might be able to help. Our four studies involved patients presented with health information in various settings to improve understanding and management of their care. We used semi-structured interviews and observations to understand patient experiences of receiving personal health information in the hospital. We learned that the return of results is desired and has the potential to promote patient engagement with care. We developed a novel method to analyze LHL, LEP caregiver experience and information needs in the community setting. The novel method increased our understanding and ability to detect differences in experiences within the same ethnic group, based on language preference. Next, we interrogated the literature for a solution to easily communicate complicated health information to disinterested, LHL, LEP individuals. We found that visualizations can help increase interest, comprehension, support faster communication, and even help broach difficult topics. Finally, our findings were used to develop a novel prototype to improve experiences of consuming genetic risk information for those having LHL and LEP. Unlike traditional approaches that focus on communicating risk numbers and probabilities, the novelty of our approach was that we focused on communicating risk as a feeling. We achieved this by leveraging vicarious learning via real patient experience materials (e.g., quotes, videos) and empathy with an emotive relational agent. We evaluated and compared the prototype to standard methods of communicating genetic risk information via a mixed methods approach that included surveys, questionnaires, interviews, observations, image analysis, and facial analysis. Main outcome variables were perceived ease of understanding, comprehension, emotional response, and motivation. We employed t-tests, ANOVAs, directed content analysis, correlation, regression, hierarchical clustering, and Chernoff faces to answer the research questions. All variables were significantly different for the prototype compared to the standard method, except for motivation as rated by 32 LHL, LEP community members. Findings revealed that LHL, LEP individuals have difficulty appropriately processing standard methods of communicating risk information, such as risk numbers supported by visual aids. Further, appealing visuals may inappropriately increase confidence in understanding of information. Visualizations affected emotions, which influenced perceived ease of understanding and motivation to take action on the information. Comprehension scores did not correlate with perceived ease of understanding, emotional response, or motivation. Findings suggest that providing access to comprehensible health information may not be enough to motivate patients to engage with their care; providing a good experience (taking into account the aesthetics and emotional response) of health information may be essential to optimize outcomes

Artificial intelligence applied to marketing management: Trends and projections according to specialists

Marketing Management is one of the areas that has been progressively integrating artificial intelligence systems, and the pace of the development of intelligent software that is very useful for marketing seems not to slow down. In fact, the growth and sophistication of technological systems promise to increase even more, which will inevitably affect operations as well as management and planning. In an attempt to assess and measure the expected impacts of AI on marketing departments in the short / medium term, a Delphi was carried out. Thereby, a panel of 21 marketing specialists (13 Portuguese and 8 international) was gathered, which was asked to evaluate on a Likert scale a series of statements, and to comment and debate among them. In this case it was a Real Time Delphi since the study was conducted using an online platform, which allowed all comments to be immediately available and visible to all participants. With this exploratory study, it was possible to conclude that the areas that are expected to be helped by intelligent systems to a greater extent – this is, the areas that will assist the automation of more operations - are customer recognition , market segmentation, sales forecasting and programmatic communication. On the other hand, the two most controversial statements among experts - thus risky to draw lessons - were statements regarding the autonomous operation of website adjustments and developments, as well as the adoption of intelligent systems to support strategic and strategic decision-making.A Gestão de Marketing é uma das áreas que tem vindo progressivamente a integrar sistemas de inteligência artificial, e a cadência do desenvolvimento de softwares inteligentes com grande utilidade para parece não abrandam. Na verdade, o crescimento e o grau de sofisticação dos sistemas tecnológicos prometem aumentar cada vez mais, o que promete afetar a vários níveis as operações e até a definição de estratégias de marketing e de gestão. Na tentativa de avaliar e medir os impactos da inteligência artificial nos departamentos de marketing no curto/médio prazo, procedeu-se à realização de um Delphi. Para isso reuniu-se um painel de 21 especialistas na área do marketing e da inteligência artificial (13 portugueses e 8 internacionais), ao qual foi colocada uma série de afirmações para que fossem avaliadas numa escala de Likert, comentadas e debatidas. Neste caso tratou-se de um Real Time Delphi uma vez que o estudo foi realizado recorrendo a uma plataforma online, o que permitiu que todos comentários ficassem imediatamente disponíveis e visíveis a todos os participantes. Com este estudo, de cariz marcadamente exploratório, concluiu-se que as áreas que se esperam vir a ser auxiliadas por sistemas inteligentes em maior medida – ou seja, as áreas que assistirão à automatização de um maior número de operações – são o reconhecimento do cliente, segmentação de mercado, previsão de vendas e comunicação programática. Por outro lado, os temas que mais controvérsia geraram entre os especialistas – sendo pouco seguro retirar ilações – referem-se à operação autónoma de ajustes e desenvolvimentos de websites, bem como à adoção de sistemas inteligentes para servirem de apoio à tomada de decisões estratégicas e de planeamento