Why, When, and How Much to Entertain Consumers in Advertisements? A Web-Based Facial Tracking Field Study

The presence of positive entertainment (e.g., visual imagery, upbeat music, humor) in TV advertisements can make them more attractive and persuasive. However, little is known about the downside of too much entertainment. This research focuses on why, when, and how much to entertain consumers in TV advertisements. We collected data in a large scale field study using 82 ads with various levels of entertainment shown to 178 consumers in their homes and workplaces. Using a novel web-based face tracking system, we continuously measure consumers' smile responses, viewing interest, and purchase intent. A simultaneous Bayesian hierarchical model is estimated to assess how different levels of entertainment affect purchases by endogenizing viewing interest. We find that entertainment has an inverted U-shape relationship to purchase intent. Importantly, we separate entertainment into that which comes before the brand versus that which comes after, and find that the latter is positively associated with purchase intent while the former is not

Crowdsourced data collection of facial responses

In the past, collecting data to train facial expression and affect recognition systems has been time consuming and often led to data that do not include spontaneous expressions. We present the first crowdsourced data collection of dynamic, natural and spontaneous facial responses as viewers watch media online. This system allowed a corpus of 3,268 videos to be collected in under two months. We characterize the data in terms of viewer demographics, position, scale, pose and movement of the viewer within the frame, and illumination of the facial region. We compare statistics from this corpus to those from the CK+ and MMI databases and show that distributions of position, scale, pose, movement and luminance of the facial region are significantly different from those represented in these datasets. We demonstrate that it is possible to efficiently collect massive amounts of ecologically valid responses, to known stimuli, from a diverse population using such a system. In addition facial feature points within the videos can be tracked for over 90% of the frames. These responses were collected without need for scheduling, payment or recruitment. Finally, we describe a subset of data (over 290 videos) that will be available for the research community.Things That Think ConsortiumProcter & Gamble Compan

Acume: A New Visualization Tool for Understanding Facial Expression and Gesture Data

Facial and head actions contain significant affective information. To date, these actions have mostly been studied in isolation because the space of naturalistic combinations is vast. Interactive visualization tools could enable new explorations of dynamically changing combinations of actions as people interact with natural stimuli. This paper describes a new open-source tool that enables navigation of and interaction with dynamic face and gesture data across large groups of people, making it easy to see when multiple facial actions co-occur, and how these patterns compare and cluster across groups of participants. We share two case studies that demonstrate how the tool allows researchers to quickly view an entire corpus of data for single or multiple participants, stimuli and actions. Acume yielded patterns of actions across participants and across stimuli, and helped give insight into how our automated facial analysis methods could be better designed. The results of these case studies are used to demonstrate the efficacy of the tool. The open-source code is designed to directly address the needs of the face and gesture research community, while also being extensible and flexible for accommodating other kinds of behavioral data. Source code, application and documentation are available at http://affect.media.mit.edu/acume.Procter & Gamble Compan

Predicting Online Media Effectiveness Based on Smile Responses Gathered Over the Internet

We present an automated method for classifying “liking” and “desire to view again” based on over 1,500 facial responses to media collected over the Internet. This is a very challenging pattern recognition problem that involves robust detection of smile intensities in uncontrolled settings and classification of naturalistic and spontaneous temporal data with large individual differences. We examine the manifold of responses and analyze the false positives and false negatives that result from classification. The results demonstrate the possibility for an ecologically valid, unobtrusive, evaluation of commercial “liking” and “desire to view again”, strong predictors of marketing success, based only on facial responses. The area under the curve for the best “liking” and “desire to view again” classifiers was 0.8 and 0.78 respectively when using a challenging leave-one-commercial-out testing regime. The technique could be employed in personalizing video ads that are presented to people whilst they view programming over the Internet or in copy testing of ads to unobtrusively quantify effectiveness.MIT Media Lab Consortiu

Measuring Voter's Candidate Preference Based on Affective Responses to Election Debates

In this paper we present the first analysis of facial responses to electoral debates measured automatically over the Internet. We show that significantly different responses can be detected from viewers with different political preferences and that similar expressions at significant moments can have very different meanings depending on the actions that appear subsequently. We used an Internet based framework to collect 611 naturalistic and spontaneous facial responses to five video clips from the 3rd presidential debate during the 2012 American presidential election campaign. Using this framework we were able to collect over 60% of these video responses (374 videos) within one day of the live debate and over 80% within three days. No participants were compensated for taking the survey. We present and evaluate a method for predicting independent voter preference based on automatically measured facial responses and self-reported preferences from the viewers. We predict voter preference with an average accuracy of over 73% (AUC 0.779)

Multi-modal fusion methods for robust emotion recognition using body-worn physiological sensors in mobile environments

High-accuracy physiological emotion recognition typically requires participants to wear or attach obtrusive sensors (e.g., Electroencephalograph). To achieve precise emotion recognition using only wearable body-worn physiological sensors, my doctoral work focuses on researching and developing a robust sensor fusion system among different physiological sensors. Developing such fusion system has three problems: 1) how to pre-process signals with different temporal characteristics and noise models, 2) how to train the fusion system with limited labeled data and 3) how to fuse multiple signals with inaccurate and inexact ground truth. To overcome these challenges, I plan to explore semi-supervised, weakly supervised and unsupervised machine learning methods to obtain precise emotion recognition in mobile environments. By developing such techniques, we can measure the user engagement with larger amounts of participants and apply the emotion recognition techniques in a variety of scenarios such as mobile video watching and online education

Predicting Ad Liking and Purchase Intent: Large-Scale Analysis of Facial Responses to Ads

Billions of online video ads are viewed every month. We present a large-scale analysis of facial responses to video content measured over the Internet and their relationship to marketing effectiveness. We collected over 12,000 facial responses from 1,223 people to 170 ads from a range of markets and product categories. The facial responses were automatically coded frame-by-frame. Collection and coding of these 3.7 million frames would not have been feasible with traditional research methods. We show that detected expressions are sparse but that aggregate responses reveal rich emotion trajectories. By modeling the relationship between the facial responses and ad effectiveness, we show that ad liking can be predicted accurately (ROC AUC = 0.85) from webcam facial responses. Furthermore, the prediction of a change in purchase intent is possible (ROC AUC = 0.78). Ad liking is shown by eliciting expressions, particularly positive expressions. Driving purchase intent is more complex than just making viewers smile: peak positive responses that are immediately preceded by a brand appearance are more likely to be effective. The results presented here demonstrate a reliable and generalizable system for predicting ad effectiveness automatically from facial responses without a need to elicit self-report responses from the viewers. In addition we can gain insight into the structure of effective ads.MIT Media Lab ConsortiumNEC CorporationMAR

Recycling of Ornamental Stones Hazardous Wastes

Sawing and polishing of the ornamental stones always generate large amount of solid and wet hazardous wastes, which pollute the environment. In Shak Al-Thoaban area, East Cairo, Egypt, huge amounts of these wastes were accumulated, during the last years, as rejects “Solid” and wet “Sahala” wastes, representing one of the main sources of environmentalpollution. The aim of this work is to characterize and evaluate these wastes for recycling in quicklime production. Hence, samples of both wastes were investigated for their chemical and mineral composition applying XRF, XRD, DTA and TGA methods. Free lime content and reactivity (RDIN) of both samples were also determined after calcination for differnt soaking times (0.25 - 2.0 h) at 1000˚C. The results were interpreted in relation to composition and microstructure of the fired samples as revealed by TLM and SEM methods. The RDIN reactivity of the resulted lime is changeable along soaking time at 1000˚C because of the microfabric of its crystallites. The lime of the “Solid” sample is preserving the original limestone microstructure that contributes in its higher RDIN reactivity values at all soaking times. The relatively higher degree of grain growth of lime crystallites in the “Sahala” sample leads to its lower reactivity.The optimum soaking times for the highest lime reactivity are 0.25 and 1 h for the “Solid” and “Sahala” samples, respectively. On increasing soaking time up to 2 h, both samples show minimum RDIN values. The “Solid” sample also gives higher free lime content than the “Sahala” one at all soaking times. It is gradually increased in the former sample up to a maximum (96% - 97%) on increasing soaking time up to 1 - 2 h. On the other side, a maximum free lime (~95%) is detected in “Sahala” sample at 0.25 h soaking time and gradually decreased to (87%) up to 2 h

Affectiva-MIT Facial Expression Dataset (AM-FED): Naturalistic and Spontaneous Facial Expressions Collected In-the-Wild

Computer classification of facial expressions requires large amounts of data and this data needs to reflect the diversity of conditions seen in real applications. Public datasets help accelerate the progress of research by providing researchers with a benchmark resource. We present a comprehensively labeled dataset of ecologically valid spontaneous facial responses recorded in natural settings over the Internet. To collect the data, online viewers watched one of three intentionally amusing Super Bowl commercials and were simultaneously filmed using their webcam. They answered three self-report questions about their experience. A subset of viewers additionally gave consent for their data to be shared publicly with other researchers. This subset consists of 242 facial videos (168,359 frames) recorded in real world conditions. The dataset is comprehensively labeled for the following: 1) frame-by-frame labels for the presence of 10 symmetrical FACS action units, 4 asymmetric (unilateral) FACS action units, 2 head movements, smile, general expressiveness, feature tracker fails and gender; 2) the location of 22 automatically detected landmark points; 3) self-report responses of familiarity with, liking of, and desire to watch again for the stimuli videos and 4) baseline performance of detection algorithms on this dataset. This data is available for distribution to researchers online, the EULA can be found at: http://www.affectiva.com/facial-expression-dataset-am-fed/

Real-Time Inference of Mental States from Facial Expressions and Upper Body Gestures

We present a real-time system for detecting facial action units and inferring emotional states from head and shoulder gestures and facial expressions. The dynamic system uses three levels of inference on progressively longer time scales. Firstly, facial action units and head orientation are identified from 22 feature points and Gabor filters. Secondly, Hidden Markov Models are used to classify sequences of actions into head and shoulder gestures. Finally, a multi level Dynamic Bayesian Network is used to model the unfolding emotional state based on probabilities of different gestures. The most probable state over a given video clip is chosen as the label for that clip. The average F1 score for 12 action units (AUs 1, 2, 4, 6, 7, 10, 12, 15, 17, 18, 25, 26), labelled on a frame by frame basis, was 0.461. The average classification rate for five emotional states (anger, fear, joy, relief, sadness) was 0.440. Sadness had the greatest rate, 0.64, anger the smallest, 0.11.Thales Research and Technology (UK)Bradlow Foundation TrustProcter & Gamble Compan