We describe a study conducted to investigate the potential

correlations between human subject responses to emotional stimuli in movies, and observed biometric responses. The experimental set-up and procedure are described, including details of the range of sensors used to detect and record observed physiological data (such as heart-rate, galvanic skin response, body temperature and movement). Finally, applications and future analysis of the results of the study are discussed

Chan, Ching Hau

Diamond, Dermot

Jones, Gareth J.F.

Lehane, Bart

O'Connor, Noel E.

Rothwell, Sandra

Smeaton, Alan F.

English

We describe a study conducted to investigate the potential\ud
correlations between human subject responses to emotional stimuli in movies, and observed biometric responses. The experimental set-up and procedure are described, including details of the range of sensors used to detect and record observed physiological data (such as heart-rate, galvanic skin response, body temperature and movement). Finally, applications and future analysis of the results of the study are discussed

Name not available

The CDVPlex Biometric Cinema: SensingPhysiological Responses to Emotional Stimuli inFilmSandra Rothwell, Bart Lehane, Ching Hau Chan, Alan F. Smeaton, Noel E.O’Connor, Gareth J.F. Jones and Dermot DiamondAdaptive Information Cluster,Dublin City University, Glasnevin, Dublin 9, Ireland.Sandra.Rothwell@computing.dcu.ieAbstract. We describe a study conducted to investigate the potentialcorrelations between human subject responses to emotional stimuli inmovies, and observed biometric responses. The experimental set-up andprocedure are described, including details of the range of sensors usedto detect and record observed physiological data (such as heart-rate,galvanic skin response, body temperature and movement). Finally, ap-plications and future analysis of the results of the study are discussed.1 IntroductionPhysiological responses to emotional stimuli and to exercise are quite similar,and can be recorded and studied using a range of biometric measurement tools.By measuring and recording the physiological reactions of movie viewers as theywatch a film, we can use this to explore new dimensions in automatic analysisof video. In this study we used new sensor technology and adapted establishedbiometric measurement techniques originally developed by researchers in healthand sports science. Using these methods we have gathered data, and are now at-tempting to detect the emotional highlights in a film, concentrating on highlightsin film music and film events. This allows us to correlate the two, which shouldhelp to validate the accuracy of the observed data. This paper does not presentan analysis of observed responses, only the details of how data was gathered.2 Experimental Set-UpThe aim of this experiment was to provide a ground-truth of movie viewers’physiological reactions over multiple full-length feature films, by gathering bio-metric data from viewers in a controlled cinema-like environment known asthe“CDVPlex”. In total, 37 movies were shown over a 10-week period resulting inover 500 hours of recorded biometric observations, half as baseline observations,and half involving people watching movies. The 37 full-length feature films usedin this study were chosen to encompass a range across 7 decades (1939–2004)and 10 movie genres (see Table 1).Tom Pfeifer et al. (Eds.): Advances in Pervasive Computing 2006Adjunct Proceedings of Pervasive 2006, Dublin, 7-10 Maybooks@OCG.at Vol. 207, ISBN 3-85403-207-2Movie Genre Sample MoviesAction/Adventure Indiana Jones, Lawrence of ArabiaAnimation Finding Nemo, ShrekComedy Who Framed Roger Rabbit?, Toy StoryDocumentary Roger and MeDrama The Godfather, Pulp FictionForeign/World Cinema Das Boot, Il PostinoHorror/Suspense Alien, The ChangelingRomance Casablanca, The English PatientScience Fiction/Fantasy Star Wars, The Lord of the RingsThriller/Mystery Minority Report, Clockers, LeonTable 1. Some of the movies used in the CDVPlex experimentThe participants in this experiment were 43 volunteers from across the univer-sity’s population of staff and graduate students. Film showings were advertiseddaily using posters and email bulletins. An online booking system and websitewere provided where interested parties could view the schedule and the avail-ability of places, and book a place for viewing a film. Participation was limitedto between 2 and 4 movies per person.In order to obtain as accurate results as possible our aim was to encourageviewers to relax and to essentially forget that they were participating in a lab-oratory experiment. Therefore it was decided to recreate an environment thatwas as close to a true cinematic experience as possible. The CDVPlex was con-structed in an air-conditioned windowless room in which a 5.1 surround soundsystem, DVD player and large-screen digital projector were installed. It seatedup to four people and the movie was rear-projected onto a large screen at one endof the room. Each of the chairs was outfitted as a Smart Chair (see Section 3).On the day of viewing, subjects wore two portable sensors (a heart-ratemonitor and a BodyMedia Armband, see section 3) from 11am for approximately5 to 6 hours or until after the film ended. The data recorded in the hours leadingup to the film viewing established a baseline of biometric responses for thatperson. Subjects were asked not to engage in strenuous activities such as physicalexercise (e.g. jogging) during this period, which would skew the readings.On entering the CDVPlex at 2pm, viewers completed a short questionnairedetailing some personal information such as age and gender, their cinema at-tendance and preferred film genres, and also whether they had previously seenthe film they were about to watch. The procedure was explained and at 2:30pmthe room was darkened, the film was started with each viewer synchronising thetimestamp buttons of both wearable devices. At the end of the film, once thecredits had finished rolling, the timestamps were again synchronised. Partici-pants were then asked to complete another short questionnaire on which aspectsof the film they most enjoyed (plot, music, etc) and were asked to give their topfive highlights from the film and how each made them feel. The devices werethen removed and the biometric data downloaded and stored.1043 Sensors Used in the CDVPlexThere were three different sensor devices used to gather observational data:Heart-rate monitor: the Polar S610iTM heart-rate monitor is a devicewhich consists of a fabric band which fits around a person’s chest and detectstheir heartbeat. It stores its measurements by transmitting them to an accom-panying wristwatch and thus tracks a person’s heart-rate over time. Heart-rateresponds quickly to stimuli and fluctuations of more than ± 10% of the b.p.m.value can be interpreted as a response to a stimulus [1]. Large spikes in the heart-rate graph (for example, a subject’s heart-rate jumping by greater than 20% andreturning to normal level within one minute) can indicate that the subject wassurprised or shocked. Similarly, a sustained period with an increased heart-ratecan indicate continuous excitement.BodyMedia Armband Sensor Array: the BodyMedia SenseWear R© PRO2is a light-weight and unobtrusive armband worn on the right upper arm. Eachwas configured to record and store up to 6.5 hours of biometric data as detectedby an array of onboard sensors. For our experiments we chose to record galvanicskin response, a measure of skin conductivity which is affected by perspirationsuch as that caused by emotional stimuli. Changes in the eccrine sweat glandscausing perspiration have been linked to measures of emotion, arousal and at-tention [2]. We also recorded skin temperature, which is linearly reflective of thebody’s core temperature activities, as well as heat flux which is the rate of heatbeing dissipated by the body. Changes in skin temperature can be due to anemotional or physical response but typically they take longer to manifest thanheart-rate changes, and fluctuations will not be as great. The final component inthe BodyMedia armband we used was an accelerometer which measures motion,and by taking into account the context in which the armband was worn (e.g.lying down, etc.), the amount of subject movement can be derived.Smart Chairs: each of the four chairs used in the experiment had a foam-based pressure sensor integrated into its backrest to record movements andchanges in the subject’s posture. This uses a polypyrrole-coated polyurethanefoam developed within our sensors laboratory [3], the electrical conductivity ofwhich changes when it is compressed. Movements such as sitting back, forward orshifting from side to side in the chair result in pressure changes on the backrest,causing the polyurethane foam to compress and relax and the resulting changesin the electrical current were recorded. This allowed for fine-grained monitoringof subjects’ movements throughout the film.4 Data Analysis and Next StepsHaving gathered over 500 hours of biometric observational data, we have nowstarted to analyse the huge volume produced by this experiment, to allow forstatistical analysis. We are presently examining different machine learning tech-niques that may aid in detecting trends within the data. Detecting emotionalhighlights in the film (i.e. periods of high emotional stimulation) is possible;105however we cannot, as yet, distinguish between positive and negative emotions,though this will be an area for future work.The main focus for the analysis of the data gathered will be in detecting movieevents and highlights. There are three different areas we are concentrating on.Movie MusicOne of our interests is to investigate the relationship between film music andthe emotional experience of movie viewing [4]. Our focus is on automaticallyanalysing a movie soundtrack to identify patterns of musical characteristics basedon pitch, beat and timbre in order to subdivide the soundtrack (and hence themovie) into logical segments based on the emotion suggested by the associatedmusic. We can then examine correlations between emotional physiological eventsas observed in the CDVPlex and emotional segments in the movies.Movie Event TypesBy automatically detecting the presence of various event types (such as conver-sations [5], exciting events [6] or montages) in each movie, we can investigatehow audience reaction differs depending on the on-screen activities. For example,of fast-paced editing and high amounts of camera movement is regularly used byfilm-makers to portray intensity [7]. By correlating areas such as these with thebiometric observations, the physiological response of viewers can be detected.Emotive AnnotationIn this work we investigate correlations between biometric measurements fromthe CDVPlex and automatically generated affective content annotations of videobased on its audio content. This is achieved by analysing the language used inAudio Descriptions of the same movies shown in the CDVPlex.Acknowledgments: This work is supported by Science Foundation Irelandunder Grant No. 03/IN.3/I361, and by the Irish Research Council for Science,Engineering and Technology.References1. M. Myrtek, “Heart and Emotion”, Hogrefe and Huber, 2004.2. J. Wagner, J. Kim and E. Andre´, “From Physiological Signals to Emotions: Imple-menting and Comparing Selected Methods for Feature Extraction and Classifica-tion”, In Proc. IEEE International Conference on Multimedia and Expo (ICME),2005.3. S. Brady, D. Diamond and K.-T. Lau, “Inherently Conducting Polymer ModifiedPolyurethane Smart Foam for Pressure Sensing”, in: Sensors and Actuators A:Physical, Vol. A119, No. 2, pp. 398-404, 13 April 2005.4. C. Gorbman, “Unheard Melodies”, Indiana Press, 1987.5. B. Lehane, N.E. O’Connor and N. Murphy, “Dialogue Sequence Detection inMovies”, in: Proc. International Conference on Image and Video Retrieval, LNCS3569, pp286-296, Singapore, 20-22 July 2005.6. B. Lehane, N.E O’Connor and N. Murphy, “Action Sequence Detection in MotionPictures”, in: Proc. European Workshop on the Integration of Knowledge, Semanticsand Digital Media Technology, London, 25-26 November 2004.7. D. Bordwell and K. Thompson, “Film Art: An Introduction”, McGraw-Hill, 1997.106

The CDVPlex biometric cinema: sensing physiological responses to emotional stimuli in film

DCU Online Research Access Service

Action Sequence Detection in Motion Pictures&quot;, in:

Andr¶ e, \From Physiological Signals to Emotions: Implementing and Comparing Selected Methods for Feature Extraction and Classi¯cation&quot;,

Dialogue Sequence Detection in Movies&quot;, in:

Film Art: An Introduction&quot;,

Heart and Emotion&quot;, Hogrefe and Huber,

Inherently Conducting Polymer Modi¯ed Polyurethane Smart Foam for Pressure Sensing&quot;, in:

Unheard Melodies&quot;,

We describe a study conducted to investigate the potential
correlations between human subject responses to emotional stimuli in movies, and observed biometric responses. The experimental set-up and procedure are described, including details of the range of sensors used to detect and record observed physiological data (such as heart-rate, galvanic skin response, body temperature and movement). Finally, applications and future analysis of the results of the study are discussed

O\u27Connor, Noel E.

Irish Universities

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.379.9871

The CDVPlex biometric cinema: sensing physiological responses to emotional stimuli in film

Abstract

Similar works

Full text

Available Versions

Name not available

DCU Online Research Access Service

Irish Universities