Volume: 1739 Host publication title: MediaEval 2016 Multimedia Benchmark Workshop Host publication sub-title: Working Notes Proceedings of the MediaEval 2016 WorkshopNon peer reviewe

Baveye, Yoann

Chamaret, Christel

Chen, Liming

Dellandrea, Emmanuel

Sjöberg, Mats Viktor

Helsingin yliopiston digitaalinen arkisto

The MediaEval 2016 Emotional Impact of Movies TaskEmmanuel Dellandréa1, Liming Chen1, Yoann Baveye2, Mats Sjöberg3 and Christel Chamaret41Ecole Centrale de Lyon, France, {emmanuel.dellandrea, liming.chen}@ec-lyon.fr2Université de Nantes, France, yoann.baveye@univ-nantes.fr3HIIT, University of Helsinki, Finland, mats.sjoberg@helsinki.fi4Technicolor, France, christel.chamaret@technicolor.comABSTRACTThis paper provides a description of the MediaEval 2016”Emotional Impact of Movies” task. It continues builds onprevious years’ editions of the Affect in Multimedia Task:Violent Scenes Detection. However, in this year’s task, par-ticipants are expected to create systems that automaticallypredict the emotional impact that video content will have onviewers, in terms of valence and arousal scores. Here we pro-vide insights on the use case, task challenges, dataset andground truth, task run requirements and evaluation met-rics.1. INTRODUCTIONAffective video content analysis aims at the automaticrecognition of emotions elicited by videos. It has a largenumber of applications, including mood based personalizedcontent recommendation [5] or video indexing [12], and ef-ficient movie visualization and browsing [13]. Beyond theanalysis of existing video material, affective computing tech-niques can also be used to generate new content, e.g., moviesummarization [7], or personalized soundtrack recommen-dation to make user-generated videos more attractive [9].Affective techniques can also be used to enhance the userengagement with advertising content by optimizing the wayads are inserted inside videos [11].While major progress has been achieved in computer vi-sion for visual object detection, scene understanding andhigh-level concept recognition, a natural further step is themodeling and recognition of affective concepts. This has re-cently received increasing interest from research communi-ties, e.g., computer vision, machine learning, with an overallgoal of endowing computers with human-like perception ca-pabilities. Thus, this task is proposed to offer researchers aplace to compare their approaches for the prediction of theemotional impact of movies. It continues builds on previousyears’ editions of the Affect in Multimedia Task: ViolentScenes Detection [10].2. TASK DESCRIPTIONThe task requires participants to deploy multimedia fea-tures to automatically predict the emotional impact of movies.We are focusing on felt emotion, i.e., the actual emotion ofthe viewer when watching the video, rather than for ex-Copyright is held by the author/owner(s).MediaEval 2016 Workshop, Oct. 20-21, 2016, Hilversum, Nether-lands.ample what the viewer believes that he or she is expectedto feel. The emotion is considered in terms of valence andarousal [8]. Valence is defined as a continuous scale frommost negative to most positive emotions, while arousal isdefined continuously from calmest to most active emotions.Two subtasks are considered:1. Global emotion prediction: given a short video clip(around 10 seconds), participants’ systems are expectedto predict a score of induced valence (negative-positive)and induced arousal (calm-excited) for the whole clip;2. Continuous emotion prediction: as an emotion felt dur-ing a scene may be influenced by the emotions felt dur-ing the previous ones, the purpose here is to considerlonger videos, and to predict the valence and arousalcontinuously along the video. Thus, a score of inducedvalence and arousal should be provided for each 1s-segment of the video.3. DATA DESCRIPTIONThe development dataset used in this task is the LIRIS-ACCEDE dataset (liris-accede.ec-lyon.fr) [3]. It is composedof two subsets. The first one, used for the first subtask(global emotion prediction), contains 9,800 video clips ex-tracted from 160 professionally made and amateur movies,with different genres, and shared under Creative Commonslicenses that allows to freely use and distribute videos with-out copyright issues as long as the original creator is cred-ited. The segmented video clips last between 8 and 12 sec-onds and are representative enough to conduct experiments.Indeed, the length of extracted segments is large enough toget consistent excerpts allowing the viewer to feel emotionsand is also small enough to make the viewer feel only oneemotion per excerpt. A robust shot and fade in/out detec-tion has been implemented using to make sure that eachextracted video clip start and end with a shot or a fade.Several movie genres are represented in this collection ofmovies such as horror, comedy, drama, action and so on.Languages are mainly English with a small set of Italian,Spanish, French and others subtitled in English.The second part of LIRIS-ACCEDE dataset is used for thesecond subtask (continuous emotion prediction). It consistsin a selection of movies among the 160 ones used to extractthe 9,800 video clips mentioned previously. The total lengthof the selected movies was the only constraint. It had to besmaller than eight hours to create an experiment of accept-able duration. The selection process ended with the choiceof 30 movies so that their genre, content, language and du-ration are diverse enough to be representative of the originalLIRIS-ACCEDE dataset. The selected videos are between117 and 4,566 seconds long (mean = 884.2sec ± 766.7secSD). The total length of the 30 selected movies is 7 hours,22 minutes and 5 seconds.In addition to the development set, a test set is also pro-vided to assess participants’ methods performance. 49 newmovies under Creative Commons licenses have been consid-ered. With the same protocol as the one used for the de-velopment set, 1,200 additional short video clips have beenextracted for the first subtask (between 8 and 12 seconds),and 10 long movies (from 25 minutes to 1 hour and 35 min-utes) have been selected for the second subtask (for a totalduration of 11.48 hours).In solving the task, participants are expected to exploitthe provided resources. Use of external resources (e.g., In-ternet data) will be however allowed as specific runs.Along with the video material and the annotations, fea-tures extracted from each video clip are also provided bythe organizers for the first subtask. They correspond to theaudiovisual features described in [3].4. GROUND TRUTH4.1 Ground Truth for the first subtaskThe 9,800 video clips included in the first part of theLIRIS-ACCEDE dataset are ranked along the felt valenceand arousal axes by using a crowdsourcing protocol [3]. Tomake reliable annotations as simple as possible, pairwisecomparisons were generated using the quicksort algorithmand presented to crowdworkers who had to select the videoinducing the calmest emotion or the most positive emotion.To cross-validate the annotations gathered from variousuncontrolled environments using crowdsourcing, another ex-periment has been created to collect ratings for a subset ofthe database in a controlled environment. In this controlledexperiment, 28 volunteers were asked to rate a subset of thedatabase carefully selected using the 5-point discrete Self-Assessment-Manikin scales for valence and arousal [4]. 20excerpts per axis that are regularly distributed have beenselected in order to get enough excerpts to represent thewhole database while being relatively few to create an ex-periment of acceptable duration.From the original ranks and these ratings, absolute affec-tive scores for valence and arousal have been estimated foreach of the 9,800 video clips using Gaussian process regres-sion models as described in [1].To obtain ground truth for the test subset, each of the1,200 additional video clips has first been ranked accordingto the 9,800 video clips from the original dataset. Then,its valence and arousal ranks have been converted into avalence and arousal score using the regression models men-tioned previously.4.2 Ground Truth for the second subtaskIn order to collect continuous valence and arousal anno-tations, 16 French participants had to continuously indicatetheir level of arousal while watching the movies using a mod-ified version of the GTrace annotation tool [6] and a joystick(10 participants for the development set and 6 for the testset). Movies have been divided into two subsets. Each anno-tator continuously annotated one subset along the inducedvalence and the other into the induced arousal. Thus, eachmovie has been continuously annotated by five annotatorsfor the development set, and three for the test set.Then, the continuous valence and arousal annotations fromthe participants have been down-sampled by averaging theannotations over windows of 10 seconds with 1 second over-lap (i.e., 1 value per second) in order to remove the noisedue to unintended moves of the joystick. Finally, these post-processed continuous annotations have been averaged in or-der to create a continuous mean signal of the valence andarousal self-assessments. The details of this processing aregiven in [2].5. RUN DESCRIPTIONParticipants can submit up to 5 runs for the first subtask(global emotion prediction). For the second subtask (con-tinuous emotion prediction), there can be 2 types of runsubmissions: full runs that concerns the whole test set (the10 movies, total duration: 11.48 hours) and light runs thatconcern a subset of the test set (5 movies, total duration:4.82 hours). In each case (light and full), up to 5 runs canbe submitted. Moreover, each subtask has a required runwhich uses no externale training data, only the provided de-velopment data is allowed. Also any features that can beautomatically extracted from the video are allowed. Bothtasks also have the possibility for optional runs in whichany external data can be used, such as Internet sources, aslong as they are marked as ”external data” runs.6. EVALUATION CRITERIAStandard evaluation metrics (Mean Square Error and Pear-son’s Correlation Coefficient) are used to assess systems per-formance. Indeed, the common measure generally used toevaluate regression models is the Mean Square Error (MSE).However, this measure is not always sufficient to analyzemodels efficiency and the correlation may be required to ob-tain a deeper performance analysis. As an example, if alarge portion of the data is neutral (i.e., its valence score isclose to 0.5) or is distributed around the neutral score, a uni-form model that always outputs 0.5 will result in good MSEperformance (low MSE). In this case, the lack of accuracyof the model will be brought to the fore by the correlationbetween the predicted values and the ground truth that willbe also very low.7. CONCLUSIONSThe Emotional Impact of Movies Task provides partici-pants with a comparative and collaborative evaluation frame-work for emotional detection in movies, in terms of valenceand arousal scores. The LIRIS-ACCEDE dataset 1 has beenused as development set, and additional movies under Cre-ative Commons licenses and ground truth annotations havebeen provided as test set. Details on the methods and re-sults of each individual team can be found in the papersof the participating teams in the MediaEval 2016 workshopproceedings.8. ACKNOWLEDGMENTSThis task is supported by the CHIST-ERA Visen projectANR-12-CHRI-0002-04.1http://liris-accede.ec-lyon.fr9. REFERENCES[1] Y. Baveye, E. Dellandre´a, C. Chamaret, and L. Chen.From crowdsourced rankings to affective ratings. InIEEE International Conference on Multimedia andExpo Workshops (ICMEW), 2014.[2] Y. Baveye, E. Dellandre´a, C. Chamaret, and L. Chen.Deep learning vs. kernel methods: Performance foremotion prediction in videos. In Humaine AssociationConference on Affective Computing and IntelligentInteraction (ACII), 2015.[3] Y. Baveye, E. Dellandre´a, C. Chamaret, and L. Chen.Liris-accede: A video database for affective contentanalysis. IEEE Transactions on Affective Computing,2015.[4] M. M. Bradley and P. J. Lang. Measuring emotion:the self-assessment manikin and the semanticdifferential. Journal of behavior therapy andexperimental psychiatry, 1994.[5] L. Canini, S. Benini, and R. Leonardi. Affectiverecommendation of movies based on selectedconnotative features. IEEE Transactions on Circuitsand Systems for Video Technology, 2013.[6] R. Cowie, M. Sawey, C. Doherty, J. Jaimovich,C. Fyans, and P. Stapleton. Gtrace: General traceprogram compatible with emotionml. In HumaineAssociation Conference on Affective Computing andIntelligent Interaction (ACII), 2013.[7] H. Katti, K. Yadati, M. Kankanhalli, and C. TatSeng.Affective video summarization and story boardgeneration using pupillary dilation and eye gaze. InIEEE International Symposium on Multimedia (ISM),2011.[8] J. A. Russell. Core affect and the psychologicalconstruction of emotion. Psychological Review, 2003.[9] R. R. Shah, Y. Yu, and R. Zimmermann. Advisor:Personalized video soundtrack recommendation bylate fusion with heuristic rankings. In ACMInternational Conference on Multimedia, 2014.[10] M. Sjo¨berg, Y. Baveye, H. Wang, V. Quang,B. Ionescu, E. Dellandre´a, M. Schedl, C.-H. Demarty,and L. Chen. The mediaeval 2015 affective impact ofmovies task. In MediaEval 2015 Workshop, 2015.[11] K. Yadati, H. Katti, and M. Kankanhalli. Cavva:Computational affective video-in-video advertising.IEEE Transactions on Multimedia, 2014.[12] S. Zhang, Q. Huang, S. Jiang, W. Gao, and Q. Tian.Affective visualization and retrieval for music video.IEEE Transactions on Multimedia, 2010.[13] S. Zhao, H. Yao, X. Sun, X. Jiang, and P. Xu. Flexiblepresentation of videos based on affective contentanalysis. Advances in Multimedia Modeling, 2013.

The MediaEval 2016 Emotional Impact of Movies Task

Abstract

Similar works

Full text

Available Versions

Helsingin yliopiston digitaalinen arkisto