In this paper we describe the Imperial College London, Technische Universitat München and University of Passau (ICL+TUM+PASSAU) team approach to the MediaEval's "Affective Impact of Movies" challenge, which consists in the automatic detection of affective (arousal and valence) and violent content in movie excerpts. In addition to the baseline features, we computed spectral and energy related acoustic features, and the probability of various objects being present in the video. Random Forests, AdaBoost and Support Vector Machines were used as classification methods. Best results show that the dataset is highly challenging for both affect and violence detection tasks, mainly because of issues in inter-rater agreement and data scarcity

Coutinho, E

Marchi, E

Ringeval, F

Schuller, B

Trigeorgis, G

Zafeiriou, S

University of Liverpool Repository

The ICL-TUM-PASSAU Approach for the MediaEval 2015“Affective Impact of Movies” TaskGeorge Trigeorgis1, Eduardo Coutinho1,Fabien Ringeval2,3, Erik Marchi2,Stefanos Zafeiriou1, Björn Schuller1,31Department of Computing, Imperial College London, UK2Machine Intelligence & Signal Processing Group, Technische Universität München, Munich, Germany3Chair of Complex & Intelligent Systems, University of Passau, Germanyg.trigeorgis@imperial.ac.ukABSTRACTIn this paper we describe the Imperial College London,Technische Universita¨t Mu¨nchen and University of Passau(ICL+TUM+PASSAU) team approach to the MediaEval’s“A↵ective Impact of Movies” challenge, which consists inthe automatic detection of a↵ective (arousal and valence)and violent content in movie excerpts. In addition to thebaseline features, we computed spectral and energy relatedacoustic features, and the probability of various objects be-ing present in the video. Random Forests, AdaBoost andSupport Vector Machines were used as classification meth-ods. Best results show that the dataset is highly challengingfor both a↵ect and violence detection tasks, mainly becauseof issues in inter-rater agreement and data scarcity.1. INTRODUCTIONThe MediaEval 2015 Challenge “A↵ective Impact ofMovies” comprises two subtasks using the LIRIS-ACCEDEdatabase [2]. Subtask 1 targets the automatic categorisa-tion of videos in terms of their a↵ective impact. The goalis to identify the arousal (calm-neutral-excited) and valence(negative-neutral-positive) levels of each video. The goalof Subtask 2 is to identify those videos that contain violentscenes. The full description of the tasks can be found in [22].2. METHODOLOGY2.1 Subtask 1: affect classificationFeature sets.In our work we have used both the baseline features pro-vided by the organisers [2], as well as our own sets of audio-visual features as described below.The extended Geneva Minimalistic Acoustic ParameterSet (eGeMAPS) was used to extract acoustic features withthe openSMILE toolkit [6]; this feature set was designed as astandard acoustic parameter set for automatic speech emo-tion recognition [5, 18, 16] and has also been successfullyused for other paralinguistic tasks [17]. The eGeMAPScomprises a total of 18 Low-Level Descriptors (LLDs), in-cluding frequency, energy/amplitude, and spectral relatedfeatures. Various functionals were then applied to the LLDsover the whole instance, giving raise to a total of 88 features.Copyright is held by the author/owner(s).MediaEval 2015 Workshop, Sept. 14-15, 2015, Wurzen, GermanyThe emotional impact of videos can be heavily influencedby the kind of objects present in a given scene [11, 12,15]. We thus computed a probability of 1000 di↵erent ob-jects to be present in a frame using a pretrained 16-layerconvolutional neural network (CNN) on the ILSVRC2013dataset [21, 4]. Let x 2 RN⇥p represents a video of thedatabase with N frames and p pixels per frame, and f(·)the trained convolutional neural net with softmax activa-tion functions in the output layer. The probability Pr(y =c|xi; ✓) for each of the 1000 classes being present inside thei-th frame of a video xi is obtained by forwarding the ppixels value through the network. By averaging the activa-tions over all the N frames of a video sequence we obtainedthe probability distribution of the 1000 ILSVRC2013 classesthat might be present in the video.Classifiers.For modelling the data we concentrated on two out-of-the-box ensemble techniques: Random Forests and AdaBoost.We used these two techniques as they are less susceptibleto the overfitting problem than other learning algorithmsdue to the combination of weak learners, they are trivial tooptimise as they have only one hyper-parameter, and theyusually provide close or on par results with the state-of-the-art for a multitude of tasks [9, 10, 23, 14]. The hyper-parameters for each classifier were determined using a 5-foldcross-validation scheme on the development set. During de-velopment the best performance was achieved with 10 treeswith Random Forests and 20 trees with AdaBoost.Runs.We submitted a total of five runs. Run 1 consisted of pre-dictions using the baseline features and the AdaBoost model.The predictions in runs 2 and 5 were obtained using thebaseline plus our audio-visual feature sets and the RandomForest and AdaBoost classifiers, respectively. By lookingat the distribution of labels in the development set, we ob-served that the most common combinations of labels are: 1)neutral valence (V n) and negative arousal (A ) (24%), and2) positive valence (V +) and negative arousal (A ) (20%).Runs 3 and 4 are thus based on the hypothesis that the labeldistribution of the test set will be similarly unbalanced. Inrun 3 every clip was predicted to be V n, A+ and in Run4 every one was V +, A . These submissions act as a san-ity check of our own models, but also other competitors’submissions for this competition.2.2 Subtask 2: violence detectionFeature sets.According to previous work [7, 13], we only consideredspectral and energy based features as acoustic descriptors.Indeed, violent segments do not necessarily contain speech;voice specific features, such as voice quality and pitch re-lated descriptors, might thus not be a reliable source of in-formation for violence. We extracted 22 acoustic low-leveldescriptors (LLDs): loudness, alpha ratio, Hammarberg’sindex, energy slope and proportion in the bands [0 500]Hzand [500  1500]Hz, and 14 MFCCs, using the openSMILEtoolkit [6]. All LLDs, with the exception of loudness andthe measures of energy proportion, were computed sepa-rately for voiced and unvoiced segments. As the framesof the movie that contain violent scenes are unknown, wecomputed 5 functionals (max, min, range, arithmetic meanand standard-deviation) to summarise the LLDs over themovie excerpt, which provided a total of 300 features. Forthe video modality, we used the same additional features de-fined in Subtask 1. We also used the metadata informationof the video genre as an additional feature, due to depen-dencies between movie genre and violent content.Classifier.Since the dataset is strongly imbalanced – only 272 ex-cerpts out of 6,144 are labelled as violent – we up-sampledthe violent instances to achieve a balanced distribution. Allfeatures were furthermore standardised with a z-score. Asclassifier, we used the libsvm implementation of SupportVector Machines (SVMs) [3] and optimised the complexityparameter, and the   coe cient of the radial basis kernelin a 5-folds cross-validation framework on the developmentset. Because the o cial scoring script requires the com-putation of a posteriori probabilities, which is more timeconsuming than the straightforward classification task, weoptimised the Unweighted Average Recall (UAR) to findthe best hyper-parameters [19, 20], and then re-trained theSVMs with the probability estimates.Runs.We first performed experiments with the full baseline fea-ture set and found that the addition of the movie genreas feature improved the Mean Average Precision (MAP)from 19.5 to 20.3, despite degrading the UAR from 72.3 to72.0. Adding our own audio-visual features provided a jumpin the performance with the MAP reaching 33.6 and UAR77.6. Because some movie excerpts contain partly relevantacoustic information, we empirically defined a threshold onloudness based on the histogram, to exclude frames beforecomputing the functionals. This procedure has improvedthe MAP to 35.9 but downgraded the UAR to 76.9. A finetuning of the complexity parameter and   coe cient yieldedthe best performance in terms of UAR with a value of 78.0,but slightly deteriorated the MAP to 35.7.We submitted a total of five runs. Run 1 – baseline fea-tures; Run 2 – all features mentioned above (except moviegenre) with loudness threshold (0.038); Run 3 – same as Run2 plus the inclusion of movie genre; Run 4 – as Run 3 butwith a fine tuning of the hyper-parameters; Run 5 – similarto Run 3 but with a higher threshold for loudness (0.078).3. RESULTSOur o cial results on the test set for both subtasks areshown in Table 1.Subtask 1. Our results for the a↵ective task indicate thatwe did not do much better than was expected by chance forarousal classification, and did slightly better than chancefor valence in run 5; we thus refrain from further interpreta-tion of results. This can be explained by the low quality ofthe provided annotations for the dataset. The initial anno-tations had a low inter-rater agreement [2], and there weremultiple processing stages afterwards [1, 22] with high levelsof uncertainty and unclear validity.Subtask 2. Results show that there is an important over-fitting in our models as the performance is divided by a fac-tor of 2 between development and test partitions. This is,however, not really surprising since only 272 instances la-belled as violent were available as training data. Moreover,the labelling task being performed not at the frame levelbut rather at the excerpt level does not allow to model pre-cisely the information that is judged as violent, making thetask highly challenging. We can nevertheless observe thatthe proposed audio-visual feature set brings a large improve-ment over the baseline feature set – the MAP is improvedby a factor superior to 2, and that the inclusion of the moviegenre as additional feature also allows a small improvementin the performance.Subtask 1 Subtask 2Run Arousal (AC) Valence (AC) Violence (MAP)1 55.72 39.99 4.92 54.71 41.00 13.33 55.55 37.87 13.54 55.55 29.02 14.95 54.46 41.48 13.9Table 1: The submission results for the arousal, va-lence, and violence classification tasks on the testpartition. AC stands for accuracy and MAP for themean average precision.4. CONCLUSIONSWe have presented our approach to the MediaEval’s “Af-fective Impact of Movies” challenge, which consists in theautomatic detection of a↵ective and violent content in movieexcerpts. Our results for the a↵ective task have shown thatwe did not do much better than a classifier that is basedon chance, although we use features and classifiers that areknown to work well in the literature for arousal and valenceprediction [2, 8]. We consider that this might be owed to apotentially noisiness of the annotations provided. As for theviolence prediction subtask, the results show that we over-fit a lot on the development set, which is not very strikinggiven the small amount of instances of the minority class.The analysis of violent content at the excerpt level is alsohighly challenging, because only few frames might containviolence, and such brief information is almost totally lost inthe computation of functionals at the full excerpt level.5. ACKNOWLEDGEMENTSThe research leading to these results has received fundingfrom the EC’s Seventh Framework Programme through theERC Starting Grant No. 338164 (iHEARu), and the EU’sHorizon 2020 Programme through the Innovative ActionNo. 644632 (MixedEmotions), No. 645094 (SEWA) and theResearch Innovative Action No. 645378 (ARIA-VALUSPA).6. REFERENCES[1] Y. Baveye, E. Dellandrea, C. Chamaret, and L. Chen.From crowdsourced rankings to a↵ective ratings. InProceedings of the IEEE International Conference onMultimedia and Expo Workshops (ICMEW 2014),pages 1–6, 2014.[2] Y. Baveye, E. Dellandrea, C. Chamaret, and L. Chen.LIRIS-ACCEDE: A Video Database for A↵ectiveContent Analysis. IEEE Transactions on A↵ectiveComputing, 6(1):43–55, January-March 2015.[3] C.-C. Chang and C.-J. Lin. LIBSVM: A library forsupport vector machines. ACM Transactions onIntelligent Systems and Technology, 2(3), April 2011.[4] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, andL. Fei-Fei. Imagenet: A large-scale hierarchical imagedatabase. In Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition (CVPR2009), pages 248–255, 2009.[5] F. Eyben, K. Scherer, B. Schuller, J. Sundberg,E. Andre´, C. Busso, L. Devillers, J. Epps, P. Laukka,S. Narayanan, and K. Truong. The Genevaminimalistic acoustic parameter set (GeMAPS) forvoice research and a↵ective computing. IEEETransactions on A↵ective Computing, In press, 2015.[6] F. Eyben, F. Weninger, F. Groß, and B. Schuller.Recent Developments in openSMILE, the MunichOpen-Source Multimedia Feature Extractor. InProceedings of the 21st ACM International Conferenceon Multimedia (MM 2013), pages 835–838, Barcelona,Spain, October 2013.[7] F. Eyben, F. Weninger, N. Lehment, and B. Schuller.A↵ective Video Retrieval: Violence Detection inHollywood Movies by Large-Scale Segmental FeatureExtraction. PLOS one, 8(12):1–12, December 2013.[8] K. Forbes-Riley and D. J. Litman. Predicting emotionin spoken dialogue from multiple knowledge sources.In Proceedings of the Conference of the NorthAmerican Chapter of the Association forComputational Linguistics - Human LanguageTechnologies (HLT-NAACL), pages 201–208, 2004.[9] F. G., J. Gall, and V. G. L. Real time head poseestimation with random regression forests. InProceedings of the IEEE International Conference onComputer Vision and Pattern Recognition (CVPR),pages 617–624, Providence (RI), USA, June 2011.[10] L. Guo, N. Chehata, C. Mallet, and S. Boukir.Relevance of airborne lidar and multispectral imagedata for urban scene classification using RandomForests. ISPRS Journal of Photogrammetry andRemote Sensing, 66(1):56–66, January 2011.[11] A. Hanjalic and L.-Q. Xu. A↵ective video contentrepresentation and modeling. IEEE Transactions onMultimedia, 7(1):143–154, 2005.[12] W. Hu, N. Xie, L. Li, X. Zeng, and M. S. A survey onvisual content-based video indexing and retrieval.IEEE Transactions on Systems, Man, andCybernetics, Part C: Applications and Reviews,41(6):797–819, October 2011.[13] B. Ionescu, J. Schlu¨ter, I. Mironica, and M. Schedl. Anaive mid-level concept-based fusion approach toviolence detection in hollywood movies. In Proceedingsof the 3rd ACM International Conference onMultimedia Retrieval (ICMR 2013), pages 215–222,Dallas (TX), USA, 2013.[14] J.-J. Lee, P.-H. Lee, S.-W. Lee, A. Yuille, andC. Koch. Adaboost for text detection in natural scene.In Proceedings of the IEEE 12th InternationalConference on Document Analysis and Recognition(ICDAR 2013), pages 429–434, Beijing, China, 2013.[15] I. Lopatovskaa and I. Arapakis. Theories, methodsand current research on emotions in library andinformation science, information retrieval andhuman–computer interaction. Information Processing& Management, 47(4):575–592, July 2011.[16] F. Ringeval, S. Amiriparian, F. Eyben, K. Scherer,and B. Schuller. Emotion recognition in the wild:Incorporating voice and lip activity in multimodaldecision-level fusion. In Proceedings of the 2ndEmotion Recognition In The Wild Challenge andWorkshop (EmotiW 2014), pages 473–480, Istanbul,Turkey, September 2014.[17] F. Ringeval, E. Marchi, M. Mehu, K. Scherer, andB. Schuller. Face reading from speech – predictingfacial action units from audio cues. In Proceedings ofINTERSPEECH 2015, 16th Annual Conference of theInternational Speech Communication Association(ISCA), to appear, Dresden, Germany, September2015.[18] F. Ringeval, B. Schuller, M. Valstar, S. Jaiswal,E. Marchi, D. Lalanne, R. Cowie, and M. Pantic.AV+EC 2015 – The First A↵ect RecognitionChallenge Bridging Across Audio, Video, andPhysiological Data. In Proceedings of the 5thInternational Workshop on Audio/Visual EmotionChallenge (AVEC), ACM MM, Brisbane, Australia,October 2015.[19] B. Schuller, S. Steidl, A. Batliner, J. Epps, F. Eyben,F. Ringeval, E. Marchi, and Y. Zhang. TheINTERSPEECH 2014 Computational ParalinguisticsChallenge: Cognitive & Physical Load. In Proceedingsof INTERSPEECH 2014, 15th Annual Conference ofthe International Speech Communication Association(ISCA), pages 427–431, Singapore, Republic ofSingapore, September 2014.[20] B. Schuller, S. Steidl, A. Batliner, S. Hantke,F. Ho¨nig, J. R. Orozco-Arroyave, E. No¨th, Y. Zhang,and F. Weninger. The INTERSPEECH 2015Computational Paralinguistics Challenge: Nativeness,Parkinson’s & Eating Condition. In Proceedings ofINTERSPEECH 2015, 16th Annual Conference of theInternational Speech Communication Association(ISCA), Dresden, Germany, September 2015.[21] K. Simonyan and A. Zisserman. Very deepconvolutional networks for large-scale imagerecognition. arXiv preprint arXiv:1409.1556, 2014.[22] M. Sjo¨berg, Y. Baveye, H. Wang, V. Quand,B. Ionescu, E. Dellandre´a, M. Schedl, C.-H. Demarty,and L. Chen. The MediaEval 2015 A↵ective Impact ofMovies Task. In Proceedings of the MediaEval 2015Workshop, Wurzen, Germany, 2015.[23] A. Stumpf and N. Kerle. Object-oriented mapping oflandslides using random forests. Remote Sensing ofEnvironment, 115(10):2564–2577, October 2011.

The ICL-TUM-PASSAU approach for the MediaEval 2015 "affective impact of movies" task

Abstract

Similar works

Full text

Available Versions

University of Liverpool Repository