International audienceThis paper presents the work done in Technicolor and INRIA regarding the A ffect Task at MediaEval 2011. This task aims at detecting violent shots in movies. We studied a bayesian network framework, and several ways of introducing temporality and multimodality in the framework

Demarty, Claire-Hélène

Gravier, Guillaume

Gros, Patrick

Penet, Cédric

INRIA a CCSD electronic archive server

HAL Id: hal-00643645https://hal.inria.fr/hal-00643645Submitted on 22 Nov 2011HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.Technicolor and INRIA/IRISA at MediaEval 2011:learning temporal modality integration with BayesianNetworksCédric Penet, Claire-Hélène Demarty, Guillaume Gravier, Patrick GrosTo cite this version:Cédric Penet, Claire-Hélène Demarty, Guillaume Gravier, Patrick Gros. Technicolor and IN-RIA/IRISA at MediaEval 2011: learning temporal modality integration with Bayesian Networks.MediaEval 2011, Multimedia Benchmark Workshop, Sep 2011, Pisa, Italy. ￿hal-00643645￿Technicolor and INRIA/IRISA at MediaEval 2011: learningtemporal modality integration with Bayesian Networks∗Cédric Penet, Claire-Hélène DemartyTechnicolor/INRIA Rennes & Technicolor1 ave de Belle Fontaine35510 Cesson-Sévigné, Francecedric.penet@technicolor.comclaire-helene.demarty@technicolor.comGuillaume Gravier, Patrick GrosCNRS/IRISA & INRIA RennesCampus de Beaulieu35042 Rennes, Franceguig@irisa.frPatrick.Gros@inria.frABSTRACTThis paper presents the work done in Technicolor and IN-RIA regarding the Affect Task at MediaEval 2011. Thistask aims at detecting violent shots in movies. We studied abayesian network framework, and several ways of introduc-ing temporality and multimodality in the framework.KeywordsViolence detection, Bayesian Networks1. INTRODUCTIONThe MediaEval 2011 Affect Task aims at detecting violencein movies. A complete description of the task and datasetsmay be found in [2].As explained in the paper cited above, violence has not beena widely studied field, despite its importance in parental con-trol. Moreover, most of the work related to violence focusesmainly on either video or audio. However, in movies, ourintuition is that every information channel is important inorder to correctly assess violence, even for a human asses-sor, and that these information streams are complementary.This paper presents our results for both video and audioonly systems and for the fusion of both sources of informa-tion. We also investigate the importance of temporality inthe framework.2. SYSTEM DESCRIPTIONThe system we developed for the task contains four differentparts:2.1 Features extractionMovies have several components that convey different in-formation, namely audio, video, subtitles, . . . We chose toextract features from the audio and video modalities.Audio modality Five audio features were extracted fromthe audio stream from 40 msec frames with 20 msecoverlap. These features were then averaged over videoshots, as the task is to be performed at the shot level.The audio features are the energy, the centroid, theasymmetry, the zero crossing rate (ZCR) and the flat-ness.∗This work was partly achieved as part of the Quaero Pro-gram, funded by OSEO, French State agency for innovation.Copyright is held by the author/owner(s).MediaEval 2010 Workshop, September 1-2, 2011, Pisa, ItalyVideo modality Four video features were extracted fromthe video for each shot: the shot duration, the averagenumber of blood pixels, the average activity and thenumber of flashes.The features histograms were then rank-normalised over 22values for each movie.2.2 ClassificationIn our process, we chose to use Bayesian networks (BN) [3]as a probability distribution modelisation process. BN ex-hibit interesting features: they model statistical distribu-tion using a graph whose structure may be learned. On thedownside, they consider each feature independently.For the MediaEval 2011 Affect Task, several types of struc-tures were tested:Naive Bayesian network (NB) This is the simplest net-work. Each feature is only linked to the classificationnode. It therefore makes the assumption that the fea-tures are all independent with respect to the classifi-cation node.Forest augmented naive Bayesian network (FAN) Thisstructure has been introduced in [4]. The idea here isto relax the assumption of features independence withrespect to the classification node. The algorithm learnsa forest between the features before connecting themto the classification node.K2 [1] This structure learning algorithm is a state-of-the-art score based greedy search algorithm. In order toreduce the number of possible graphs to test, it re-quires a nodes ordering.We used the Bayes Net Toolbox1.2.3 Temporal integrationConsidering the temporal structure of movies we decided totry several types of temporal integrations: we used contex-tual features2 over n ∈ [−5,+5], and two types of temporalfiltering over 5 samples, that are used to smooth decisions:decision maximum vote This intervenes once the deci-sion has been taken and consists in taking the maxi-mum decision over a few samples.1http://code.google.com/p/bnt/2Considering Xt = [x1, · · · , xK ] the feature vector for sam-ple at time t, the contextual features vector becomes Xct =[Xt−n, · · · , Xt, · · · , Xt+n].Description MC FLF C A: Me V: Ma A: N V: K2 0.761 0.397LF C A: Me V: Me A: N V: K2 0.774 0.391V C Ma K2 0.784 0.305A C Me K2 0.805 0.295V C Me K2 0.840 0.297A C Me N 0.843 0.354EF C Ma K2 0.892 0.284A C Ma K2 0.943 0.268V - Ma N 0.950 0.255A - - K2 0.967 0.251EF C - K2 0.998 0.266V - Ma FAN 1.009 0.276Table 1: Results for runs submitted ordered byincreasing value of MediaEval Cost (MC) (F: F-measure, LF: late fusion, EF: early fusion, A: au-dio, V: video, C: contextual, N: naive BN, Ma: maxdecision vote, Me: mean probability).probability averaging This intervenes before taking thedecision, by directly averaging the samples probabili-ties of being violent.2.4 Modalities fusionAs for multimodal fusion, two cases were considered: latefusion and early fusion. For early fusion, we simply fusedthe features from both modalities before learning, while forlate fusion, we fused the probability of both modalities forthe ith shot si using:P sifused(Psiva , Psivv ) = max(Psiva , Psivv ) if both are violentmin(P siva , Psivv ) if both are non violentP siva ∗ Psivv otherwise(1)where P siva (respectively Psivv ) is the probability of being vi-olent for the audio (respectively video) modality for the ithshot.This rule gives high scores when both audio and video finda violent segment, a low score if they both do not and anintermediate score if only one answers yes.3. RUNS SUBMITTED AND RESULTSThis section describes the runs submitted to the MediaEval2011 Affect Task. For the audio and video experiments, wechose to submit the two best runs according to the MediaE-val cost and to the false alarm vs missed curve using crossvalidation on the learning set, while for the multimodal runs(namely, early and late fusion), we chose the best ones ac-cording to both metrics. The selected runs and their resultsare presented in table 1.Most of the obtained scores have values lower than < 1 whichis better than the simple case where each sample is classifiedas violent, i.e. false alarm rate is 100% and missed detectionrate is 0%.The analysis of the produced graphs yields nice and encour-aging observations on the quality of the structure learningalgorithms. Firstly, the links between features may be easilyinterpreted. The ZCR and centroid are linked as they repre-sent the same information, the activity is linked to the shotlength as the shot detector used tends to oversegment whenthe activity is high, and finally blood is not connected toviolence, which seems logical considering that the presenceof blood in the violent scenes highly depends on the movieand the chosen definition for violence. As for EF, while wethought it would improve the results and find correlationsbetween audio and video, it seems that for non contextualdata, only the video features are linked to the violence node,and that for contextual data the links are messy. This andthe better results obtained using LF tend to indicate thatthe features used in both modalities are from different levelsand cannot be compared as such. Secondly, it seems that thealgorithms produce a strict temporal structure, i.e. the fea-tures from time t = n are linked together and not to featuresfrom different times unless they are in chains. There are fourchains in the graphs: flatness, energy, activity and blood. Itis easy to see that these features have a temporal structure.On the other hand, the flash feature is connected only tothe violence node and forms no chain, which is again logicalas the flash feature only detects high luminance variations,and has therefore no well definite temporal structure.The use of contextual seems to provide good and promisingresults, which tends to confirm the importance of the tempo-ral structure of movies. The depth used for this evaluationhas been chosen arbitrarily, however it should be interestingto also consider other depths. On the downside, it seemsthat these results depend on the algorithm used for learningthe BN structures: FAN and non contextual data seem towork better, while K2 and contextual data seem to give thebest results.This concludes the preliminary analysis that can be inferredfrom the evaluation.4. CONCLUSIONThis paper presents a simple framework based on temporalintegration, multimodality and Bayesian network. First, it isexperimentally shown that the structure learning algorithmoutput logical graph with respect to the provided data: theyare able to capture the links between features and providea coherent temporal structure. It is also shown that earlyfusion with features that have different nature yields to poorresults, while late fusion seems to be more promising. Sec-ond, the use of contextual data seems to improve the result.This work provides a promising baseline for future work onthe subject. We have several improvement ideas. We wantto add features from the text modality, as we think it alsocontains important information on the violent nature of thevideo shots. We also want to investigate more the contextualdata and test other structure learning algorithms.5. REFERENCES[1] G. F. Cooper and E. Herskovits. A Bayesian methodfor the induction of probabilistic networks from data.Machine Learning, 9:309–347, 1992.[2] C.-H. Demarty, C. Penet, G. Gravier, andM. Soleymani. The MediaEval 2011 Affect Task:Violent Scenes Detection in Hollywood Movies. InMediaEval 2011 Workshop, Pisa, Italy, September 1-22011.[3] D. Heckerman. A Tutorial on Learning with BayesianNetworks. Technical report, Microsoft Research, 1995.[4] P. Lucas. Restricted Bayesian Network StructureLearning. In Advances in Bayesian Networks, Studiesin Fuzziness and Soft Computing, pages 217–232, 2002.

Technicolor and INRIA/IRISA at MediaEval 2011: learning temporal modality integration with Bayesian Networks

https://hal.inria.fr/hal-00643645/document

Technicolor and INRIA/IRISA at MediaEval 2011: learning temporal modality integration with Bayesian Networks

Abstract

Similar works

Full text

Available Versions

INRIA a CCSD electronic archive server