We present the results of the EUMSSI team’s participation in the Multimodal Person Discovery task. The goal is to identify all people who simultaneously appear and speak in a video corpus. In the proposed system, besides improving each modality, we emphasize on the ranking of multiple results from both audio stream and visual stream

Le, Nam

Meignier, Sylvain

Odobez, Jean-Marc

Infoscience - École polytechnique fédérale de Lausanne

EUMSSI team at the MediaEval Person DiscoveryChallenge 2016Nam Le1,2, Sylvain Meignier3, Jean-Marc Odobez1,21 Idiap Research Institute, Martigny, Switzerland2 École Polytechnique Fédéral de Lausanne, Switzerland3 LIUM, University of Maine, Le Mans, France{nle, odobez}@idiap.ch, sylvain.meignier@univ-lemans.frFigure 1: Architecture of our systemABSTRACTWe present the results of the EUMSSI team’s participationin the Multimodal Person Discovery task. The goal is toidentify all people who simultaneously appear and speak ina video corpus. In the proposed system, besides improv-ing each modality, we emphasize on the ranking of multipleresults from both audio stream and visual stream.1. INTRODUCTIONAs the retrieval of information on people in videos is ofhigh interest for users, algorithms indexing identities of peo-ple and retrieving their respective quotations are indispens-able for searching archives. This practical need leads to re-search problems on how to identify people presence in videos.Given the raw TV broadcasts, each shot must be automat-ically tagged with the name(s) of people who can be bothseen as well as heard in the shot along with the confidentscore. The list of people is not known apriori and theirnames must be discovered from video text overlay or speechtranscripts [6]. To this end, a video must be segmented inan unsupervised way into homogeneous segments accordingto person identity, like speaker diarization and face diariza-tion, to be combined with the extracted names. Our goalis to benchmark our recent improvements in all componentsand address the fusion of multimodal results.2. PROPOSED SYSTEMThe system we proposed is illustrated in Fig. 1. It con-sists of 4 main parts: video optical character recognition(OCR) and named entity recognition (NER), face diaria-tion, speaker diarization, and fusion naming.2.1 Video OCR and NERTo detect OCR segments in videos and exploit them forretrieval, we first relied on the approaches described in [2, 1]Copyright is held by the author/owner(s).MediaEval 2016 Workshop, Oct. 20-21, 2016, Hilversum, Nether-landsfor text recognition in videos, and on [3, 15] for text recog-nition and indexing. In brief, given an input video, twomain steps are applied: first the video is preprocessed witha motion filtering to reduce noise, and individual frames areprocessed to localize and binarize the text regions for textrecognition. As compared to printed documents, OCR inTV news videos encounters several challenges: low resolu-tion of text regions, sequence of different texts continuouslydisplayed, or small amount of text to be recognized etc. Totackle these, multiple image segmentations of the same textregion are decoded, and then all results are compared andaggregated over time to produce several hypotheses. Thebest hypothesis is used to extract people names for identi-fication. To recognize names from texts, we use the MITIEopen library 1, which provides state-of-the-art NER tool. Toimprove the raw MITIE results, a heuristics preprocessingstep identifies names of editorial staff based on their roles(cameraman, editor, or writer) because they do not appearwithin the video, thus are not useful for identification.2.2 Face diarizationGiven the video shots, face diarization process consists of(i) face detection, (ii) face tracking, and (iii) face clustering.Detection & tracking. Detecting and associating facescan be challenging due to the wide range of media content,where faces can appear with varied illumination and noise.To overcome these challenges, we use a fast version of de-formable part-based model (DPM) [5, 11, 4] to detect facesat multiple poses and variation. Tracking is performed usingthe CRF-based multi-target tracking framework [7], whichrelies on the unsupervised learning of time sensitive associ-ation costs for different features. Because the bottle-neck ofthe system is detection, the detector is only applied 4 timesper second. We also trained an explicit false alarm classifierat the track level to efficiently filter out false tracks. Furtherdetails can be found in [9].Face clustering. We hierarchically merge face tracks acrossall shots using matching and biometric similarity measuressimilarly to [8] with two improvements: shot-constrainedface clustering (SCFC) and the use of total variability mod-eling (TVM). SCFC is a divide-and-conquer strategy. Faceclustering is first applied limiting within each group of sim-ilar shots. Then all resulting face clusters, which are nowmuch fewer in quantity, are hierarchically merged. TVM is astate-of-the-art biometrics method that can represent faceswhich can appear in widely different contexts and sessions1https://github.com/mit-nlp/MITIE[17, 16]. To compute similarity between face clusters, wesimply use the average distance between all pairs of facesusing the cosine distance between i-vectors.2.3 Speaker diarizationThe speaker diarization system is based on the LIUMSpeaker Diarization system[14], which is publicly distributed2.It is provided to all participants as the baseline method.2.4 Identification and result rankingAfter obtaining homogeneous clusters during which dis-tinct identities speak or appear, one needs to assign eachname output from NER module to the correct clusters. How-ever, associating auditory voices with visual person clustersor names has two major difficulties. The visible person maynot be the current speaker and the speaking person can bedubbed by a narrator in a different language. Although wehave introduced a temporal learning method to solve thedubbing problem [10], incorporating it into an AV diariza-tion system is still an open question. Because of these prob-lems of AV association, we use a direct naming method [13]which finds the mapping between clusters and names tomaximize the co-occurrences between them.Identification. Names are propagated based on the out-puts of face diarization and speaker diarization indepen-dently. The direct naming method is applied to speakerclusters to produce a mapping between names and clusters.All shots which overlap with the clusters are tagged withthe corresponding names with equal confident scores. Thesame direct method is applied to face clusters to producea set of named clusters. Unlike speaker naming, for oneshot, a name coming from face naming is ranked based onthe talking score of the cluster’s segment within that shot.The talking score is predicted using lip motion and tempo-ral modeling with LSTM [10]. Based on the two results, wepropose a strategy to appropriately combine them.Ranking. Let S = {sk} be the list of testing shots. Withineach shot, {NFi , t(NFi )} is the set of names returned by facenaming and the corresponding talking scores and {NAi , 1.0}is the set of names returned by speaker naming, each isranked equally with score 1.0. The names which the twomethods agree on are ranked highest. Then, names fromface naming are ranked higher than speaker naming becausewe found that face naming is more reliable in empirical ex-periments. Alternative strategies that rank speaker namingequal or higher than face naming gave inferior results. Ourranking strategy is described in Algo. 1.Further fusion. Finally, replacing individual component inour system with baseline NER [12] and face diarization 3 canproduce complementary results. Therefore, these results areadded to our final submission with lower confident scores.3. EVALUATIONParticipants are scored based on a set of queries. Eachquery is a person name in the corpus, each participant hasto return all shots when that person appears and talks. Themetric is Mean Average Precision (MAP) over all queries. InTab. 1, we report our result on the test set as of 24/09/2016 4.2www-lium.univ-lemans.fr/en/content/liumspkdiarization3http://pyannote.github.io/4The groundtruth is still updated by a collaborative anno-tation process.Algorithm 1 Ranking names within shots1: for sk ∈ S do2: Qsk = ∅3: Face naming(sk)⇒ (NFi , t(NFi ))4: Speaker naming(sk)⇒ (NAj , 1.0)5: for each NFi do6: if ∃NAj /NAj = NFi then7: Qsk = Qsk ∪ {(NFi , t(NFi ) + 2.0)}8: else9: Qsk = Qsk ∪ {(NFi , t(NFi ) + 1.0)}10: for each NAj do11: if not ∃NFi /NFi = NAj then12: Qsk = Qsk ∪ {(NAj , 1.0)}MAP@1 MAP@10 MAP@100Sub. (1) 30.3 22.0 21.0Sub. (2) 58.6 42.9 42.0Sub. (3) 64.2 53.1 52.1Sub. (4) 68.3 56.2 54.7Sub. (5) 79.2 65.2 63.4Table 1: Benchmarking results of our submissions.Details of each submission in the text.Each of our 5 submissions (Sub.) is as following:• Sub. (1) and Sub. (2) used our face naming withouttalking score with baseline OCR-NER (1) or with ourOCR-NER (2).• Sub. (3) used our face naming with talking score.• Sub. (4) used the combination of talking face namingin sub. (3) with speaker naming.• And sub. (5) used the combination of sub. (4) withother systems using baseline OCR-NER or baselineface diarization. This is also our primary submission.When comparing sub. (1) and sub. (2), one can observethat our OCR-NER outperforms the baseline OCR-NER bya large margin. This may be contributed by the high re-call of our system. Because the metric is averaged overall queries, any missing name can significantly decrease theoverall MAP. On the other hand, false names are less prob-lematic because of two reasons: they may not be associatedwith any clusters and they are not queried at all. In sub.(3), using talking face detection with LSTM, we can furtherimprove by 5.6%. By combining face naming and speakernaming, we manage to increase the precision. This shows thepotential for further research of better audio-visual naming.In our primary submission (5), the result are greatly boostedwhen other methods are added. From this we can note thatthese methods are complementary to each other and how toexploit their advantages is an open question in the future.4. CONCLUSIONWe have presented our system in MediaEval challenge2016. This system consists of our recent advances in videoprocessing and temporal modeling. Although each modal-ity shows positive performance, the current system has nottaken full advantage of both audio and visual streams. There-fore, the testing results serve as the basis for us to workfurther in this direction.Acknowledgement This research was supported by theEuropean Union project EUMSSI (FP7-611057).5. REFERENCES[1] D. Chen and J.-M. Odobez. Video text recognitionusing sequential monte carlo and error voting methods.Pattern Recognition Letters, 26(9):1386–1403, 2005.[2] D. Chen, J.-M. Odobez, and H. Bourlard. Textdetection and recognition in images and video frames.Pattern Recognition, 37(3):595–608, 2004.[3] N. Daddaoua, J.-M. Odobez, and A. Vinciarelli. Ocrbased slide retrieval. In Eighth InternationalConference on Document Analysis and Recognition(ICDAR’05), pages 945–949. IEEE, 2005.[4] C. Dubout and F. Fleuret. Deformable part modelswith individual part scaling. In BMVC, 2013.[5] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, andD. Ramanan. Object detection with discriminativelytrained part-based models. IEEE Transactions onPattern Analysis and Machine Intelligence,32(9):1627–1645, 2010.[6] C. B. H. Bredin, C. Guinaudeau. Multimodal persondiscovery in broadcast tv at mediaeval 2016. In Proc.of the MediaEval 2016 Workshop, Hilversum,Netherlands, Oct. 2016.[7] A. Heili, A. Lopez-Mendez, and J.-M. Odobez.Exploiting long-term connectivity and visual motionin crf-based multi-person tracking. IEEE Transactionson Image Processing, 23(7):3040–3056, 2014.[8] E. Khoury, P. Gay, and J.-M. Odobez. FusingMatching and Biometric Similarity Measures for FaceDiarization in Video. In ACM ICMR, 2013.[9] N. Le, A. Heili, D. Wu, and J.-M. Odobez. Temporallysubsampled detection for accurate and efficient facetracking and diarization. In International Conferenceon Pattern Recognition. IEEE, Dec. 2016.[10] N. Le and J.-M. Odobez. Learning multimodaltemporal representation for dubbing detection inbroadcast media. In ACM Multimedia. ACM, Oct.2016.[11] M. Mathias, R. Benenson, M. Pedersoli, andL. Van Gool. Face detection without bells andwhistles. In ECCV, pages 720–735. Springer, 2014.[12] J. Poignant, L. Besacier, G. Que´not, and F. Thollard.From text detection in videos to person identification.In 2012 IEEE International Conference on Multimediaand Expo (ICME), pages 854–859. IEEE, 2012.[13] J. Poignant, H. Bredin, V.-B. Le, L. Besacier,C. Barras, and G. Que´not. Unsupervised speakeridentification using overlaid texts in tv broadcast. InInterspeech, page 4p, 2012.[14] M. Rouvier, G. Dupuy, P. Gay, E. Khoury, T. Merlin,and S. Meignier. An open-source state-of-the-arttoolbox for broadcast news diarization. In Interspeech,Lyon (France), 25-29 Aug. 2013.[15] A. Vinciarelli and J.-M. Odobez. Application ofinformation retrieval technologies to presentationslides. IEEE Transactions on Multimedia,8(5):981–995, 2006.[16] R. Wallace and M. McLaren. Total variabilitymodelling for face verification. Biometrics, IET,1(4):188–199, 2012.[17] R. Wallace, M. McLaren, C. McCool, and S. Marcel.Inter-session variability modelling and joint factoranalysis for face authentication. In Biometrics (IJCB),2011 International Joint Conference on, pages 1–8.IEEE, 2011.

EUMSSI team at the MediaEval Person Discovery Challenge 2016

http://infoscience.epfl.ch/record/223040/files/Le_MEDIAEVAL_2016.pdf

EUMSSI team at the MediaEval Person Discovery Challenge 2016

Abstract

Similar works

Full text

Available Versions

Infoscience - École polytechnique fédérale de Lausanne