Volume: 1739 Host publication title: MediaEval 2016 Multimedia Benchmark Workshop Host publication sub-title: Working Notes Proceedings of the MediaEval 2016 WorkshopNon peer reviewe

Demarty, Claire-Hélène

Do, Thanh-Toan

Duong, Ngoc Q. K.

Ionescu, Bogdan

Lefèbvre, Frédéric

Sjöberg, Mats Viktor

Wang, Hanli

Helsingin yliopiston digitaalinen arkisto

MediaEval 2016 Predicting Media Interestingness TaskClaire-Hélène Demarty1, Mats Sjöberg2, Bogdan Ionescu3, Thanh-Toan Do4,Hanli Wang5, Ngoc Q. K. Duong1, Frédéric Lefebvre11Technicolor, Rennes, France2HIIT, University of Helsinki, Finland3LAPI, University Politehnica of Bucharest, Romania4Singapore University of Technology and Design, Singapore & University of Science, Vietnam5Tongji University, China{claire-helene.demarty, quang-khanh-ngoc.duong, frederic.lefebvre}@technicolor.commats.sjoberg@helsinki.fi, bionescu@imag.pub.ro, thanhtoan do@sutd.edu.sg, hanliwang@tongji.edu.cnABSTRACTThis paper provides an overview of the Predicting MediaInterestingness task that is organized as part of the Media-Eval 2016 Benchmarking Initiative for Multimedia Evalua-tion. The task, which is running for the first year, expectsparticipants to create systems that automatically select im-ages and video segments that are considered to be the mostinteresting for a common viewer. In this paper, we presentthe task use case and challenges, the proposed data set andground truth, the required participant runs and the evalua-tion metrics.1. INTRODUCTIONThe ability of multimedia data to attract and keep peo-ple’s interest for long periods of time is gaining more andmore importance in the field of multimedia, where conceptssuch as memorability [4], aesthetics [9], interestingness [13,11], attractiveness [14], affective value [25], are intenselystudied, especially in the context of the ever growing mar-ket value of social media and advertising. In particular,although interestingness has been studied for a long time inthe psychology community [21, 2, 22] and more recently butactively in the image processing community [23, 5, 1, 6, 10,18], no common definition exists in the literature. Moreoverdatasets publicly available are only a few and no benchmarkexists for the evaluation of what makes a media interesting.In this paper we introduce the 2016 MediaEval1 Predict-ing Media Interestingness Task, which is a pioneer bench-marking initiative for automatic prediction of image andvideo interestingness. The task which is in its first yearderives from a practical use case at Technicolor2. It in-volves helping professionals to illustrate a Video on Demand(VOD) web site by selecting some interesting frames and/orvideo excerpts for the posted movies. The frames and ex-cerpts should be suitable in terms of helping a user to makehis/her decision about whether he/she is interested in watch-ing the underlying movie. The data in this task is thereforeadapted to this particular context which provides a morefocused definition for interestingness.1http://multimediaeval.org/2http://www.technicolor.com/Copyright is held by the author/owner(s).MediaEval 2016 Workshop, Oct. 20-21, 2016, Hilversum, Nether-lands.2. TASK DESCRIPTIONThe task requires participants to deploy algorithms thatautomatically select images and video segments of Hollywood-like movies which are considered to be the most interestingfor a common viewer. Interestingness of the media is judgedbased on the visual appearance, audio information and textaccompanying the data. Therefore, the multimodal facet ofinterestingness prediction can be investigated.Two different subtasks are provided, which correspond tothe two types of available media content, namely:• predicting image interestingness — given a set of key-frames extracted from a movie, the task requires toautomatically identify those images for the given moviethat viewers report to be the most interesting. To solvethe task, participants can make use of visual contentas well as external metadata, e.g., Internet data aboutthe movie, social media information, etc;• predicting video interestingness — given the video shotsof a movie, the task requires to automatically identifythose shots that viewers report to be the most interest-ing in the given movie. To solve the task, participantscan make use of visual and audio data as well as ex-ternal data, e.g., subtitles, Internet data, etc.In both cases, the task is a binary classification task, thusparticipants are expected to label the provided data as beinginteresting or not (Note that prediction will be carried out ona per movie basis). However, a confidence value is requiredfor the provided prediction.3. DATA DESCRIPTIONThe 2016 data is extracted from Creative Commons li-censed trailers of Hollywood-like movies. It consists of adevelopment data intended for designing and training themethods (information is extracted from 52 trailers) and atesting data which is used for the final benchmarking (withinformation from 26 trailers). The choice of using trailersinstead of full movies is driven by the need to find data,both freely distributable and still representative in contentand quality of Hollywood movies. Trailers are the results ofsome manual filtering of movies to keep interesting scenes,but also less attractive shots to balance their content. Wetherefore believe they are still representative for the task.For the predicting video interestingness subtask, the dataconsists of the video shots obtained after the manual seg-mentation of the videos (video shots are the continuousframe sequences recorded between a camera turn on andoff), i.e., 5,054 shots for the development data, and 2,342shots for the test data.For the predicting image interestingness subtask, the dataconsists of collections of key-frames extracted from the videoshots used for the video subtask. One single key-frame isextracted per shot, therefore leading to 5,054 key-frames forthe development set and 2,342 for the test set. This singlekey-frame is chosen as the middle frame, as it is highly likelyto capture the most important information of the shot.To facilitate participation from various communities, wealso provide some pre-computed content descriptors, namely:low level features — dense SIFT (Scale Invariant FeatureTransform) which are computed following the original workin [17], except that the local frame patches are densely sam-pled instead of using interest point detectors. A codebookof 300 codewords is used in the quantization process with aspatial pyramid of three layers [15]; HoG descriptors (His-tograms of Oriented Gradients) [7] are computed over denselysampled patches. Following [24], HoG descriptors in a 2 ×2 neighborhood are concatenated to form a descriptor ofhigher dimension; LBP (Local Binary Patterns) [19]; GISTare computed based on the output energy of several Gabor-like filters (8 orientations and 4 scales) over a dense framegrid like in [20]; color histogram computed in the HSV space(Hue-Saturation-Value); MFCC (Mel-Frequency Cepstral Co-efficients) computed over 32ms time-windows with 50% over-lap. The cepstral vectors are concatenated with their firstand second derivatives; fc7 layer (4,096 dimensions) andprob layer (1,000 dimensions) of AlexNet [12]; mid level facedetection and tracking related features3 — obtained by facetracking-by-detection in each video shot with a HoG detec-tor [7] and the correlation tracker proposed in [8].4. GROUND TRUTHAll data was manually annotated in terms of interesting-ness by human assessors. A dedicated web-based tool wasdeveloped to assist the annotation process. Overall, morethan 312 annotators participated to the annotation for thevideo data and 100 for the images. The cultural distributionis over 29 different countries in the world.We use a pair-wise comparison protocol [3] where anno-tators are provided with a pair of images/shots at a timeand asked to tag which of the content is more interestingfor them. The process is repeated by scanning the wholedataset. As an exhaustive comparison of all possible pairs isbasically impossible due to the required human resources, aboosting selection was used instead, i.e., a modified versionof the adaptive square design method [16], for which severalannotators participate to each iteration.To achieve the final ground truth, pair-based annotationsare aggregated with the Bradley-Terry-Luce (BTL) modelcomputation [3] resulting in an interestingness degree foreach image/shot. The final binary decisions are obtainedafter the following processing steps: (i) the interestingnessvalues are ranked in increasing order and normalized be-tween 0 and 1; (ii) the resulting curve is smoothed with ashort averaging window, and the second derivative is com-puted; (iii) for both shots and images, and for all videos,a threshold empirically set to 0.01 is applied on the secondderivative to find the first point whose value is above the3http://multimediaeval.org/mediaeval2016/persondiscovery/threshold. This position corresponds to the limit betweennon interesting and interesting shots/images. The underly-ing motivation for this empirical rule is the following: thenon interesting population has rather similar interestingnessvalues which increase slowly, while a gap happens when oneswitches from this non interesting population to the popu-lation of more interesting samples. The second derivativewas chosen preferably to the first derivative, as it allowed toselect those gaps more precisely.Ground truth is provided in binary format, i.e., 1 for in-teresting and 0 for non interesting, for each image and videoin the two subtasks.5. RUN DESCRIPTIONEach participating team is expected to submit up to 5runs for both subtasks altogether. Among these 5 runs, tworuns are required, one per subtask: for the predicting imageinterestingness subtask, the required run is built on visualinformation only and no external data is allowed; for the pre-dicting video interestingness subtask only audio and visualinformation is allowed (no external data) for the requiredrun.Note that in this context, external data can be understoodas: (i) additional datasets and annotations dedicated to in-terestingness classification; (ii) pre-trained models, features,detectors obtained from such dedicated additional datasets;and (iii) additional metadata that could be found on theInternet on the provided content (e.g., from IMDB4).On the contrary, CNN features trained on generic datasetssuch as ImageNet (typically the provided CNN features) areallowed for use in the required runs. By generic datasets, wemean datasets that were not designed to support research inthe task area, i.e., for the classification/study of image andvideo interestingness.6. EVALUATIONThe official evaluation metric is the mean average preci-sion (MAP) over the interesting class, i.e., the mean overthe average precision scores computed for each trailer. Thismetric, adapted to retrieval tasks, fits perfectly the chosenuse case in which we want to help a user choose between dif-ferent samples by providing him a list of suggestions, rankedaccording to interestingness. For assessing the performance,we use the trec_eval tool provided by NIST5. In additionto MAP, other commonly used metrics such as precision andrecall will be provided to participants.7. CONCLUSIONSThe 2016 Predicting Media Interestingness task providesparticipants with a comparative and collaborative evalua-tion framework for predicting content interestingness withexplicit focus on multimedia approaches. Details on themethods and results of each individual participant team canbe found in the working note papers of the MediaEval 2016workshop proceedings.8. ACKNOWLEDGMENTSWe would like to thank Yu-Gang Jiang and Baohan Xufrom the Fudan University, China, and Herve´ Bredin, fromLIMSI, France for providing the features that accompany thereleased data, and Alexey Ozerov and Vincent Demoulin fortheir valuable inputs to the task definition.4http://www.imdb.com/5http://trec.nist.gov/trec\_eval/9. REFERENCES[1] X. Amengual, A. Bosch, and J. L. de la Rosa. Reviewof Methods to Predict Social Image Interestingness andMemorability, pages 64–76. Springer, 2015.[2] D. E. Berlyne. Conflict, arousal and curiosity.Mc-Graw-Hill, 1960.[3] R. A. Bradley and M. E. Terry. Rank analysis ofincomplete block designs: the method of pairedcomparisons. Biometrika, (39 (3-4)):324–345, 1952.[4] Z. Bylinskii, P. Isola, C. Bainbridge, A. Torralba, andA. Oliva. Intrinsic and extrinsic effects on imagememorability. Vision research, (116):165–178, 2015.[5] C. Chamaret, C.-H. Demarty, V. Demoulin, andG. Marquant. Experiencing the interestingnessconcept within and between pictures. In Proceeding ofSPIE, Human Vision and Electronic Imaging, 2016.[6] S. L. Chu, E. Fedorovskaya, F. Quek, and J. Snyder.The effect of familiarity on perceived interestingness ofimages. volume 8651, pages 86511C–86511C–12, 2013.[7] N. Dalal and B. Triggs. Histograms of orientedgradients for human detection. In IEEE CVPRConference on Computer Vision and PatternRecognition, 2005.[8] M. Danelljan, G. Hager, F. S. Khan, and M. Felsberg.Accurate scale estimation for robust visual tracking.In British Machine Vision Conference, 2014.[9] S. Dhar, V. Ordonez, and T. L. Berg. High leveldescribable attributes for predicting aesthetics andinterestingness. In IEEE International Conference onComputer Vision and Pattern Recognition, 2011.[10] H. Grabner, F. Nater, M. Druey, and L. Van Gool.Visual interestingness in image sequences. InProceedings of the 21st ACM International Conferenceon Multimedia, pages 1017–1026, New York, NY,USA, 2013.[11] M. Gygli, H. Grabner, H. Riemenschneider, F. Nater,and L. van Gool. The interestingness of images. InICCV International Conference on Computer Vision,2013.[12] Y.-G. Jiang, Q. Dai, T. Mei, Y. Rui, and S.-F. Chang.Super fast event recognition in internet videos. IEEETransactions on Multimedia, 177(8):1–13, 2015.[13] Y.-G. Jiang, Y. Wang, R. Feng, X. Xue, Y. Zheng,and H. Yan. Understanding and predictinginterestingness of videos. In AAAI Conference onArtificial Intelligence, 2013.[14] S. Kalayci, H. K. Ekenel, and H. Gunes. Automaticanalysis of facial attractiveness from video. In IEEEICIP International Conference on Image Processing,pages 4191–4195, 2014.[15] S. Lazebnik, C. Schmid, and J. Ponce. Beyond bags offeatures: Spatial pyramid matching for recognizingnatural scene categories. In IEEE CVPR Conferenceon Computer Vision and Pattern Recognition, pages2169–2178, 2006.[16] J. Li, M. Barkowsky, and P. L. Callet. Boosting pairedcomparison methodology in measuring visualdiscomfort of 3dtv: performances of three differentdesigns. In SPIE Electronic Imaging, StereoscopicDisplays and Applications, volume 8648, 2013.[17] D. Lowe. Distinctive image features fromscale-invariant keypoints. International Journal onComputer Vision, (60):91–110, 2004.[18] J. Machajdik and A. Hanbury. Affective imageclassification using features inspired by psychologyand art theory. In Proceedings of the 18th ACMInternational Conference on Multimedia, pages 83–92,New York, NY, USA, 2010.[19] T. Ojala, M. Pietikainen, and T. Maenpaa.Multiresolution gray-scale and rotation invarianttexture classification with local binary patterns. IEEETransactions on Pattern Analysis and MachineIntelligence, (24(7)):971–987, 2002.[20] A. Oliva and A. Torralba. Modeling the shape of thescene: a holistic representation of the spatial envelope.International Journal of Computer Vision,(42):145–175, 2001.[21] P. J. Silvia. Exploring the psychology of interest.Oxford University Press, 2006.[22] C. Smith and P. Ellsworth. Patterns of cognitiveappraisal in emotion. Journal of Personality andSocial Psychology, 48(4):813–838, 1985.[23] M. Soleymani. The quest for visual interest. InProceedings of the 23rd ACM International Conferenceon Multimedia, pages 919–922, New York, NY, USA,2015.[24] J. Xiao, J. Hays, K. Ehinger, A. Oliva, andA. Torralba. Sun database: Large-scale scenerecognition from abbey to zoo. In IEEE CVPRConference on Computer Vision and PatternRecognition, pages 3485–3492, 2010.[25] A. Yazdani, E. Skodras, N. Fakotakis, andT. Ebrahimi. Multimedia content analysis foremotional characterization of music video clips.EURASIP Journal on Image and Video Processing,(26), 2013.

MediaEval 2016 Predicting Media Interestingness Task

Abstract

Similar works

Full text

Available Versions

Helsingin yliopiston digitaalinen arkisto