NewsREEL Multimedia premiers 2018 as part of the MediaEval

Benchmarking Initiative. The NewsREEL task combines recommen-

dation algorithms with image and text analysis. Participants must

predict the popularity of news items based on text snippets and

annotated images. Several major German news portals have sup-

plied data. The algorithms are evaluated in terms of

Precision

and

Average Precision

on unknown data. This paper describes the task

and the provided data in detail and explains the applied evaluation

approach

Hopfgartner, F.

Kille, B.

Lommatzsch, A.

Ramming, L.

White Rose Research Online

This is a repository copy of NewsREEL Multimedia at MediaEval 2018: News Recommendation with Image and Text Content.White Rose Research Online URL for this paper:http://eprints.whiterose.ac.uk/138015/Version: Published VersionProceedings Paper:Lommatzsch, A., Kille, B., Hopfgartner, F. orcid.org/0000-0003-0380-6088 et al. (1 more author) (2018) NewsREEL Multimedia at MediaEval 2018: News Recommendation with Image and Text Content. In: Working Notes Proceedings of the MediaEval 2018 Workshop. MediaEval 2018, 29-31 Oct 2018, Cagnes-sur-Mer, France. CEUR-WS , Cagnes-sur-Mer, France . Lommatzsch, A. et al : NewsREEL Multimedia at MediaEval 2018: News Recommendationwith Image and Text Content. Working Notes Proceedings of the MediaEval 2018 Workshop, Cagnes-sur-Mer, France, October 29-31, 2018, CEUR-WS.org, online http://ceur-ws.org/Vol-2283/MediaEval_18_paper_5.pdf.eprints@whiterose.ac.ukhttps://eprints.whiterose.ac.uk/Reuse Items deposited in White Rose Research Online are protected by copyright, with all rights reserved unless indicated otherwise. They may be downloaded and/or printed for private study, or other acts as permitted by national copyright laws. The publisher or other rights holders may allow further reproduction and re-use of the full text version. This is indicated by the licence information on the White Rose Research Online record for the item. Takedown If you consider content in White Rose Research Online to be in breach of UK law, please notify us by emailing eprints@whiterose.ac.uk including the URL of the record and the reason for the withdrawal request. NewsREEL Multimedia at MediaEval 2018:News Recommendation with Image and Text ContentAndreas Lommatzsch,Benjamin KilleTU Berlin, Berlin, Germany{irstname.lastname}@dai-labor.deFrank HopfgartnerUniversity of SheieldSheield, UKf.hopfgartner@sheield.ac.ukLeif Rammingplista GmbHBerlin, Germanyleif.ramming@plista.comABSTRACTNewsREEL Multimedia premiers 2018 as part of the MediaEvalBenchmarking Initiative. The NewsREEL task combines recom-mendation algorithms with image and text analysis. Participantsmust predict engagement with news items based on text snippetsand annotated images. Several major German news portals havesupplied data. The algorithms are evaluated in terms of precisionon unknown data. This paper describes the task and the provideddata in detail and explains the applied evaluation approach. Thealgorithms are evaluated based on Precision and Average-Precisionfor the top news items.KEYWORDSMultimedia, News, Recommender Systems1 INTRODUCTIONRecommender systems help users to ind the most interesting itemsin huge sets of available items [7, 12]. Traditionally, recommendersystems focus on Collaborative Filtering (CF), which makes useof users sharing similar tastes [9]. CF-based approaches rely onusers being trackable with a user ID and on the possibility to collectenough user feedback or interaction data. If the majority of usersare anonymous and if items have short lifecycles and receive fewinteractions, the resulting łcold startž issue impedes CollaborativeFiltering. A majority of anonymous users and items with shortlifecycles induce a permanent cold-start setting [5]. As a result,publishers struggle to apply collaborative iltering in their newsrecommender systems. Content-based recommendation approachesofer an alternative way to address the problem [11]. Usually, newsarticles come in the form of text accompanied by an image. Bothafect readers’ perception. NewsREEL Multimedia tasks partici-pants to predict items’ popularity based on text snippets and imagefeatures.The remaining paper is structured as follows: Section 2 describesthe NewsREEL task in detail. Section 3 introduces the provideddataset. The evaluation procedure is discussed in Section 4. Sec-tion 5 describes the experiment. Section 6 discusses the baselineevaluation results. Finally, Section 7 concludes the paper.2 TASK DESCRIPTIONNewsREEL Multimedia tasks participants to predict news items’popularity from texts and images. The task dataset comprises thenews items collected by several German publishers over the courseMediaEval ’18, 29-31 October 2018, Sophia Antipolis, France© 2018 Copyright held by the owner⁄author(s).of thirteen weeks. The task focuses on non-personalized recom-mendation. It measures popularity by counting how many readersengage with items. In other words, participants must computewhich articles receive the most impressions. The data include tex-tual features, such as headline and text snippet, visual featuresextracted from images, and interaction features, derived from webserver logs, such as the number of impressions. Participants receiveall item-related data for the entire thirteen weeks. The training setcovers item access data of the weeks 0±2 and 6±8. Participants mustpredict items’ popularity for weeks 4, 10, 11, and 12 (evaluation set).The popularity data for the weeks 3, 5, and 9 have been excludedto prevent extrapolation of time series. Instead, participants shouldfocus on image and text content. Item IDs and features are availablefor all weeks. Participants must predict the most popular items forthe evaluation weeks as well as the number of impressions for themost popular news items.3 DATA DESCRIPTIONThe dataset covers thirteen weeks of four selected publishers, whopublish predominantly German articles. Ωe encounter 51 289 im-ages displayed alongside articles during this period. The images dis-tribute unequally with one publisher accounting for 42 003 images.In addition, we provide a total of 1 691 unique labels automaticallyassigned to images by seven annotators trained on ImageNet [4].The dataset amounts to approximately 8.6 GB in size. Ωe observea total of about 153million impressions, 397million recommenda-tions, and 1.1million clicks.The dataset includes the following features for each item:1) item data (ID, item URL, item image URL, timestamp)2) textual features (headline, text snippet; in German)3) image features (up to ten labels per image and a weighting, ac-tivation weights of a standard deep learning network encodingthe image). The images have been annotated by means of diferentframeworks (Keras [2], TensorFlow [1] and existing, pre-trainedmodels (VGG16, VGG19 [13]).4) Items’ popularity data (number of views, number of clicks, num-ber of times recommended). The image popularity data are providedonly for the training weeks.In addition to the provided features, participants may computeadditional features or integrate data from external sources. Corsiniand Larson [3] discuss how to apply image feature extraction fornews recommendation. Kille et al. [8] and Gulla et al. [6] describeadditional datasets for news recommendation.The entire dataset has been collected by plista GmbH. Access tothe data is subject to a usage agreement with their providers.MediaEval ’18, 29-31 October 2018, Sophia Antipolis, France4 EVALUATION AND GROUND TRUTHNews recommender systems determine the most relevant articles.For the NewsREEL challenge, we have computed the number ofimpressions for the items published on selected news portals. Ωesplit the data into a training and a test set. For the test set we haveremoved the number of impressions. Task participants must predictthe unknown number of impressions for items in selected weeks.Ωe consider the number of impressions as a proxy for relevance. Thequality of the predictions is computed by comparing the predictionswith the true number of impressions (observed in the test weeks).In the evaluation we consider diferent metrics.Precision measures how precisely participants identify rele-vant items. Ωe consider two cut-of points. First, we compute thePrecision@n to check whether participants manage to identifythe most popular items. In the challenge, we analyze n = 10 andn = 0.1∗number of items in the bin. Second, we compute theAver-age Precision@n (AP). Ωe deine the AP as the mean of the top nprecision scores: AP =∑Mn=11M∗ Precision@n, whereM describesthe number of elements in the test set. For computing Precision@nwe assume the top n items to be the target. In other words, task par-ticipants succeed if they manage to identify the most relevant items.Ωe compute the precision metrics for each publisher separately.Baseline strategies and the observed evaluation results are dis-cussed in the subsequent section. Baseline strategies and their eval-uation results are discussed in [10].5 RUN DESCRIPTIONThe data cover thirteen weeks indexed from 0 to 12. Participantsreceive the content-related features to all items read in this period.The interaction-related features, such as impressions and clicks,remained unavailable for the weeks 3 to 5 as well as 9 to 12. Par-ticipants must create a predictor using the data from weeks 0 to2 and estimate the number of impressions for items in week 4.Likewise, they ought to use the data from weeks 6 to 8 to predictimpressions in weeks 10 to 12. Ωe obtain a prediction for each com-bination of item and week in the speciied periods. For each of theweeks we compute 4 metrics: Precision@10, Precision@Top10%,and AP@Top10%. Ωe average those measurements over the weeksto determine the overall score of the submission.6 EVALUATIONΩe have implemented three baseline algorithms:(1) The random recommender shules the itemIDs randomlyand assigns each item the average number of impressions for anitem at that rank as the prediction.(2) The text similarity-based recommender computes the sim-ilarity of each item in the test set with all items in the training set.The similarity computation is done using the cosine similarity ofthe term vectors provided for the news items. Subsequently, thesimilarity-weighted average of the number of impressions for allitems (similar to the current news item) is computed and used asprediction.(3) The image label-based recommender computes similarityof news items based on the overlap of images labels (having a prob-ability bigger than 0.3). Given an item of the test set, the algorithmestimates the number of impressions by the a similarity-weightedTable 1: Baseline Evaluation Results (portal 13554).Random Text-based Image-basedP@ P@ AP@ P@ P@ AP@ P@ P@ AP@Ωeek 10 10% 10% 10 10% 10% 10 10% 10%04 0.00 0.11 0.07 0.40 0.19 0.23 0.40 0.21 0.1810 0.00 0.10 0.06 0.40 0.19 0.22 0.30 0.24 0.1811 0.00 0.09 0.04 0.40 0.18 0.21 0.30 0.22 0.1612 0.00 0.10 0.04 0.40 0.19 0.21 0.30 0.22 0.17avg. 0.00 0.10 0.05 0.40 0.18 0.21 0.33 0.22 0.17Table 2: Baseline Evaluation Results (portal 39234).Random Text-based Image-basedP@ P@ AP@ P@ P@ AP@ P@ P@ AP@Ωeek 10 10% 10% 10 10% 10% 10 10% 10%04 0.00 0.08 0.04 0.30 0.13 0.09 0.10 0.13 0.0910 0.00 0.12 0.07 0.30 0.14 0.09 0.10 0.08 0.0511 0.00 0.10 0.04 0.30 0.12 0.05 0.00 0.10 0.0612 0.00 0.11 0.06 0.40 0.11 0.06 0.00 0.10 0.04avg. 0.00 0.10 0.05 0.33 0.13 0.07 0.05 0.10 0.06average of the number of impression (similar to the current item inthe training set).The evaluation results for two publishers are listed in Table 1and 2. The random recommender reaches a very low precision. Theimage label-based recommender shows a slightly better precision.The text-based recommender outperforms the image label-basedrecommender. This indicates that text provides more informationthat image label when computing the popularity in news items.Moreover, we observe big diferences between the portals. Thisindicates that the importance of images depends on the speciicnews portal. In addition, the variance in the weeks indicates thatthe dataset is sparse and diferences in the user behavior in theconsidered weeks exist.7 CONCLUSIONNewsREEL Multimedia is a challenging task combining recom-mendation with text and image analysis. The challenge provides areal-world data set collected by several major German news portals.The evaluation centers on anticipating the most popular articlesby their contents. Details on the developed methods and the ob-tained results are reported in the workshop working notes of theMediaEval workshop.REFERENCES[1] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat,G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray,B. Steiner, P. Tucker, V. Vasudevan, P. Ωarden, M. Ωicke, Y. Yu, and X. Zheng.Tensorlow: A system for large-scale machine learning. In Proceedings of the 12thUSENIX Conference on Operating Systems Design and Implementation, OSDI’16,pages 265±283, Berkeley, CA, USA, 2016. USENIX Association.[2] F. Chollet et al. Keras. https:⁄⁄keras.io, 2015.[3] F. Corsini and M. Larson. CLEF NewsREEL 2016: Image-based Recommendation.CLEF 2016: Ωorking Notes of CLEF 2016 - Conference and Labs of the EvaluationForum, 2016.[4] J. Deng, Ω. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-ScaleHierarchical Image Database. In IEEE Conference on Computer Vision and PatternRecognition, 2009.NewsREEL Multimedia at MediaEval 2018 MediaEval ’18, 29-31 October 2018, Sophia Antipolis, France[5] F. Garcin, B. Faltings, O. Donatsch, A. Alazzawi, C. Bruttin, and A. Huber. Of-line and online evaluation of news recommender systems at swissinfo.ch. InProceedings of the 8th ACM Conference on Recommender systems, pages 169±176.ACM, 2014.[6] J. A. Gulla, L. Zhang, P. Liu, Ö. Özgöbek, and X. Su. The adressa dataset fornews recommendation. In Proceedings of the International Conference on WebIntelligence, pages 1042±1048. ACM, 2017.[7] D. Jannach, M. Zanker, A. Felfernig, and G. Friedrich. Recommender systems: anintroduction. Cambridge University Press, 2010.[8] B. Kille, F. Hopfgartner, T. Brodt, and T. Heintz. The plista dataset. In Proceedingsof the 2013 International News Recommender Systems Workshop and Challenge,pages 16±23. ACM, 2013.[9] Y. Koren and R. Bell. Advances in collaborative iltering. In Recommender systemshandbook, pages 77±118. Springer, 2015.[10] Lommatzsch, Andreas and Kille, Benjamin. Baseline Algorithms for Predictingthe Interest in News based on Multimedia-Data. In Proceedings of the MediaEvalWorkshop, 2018.[11] P. Lops, M. De Gemmis, and G. Semeraro. Content-based recommender systems:State of the art and trends. In Recommender systems handbook, pages 73±105.Springer, 2011.[12] F. Ricci, L. Rokach, and B. Shapira. Recommender systems: introduction andchallenges. In Recommender systems handbook, pages 1±34. Springer, 2015.[13] K. Simonyan and A. Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. International Conference on Learning Representations,2015.

NewsREEL Multimedia at MediaEval 2018: News Recommendation with Image and Text Content

https://eprints.whiterose.ac.uk/138015/1/NewsREELMultimedia.pdf

NewsREEL Multimedia at MediaEval 2018: News Recommendation with Image and Text Content

Abstract

Similar works

Full text

Available Versions

White Rose Research Online