This document presents the contribution of the UPC team to the Social Event Detection (SED) Subtask 1 in MediaEval 2014. This contribution extends the solution tested in the previous year with a better optimization of the parameters that determine the clustering algorithm, and by introducing an additional pass that considers the merges of all pairs of mini-clusters generated during the two first passes. Our proposal also addresses the problem of incomplete metadata by generating additional textual tags based on geolocation and natural language processing techniques.Postprint (published version

Giró Nieto, Xavier

Gris-Sarabia, Irene

Manchon Vizuete, Daniel

English

This document presents the contribution of the UPC team to the Social Event Detection (SED) Subtask 1 in MediaEval 2014. This contribution extends the solution tested in the previous year with a better optimization of the parameters that determine the clustering algorithm, and by introducing an additional pass that considers the merges of all pairs of mini-clusters generated during the two first passes. Our proposal also addresses the problem of incomplete metadata by generating additional textual tags based on geolocation and natural language processing techniques

UPCommons

UPC at MediaEval 2014 social event detection task

UPCommons. Portal del coneixement obert de la UPC

UPC at MediaEval 2014 Social Event Detection TaskDaniel Manchon-Vizuete, Irene Gris-Sarabia and Xavier Giro-i-NietoUniversitat Politecnica de CatalunyaBarcelona, Cataloniaxavier.giro@upc.eduABSTRACTThis working notes paper presents the contribution of theUPC team to the Social Event Detection (SED) Subtask 1in MediaEval 2014. This contribution extends the solutiontested in the previous year with a better optimization ofthe parameters that determine the clustering algorithm, andby introducing an additional pass that considers the mergesof all pairs of mini-clusters generated during the two firstpasses. Our proposal also addresses the problem of incom-plete metadata by generating additional textual tags basedon geolocation and natural language processing techniques.1. MOTIVATIONThis document describes the algorithms tested by theUPC team in the MediaEval 2014 Social Event Detection(SED) Subtask 1, which addressed the problem of full clus-tering of a photo collection. The reader is referred to [6] forthe task description in MediaEval 2014 and further detailsabout the study case, dataset and metrics .The proposed approach extends our submission in the pre-vious MediaEval SED 2013 [5]. That work solved the FullClustering subtask by firstly generating a temporal-basedover-segmentation of the photo collection as proposed byPhotoTOC [7]. In a second pass, the mini-clusters in aclose temporal neighbourhood were clustered based on ge-olocation, textual tags and user ID metadata. That solutionpresented difficulties when addressing the realistic scenariowhere many photos had missing metadata information andnoisy tags.We observed that the images in the provided collection areheterogeneous in terms of the type of metadata they haveassociated with. Some of the photos in the dataset containedgeolocation data, while some others did not. On the otherhand, users also present diverse behaviour regarding phototagging. While some users will provide textual tags to theirpictures, others will just share the photos with no tags atall.This situation generates an unbalanced description of thephotos in the dataset. In our previous approach [5], we ap-plied different similarity functions when assessing the merg-ing of mini-clusters, depending on which metadata was avail-able from the mini-clusters.On the other hand, when tags are present, there is alsoa problem related to natural language processing. TextualCopyright is held by the author/owner(s).MediaEval 2014 Workshop, October 16-17, 2014, Barcelona, Spaintags have a semantic dimension that was not captured in ourprevious approach, where we compared tag similarity solelybased on the string of characters, with no semantic inter-pretation at all. This approach fails into detecting relatedtags and synonyms, as they are considered as completelydifferent.In this MediaEval SED 2014 edition, we addressed theseproblems by assessing the mergings between mini-clustersbased on the textual tags only. This way it was not nec-essary to define and train different merging strategies de-pending on the available metadata. This approach requiredthe introduction of strategies to populate the tags associatedto the photos from the available data. In addition, in thisyear’s submission, we also optimized the merging of mini-clusterings by adopting a three-pass approach, instead of atwo-pass one as in the previous year.This paper is structured as follows. Section 2 presents thenovelties that have been introduced this year compared toour previous submission. The performance of the solutionis assessed in Section 3 with the results obtained on theMediaEval SED 2014 task. Finally, Section 4 provides theinsights learned with this work.2. APPROACHThis section presents the new advances we have intro-duced in our algorithm with respect to the solution submit-ted in MediaEval SED 2013 [5].2.1 Text-based cluster mergingWhile in the previous year mini-clusters were merged witha similarity that combined geolocation, tags and user IDs, in2014 we have used only the textual tags of the mini-clustersto assess their merges. In addition, instead of using theJaccard Index on the tags to assess the similarity betweenmini-clusters, this year we have switched to TF-IDF descrip-tors, compared with the cosine distance. These changes haveallowed a better estimation of the relevance of the differentterms with respect to others.2.2 Reverse geocodingA first solution to increase the amount of textual tags wasusing the geolocation coordinates by obtaining the namesof the locations where the photos were taken. This strat-egy is known as Reverse geocoding, and it allows transform-ing numerical coordinates for geolocation into a readabletext description. Our system has used the MapQuest OpenGeocoding API Web Service, which provides an easy accessto data stored under the OpenStreeMap (OSM) project [3].2.3 Tag expansion with hypernymsOne of the main challenges when dealing with textual tagsis their semantic interpretation. The case of synonyms can-not be captured with a character-based descriptor. It re-quires the introduction of some semantic information, typi-cally coded in an ontology.Another novelty in this year has been the enrichment ofthe tag set with the hypernyms, that is, those terms thatprovide a generalization of each original term. This waywe have tried to capture better the synonym terms. TheNatural Language Processing Toolkit (NLTK) [4] has beenused for this task, which exploits the knowledge containedin WordNet [2].2.4 Three clustering passesThe creation of clusters begins with a sequential processin the temporal dimension, as in our previous submissionin 2013, which was inspired by the original PhotoTOC al-gorithm [7]. This approach allows a fast converge of theproblem, as clusters are initially only merged in a local time-based neighbourhood.In particular, our approach defines three passes. In thefirst pass, the photos of each user are treated separately,comparing the time stamps of the photos in a local neigh-bourhood. In a second pass, the mini-clusters generatedfrom the first pass are compared in also a small temporalneighbourhood and a relaxed similarity threshold, assessedwith the TF-IDF descriptors of each mini-cluster. At thisstage, the user ID associated to each photo is also addedas an additional textual tag. Finally, a third pass considersall pairs of existing mini-clusters in a global neighborhood.However, in this case the similarity threshold for merging ismuch more strict.3. EXPERIMENTS AND RESULTSThe UPC participated in Subtask 1 with the results shownin Table 1. The description of the different runs is the fol-lowing:Run 1: No tags were added. The parameters that opti-mized the clustering algorithm were found with hyper-opt Python package [1].Run 2: Configuration from Run 1 was modified by addingtextual tags obtained with reverse geolocation , as de-scribed in Section 2.2.Run 3: Configuration from Run 1 was modified by addingtextual tags obtained with the hypernyms provided bythe NLTK Python package [4], as described in Section2.3.Run 4: Configurations from Run 2 and Run 3 were com-bined.GeoTag Hypernym F1 NMI Div. F1Run 1 0.9240 0.9820 0.9231Run 2 • 0.9165 0.9793 0.9155Run 3 • 0.8141 0.9432 0.8127Run 4 • • 0.8112 0.9393 0.8097Table 1: UPC results in Subtask 1.4. CONCLUSIONSThe UPC contribution to the Social Event Detection taskin MediaEval 2014 has been an extension of the approachpreviously tested in the 2013 edition [5]. The implementedalgorithm provides a fast solution for photo clustering givenits time-based first- and second- passes, which compare pho-tos and mini-clusters in a local neighbourhood. A first im-provement from the previous submission has been the op-timization of the parameters which control the creation ofthese mini-clusters. The main novelty has been introducedin the merging of these mini-clusters which, instead of com-bining different modalities (tags, geolocation and user ID),have been solely based on textual tags. Nevertheless, thesetextual tags have been extended in two directions: by re-verse geocoding and with hypernyms. However, the resultsthat have been obtained in the test dataset do not showany improvement by the introduction of the new tags. Infact, performance decreases, especially when using the hy-pernyms. This behaviour on the test dataset was unex-pected, as during the development of our solution with thetraining dataset, an improvement had been observed.5. ACKNOWLEDGMENTSThis work has been developed in the framework of theproject TEC2013-43935-R, funded by the Spanish Ministeriode Economia y Competitividad and the European RegionalDevelopment Fund (ERDF).6. REFERENCES[1] J. Bergstra, D. Yamins, and D. D. Cox. Making ascience of model search: Hyperparameter optimizationin hundreds of dimensions for vision architectures. InICML (1), volume 28 of JMLR Proceedings, pages115–123. JMLR.org, 2013.[2] C. Fellbaum. WordNet. Wiley Online Library, 1998.[3] M. Haklay and P. Weber. OpenStreetMap:User-generated street maps. Pervasive Computing,IEEE, 7(4):12–18, 2008.[4] E. Loper and S. Bird. NLTK: The natural languagetoolkit. In Proceedings of the ACL-02 Workshop onEffective Tools and Methodologies for Teaching NaturalLanguage Processing and Computational Linguistics -Volume 1, ETMTNLP ’02, pages 63–70, Stroudsburg,PA, USA, 2002. Association for ComputationalLinguistics.[5] D. Manchon Vizuete, I. Gris-Sarabia, and X. Giro-iNieto. Photo clustering of social events by extendingPhotoTOC to a rich context. In Proc. Social Events inWeb Multimedia (SEWN), 2014.[6] G. Petkos, S. Papadopoulos, V. Mezaris, andY. Kompatsiaris. Social Event Detection at MediaEval2014: Challenges, datasets, and evaluation. InMediaEval 2014 Workshop, Barcelona, Spain, October16-17 2014.[7] J. C. Platt, M. Czerwinski, and B. Field. PhotoTOC:automatic clustering for browsing personalphotographs. In Proc. 4th Pacific Rim Conference onMultimedia., volume 1, pages 6–10, 2003.

https://upcommons.upc.edu/bitstream/2117/24444/1/mediaeval2014_submission_58.pdf

UPC at MediaEval 2014 social event detection task

Abstract

Similar works

Full text

Available Versions

UPCommons

UPCommons. Portal del coneixement obert de la UPC