This paper describes our diversity featured retrieval system which are built for the task

of ImageCLEFPhoto 2008. Two existing tools are used: Solr and Carrot. We have

experimented with different settings of the system to see how the performance changes.

The results suggest that the system can indeed increase diversity of the retrieved results

and keep the precision about the same

Arni, T.

Clough, P.

Sanderson, M.

Tang, J.

English

White Rose Research Online

promoting access to White Rose research papers    White Rose Research Online   Universities of Leeds, Sheffield and York http://eprints.whiterose.ac.uk/    White Rose Research Online URL for this paper: http://eprints.whiterose.ac.uk/4687/    Workshop paper Tang, J., Arni, T., Sanderson, M. and Clough, P. (2008) Building a diversity featured search system by fusing existing tools. In: Working Notes for the CLEF 2008 Workshop. CLEF 2008 Workshop, 17-19 September 2008, Aarhus, Denmark eprints@whiterose.ac.uk  Building a Diversity Featured Search System byFusing Existing ToolsJiayu Tang, Thomas Arni, Mark Sanderson, Paul CloughDepartment of Information Studies, University of Sheffield, UK{j.tang, t.arni, m.sanderson, p.d.clough}@sheffield.ac.ukAbstractThis paper describes our diversity featured retrieval system which are built for the taskof ImageCLEFPhoto 2008. Two existing tools are used: Solr and Carrot2. We haveexperimented with different settings of the system to see how the performance changes.The results suggest that the system can indeed increase diversity of the retrieved resultsand keep the precision about the same.Categories and Subject DescriptorsH.3 [Information Storage and Retrieval]: H.3.1 Content Analysis and Indexing; H.3.3 Infor-mation Search and Retrieval; H.3.4 Systems and Software; H.3.7 Digital LibrariesGeneral TermsMeasurement, Performance, ExperimentationKeywordsSolr, Carrot2, Image Retrieval, Indexing Expansion, Query Expansion, Diversity, Clustering1 IntroductionIn this paper, we describe how to quickly set up a diversity featured search system by combiningexisting tools that are available to the public, namely Solr [2] and Carrot2 [3]. We further describehow to tune the system for better performance in the context of ImageCLEFPhoto 2008.2 ToolsWe use Solr for text search and Carrot2 for increasing diversity of results.Solr is a text search server based on the popular search library Lucene [1]. Solr provides somevery useful and convenient features. The web-services like API allows users to index documentsvia XML over HTTP, query it via HTTP GET and receive results in XML format. The fields andfield types of documents can be easily defined in the schema.xml file. In this file, users can alsospecify the Solr “out of the box” tokenizers and token filters to use for indexing and query. Inaddition, the HTML administration interface gives users comprehensive insight into the system.Carrot2 is a open source search results clustering engine. It provides five different algorithmsfor automatically organising search results into thematic categories. Carrot2 works as a pipelineof three kinds of components: input components, filter components and visualisation components.Input components obtain search results from a source of choice (e.g. YahooAPI, GoogleAPI,Lucene, Solr, etc.), then filter components apply clustering algorithms to the search results, andfinally visualisation components render the clustered results to the user.3 System SetupIn ImageCLEFPhoto 2008, each image comes with 9 fields of data: DOCNO, TITLE, DESCRIP-TION, NOTES, LOCATION, DATE, IMAGE, THUMBNAIL and TOPIC. We decided that TI-TLE, DESCRIPTION, NOTES, LOCATION are the fields that would provide useful informationfor text based image retrieval. Therefore, we constructed a new field named TEXT by combiningthe text from the four fields, and specified it as the default search field in Solr. Solr’s default con-figuration of tokenizers and token filters is used in our experiments, namely WhitespaceTokenizer,SynonymFilter, StopFilter, WordDelimiterFilter, LowerCaseFilter, EnglishPorterFilter and Re-moveDuplicatesTokenFilter. More details on each tokenizer and token filter can be found on [2].After feeding the Solr server with all the 20,000 documents in XML format, we have a runningsearch engine that is able to return a ranked list of documents based on the query submitted bythe user. Note that all the fields are indexed by Solr.In Carrot2, we construct an input component for acquiring results from the Solr server. Then,a filter component is used for clustering the results, and a visualisation component is used fordisplaying the clusters and their members. Since ImageCLEFPhoto 2008 assesses the S-recall [5]of the top 20 results of the list to be submitted, we use the following procedure to find the best20 results from the input/ranked list generated by Solr:1: repeat2: for each doc in the input list do3: if membership of the current doc not exist in the list of cluster ID then4: add the current doc to output list5: remove the current doc from input list6: add its membership to the list of cluster ID7: if length of output list equals 20 then8: break9: end if10: if length of cluster ID list equals number of clusters found by Carrot2 then11: clear the list of cluster ID12: break13: end if14: end if15: end for16: until length of output list equals 20Basically, the above pseudo code chooses documents based on two criteria: 1. appear as earlyas possible in the input/ranked list, 2. cover as many different clusters as possible. The 20documents chosen by the above procedure form the list to be submitted for assessment.4 ExperimentsWe have submitted 32 runs, all of which are EN-EN-AUTO-TXT, meaning that all the submissionsare English-English monolingual runs using fully automated text clustering methods. Amongthem, there are 8 groups, each of which includes 4 runs with the same system configuration exceptthat different number of documents are used for clustering. In other words, for each group, wevary the number of documents (40, 60, 80 and 100) that are used from the top of the input listfor clustering by Carrot2, in order to get different output lists. For example, a run with top 40documents applies clustering on the 40 documents and chooses 20 for submission based on theprocedure in Section 3.We increasingly change another 3 kinds of settings of the system to examine how the per-formances would change. Firstly, for topics whose CLUSTER field is Country, City, State orLocation, we specify Carrot2 to cluster the documents by the LOCATION filed, otherwise by theTEXT field. Secondly, we change the parameters of the clustering algorithm used in Carrot2.Finally, we apply expansion to the indexing and query stage. In the following, we describe eachgroup of runs.4.1 BaselineThis group uses the default settings of Solr and Carrot2. In terms of text retrieval, expansionis not applied during indexing or query. Clustering is applied to the TEXT field for all topics.Carrot2’s default clustering algorithm (Lingo [4]) and default parameters (0.150, 0.775) are used.This group is used as a comparison baseline for other groups.4.2 Clustering by LocationIt seems that for topics that have been specified to be clustered by Country, City, State andLocation, the LOCATION filed in each document contains the essential information for clustering.Therefore, such topics are clustered based on the LOCATION field. This group is different fromthe Baseline group in that clustering on some topics are based on the LOCATION field, whileothers based on the TEXT field.4.3 Parameters of Lingo set to (0.05, 0.95)Lingo [4] is a singular value decomposition based clustering algorithm that has been implementedin Carrot2. The first parameter 0.05 is the Cluster Assignment Threshold, determining how precisethe assignment of documents to clusters should be. Lower threshold assign more documents toclusters and less to “Other Topics”, which contains unclassified documents. With a low threshold,more irrelevant documents are also assigned to the clusters. The second parameter 0.95 is theCandidate Cluster Threshold, determining how many clusters Lingo will try to create. Highervalues give more clusters. This group is based on group 4.2, but uses (0.05, 0.95) as the parameters.4.4 Parameters of Lingo set to (0.10, 0.90)This group is the same as group 4.3 except that it uses (0.10, 0.90).4.5 Parameters of Lingo set to (0.15, 0.85)This group is the same as group 4.3 except that it uses (0.15, 0.85).4.6 Indexing and Query Expansion with (0.05, 0.95)This group is based on group 4.3, but applies indexing and query expansion in Solr. Specific domainontologies have been a popular choice for expansion. Cyclopedia websites such as Wikipedia havealso been adopted for expansion. Due to limited time, we used neither of the approaches. Instead,we examined the topics and data-set, and then manually built an indexing expansion list andquery expansion list, as shown in Appendix. These lists are used as the synonym lists in Solr forindexing and query. In the expansion lists, for lines containing “>=”, any of the words before“>=” are replaced by words after “>=” during expansion. For example, “ship, ships => ship,vehicle” will replace any “ship” or “ships” by “ship, vehicle”. For lines without “>=”, any wordfrom the line will be replaced by the whole set of words from the same line. For example, “USA”or “United States of America” or “US” will be replaced by “USA, United States of America, US”.4.7 Indexing and Query Expansion with (0.10, 0.90)This group is the same as group 4.6 except that it uses (0.10, 0.90) in Lingo.4.8 Indexing and Query Expansion with (0.15, 0.85)This group is the same as group 4.6 except that it uses (0.15, 0.85) in Lingo.5 Results and DiscussionsFigure 1 and 2 depict the precision and cluster recall at 20 of all submitted runs. For example,in Figure 2, group 1 corresponds to the results of cluster recall generated by the group of runsdescribed in Section 4.1.00.050.10.150.20.250.30.350.41 2 3 4 5 6 7 8406080100Precision@20GroupFigure 1: Precision at 20 for all groups of runs.As shown in Figure 1, group 6, 7 and 8 have clearly higher precisions than the other groups.This is due to indexing and query expansion using the expansion lists in Appendix. As have beenmentioned, the expansion lists are built based on an examination of the data-set. For example,in Topic 48 “vehicle in South Korea”, “South Korea” normally means the country, so there isnot much ambiguity. However, “vehicle” can mean many things, e.g. car, bus, boat. Intuitively,expanding names of different types of vehicle with the word “vehicle” during indexing will boostthe precision, because many images are only annotated with specific vehicle names rather thanthe word “vehicle”. Therefore, after indexing expansion, “car” becomes “car vehicle”. Similarly,we expanded specific animal names with the word “animal”, so “fish” becomes “fish animal”.In terms of cluster recall, we can see in Figure 2 that different parameters of the clusteringalgorithm (Lingo) have led to different performances. It is a little surprising that group 2 (Section4.2) performed worse than group 1 (Section 4.1), the reason of which needs to be examined. Inaddition, it seems that low Cluster Assignment Threshold (i.e. more documents are clustered)and high Candidate Cluster Threshold (i.e. more clusters are created) give better cluster recall.In our experiments, (0.05, 0.95) gives the best results: group 3 is better than group 4 and 5; group6 is better than group 7 and 8.00.050.10.150.20.250.30.350.40.451 2 3 4 5 6 7 8406080100GroupCluster Recall@20Figure 2: Cluster recall at 20 for all groups of runs.By comparing the two figures, it can also be noticed that groups with the same settings of Solrhave very similar precisions, no matter what settings of Carrot2 were used. For example, withdifferent parameters of Lingo, the precisions of group 6, 7 and 8 are relative stable, but the valuesof cluster recall vary. This can be seen as an evidence that the diversity featured retrieval systemcan make the results more diverse while maintaining the precision.6 Conclusions and Future WorkThis paper has described our submissions to ImageCLEFPhoto 2008. We have changed 4 kindsof settings: the field used for clustering, the number of images used for clustering, indexing andquery expansion, and parameters of the clustering algorithm. The results suggest that indexingand query expansion can fairly improve precision. Appropriately chosen clustering method canincrease diversity of the results while keeping precision almost the same.As we have mentioned, the performance of group 2 is a little against our initial anticipation. Itwould be interesting to find out why. On the other hand, we plan to build an automatic expansionapproach using resources such as ontologies, rather than using the manually built expansion lists.7 AcknowledgementsWork undertaken in this paper is supported by the EU-funded TrebleCLEF project (Grant agree-ment: 215231) and by Tripod (Contract No. 045335).References[1] Apache lucene project. http://lucene.apache.org. [Visited 23/07/08].[2] Apache solr project. http://lucene.apache.org/solr/index.html. [Visited 23/07/08].[3] Carrot2 project. http://project.carrot2.org/. [Visited 23/07/08].[4] S. Osinski, J. Stefanowski, and D. Weiss. Lingo: Search results clustering algorithm based onsingular value decomposition. In Proceedings of the International Conference on IntelligentInformation Systems, pages 359–368, Zakopane, Poland, 2004.[5] Cheng Xiang Zhai, William W. Cohen, and John Lafferty. Beyond independent relevance:methods and evaluation metrics for subtopic retrieval. In Proceedings of the 26th annualinternational ACM SIGIR conference on Research and development in informaion retrieval,pages 10 – 17, Toronto, Canada, 2003.A AppendixA.1 Indexing Expansion List# vehiclesship, ships => ship, vehiclecutter, cutters => cutter, vehicletrain, trains => train, vehiclerail, rails => rail, vehiclelocomotives, locomotive => locomotive, vehiclewagon, wagons => wagon, vehicletractor, tractors => tractor, vehiclebus, buses => bus, vehiclecar, cars => car, vehicleforklift, forklifts => forklift, vehicleboat, boats => boat, vehicle# animalsAlligator, Alligators => Alligator, animalTurtle, Turtles => Turtles, animalDucks, Duck => Duck, animalDolphins, Dolphin => Dolphins, animalFish => Fish, animalWhale, Whales => Whale, animalPelicans, Pelican => Pelicans, animalblowfish => blowfish, animalshoal => shoal, animalShark, Sharks => Shark, animalCrocodile => Crocodile, animalorcas => orcas, animalAngelfish => Angelfish, animalkangaroo, kangaroos => kangaroo, animalwallaby, wallabies => wallaby, animalkoalas, koala => koala, animalwombats, wombat => wombat, animalquokkas, quokka => quokka, animalplatypuses, platypus => platypus, animalpossums, possum => possum, animalTasmanian devils, Tasmanian devil => Tasmanian devil, animalgull => gull, animalflamingo, flamingos => flamingo, animalanacondas, anaconda => anacondas, animalocelot, ocelots => ocelot, animalPenguin, Penguins => penguin, animalcondors, condor => condor, animalmonkey, monkeys => monkey, animalbird, birds => bird, animaliguana, iguanas => iguana, animalsnail, snails => snail, animaltoucan, toucans => toucan, animallion, lions => lion, animalllama, llamas => llama, animalsnake, snakes => snakes, animalTortoise, Tortoises => Tortoise, animalparrot, parrots => parrot, animalBooby, Boobies => Booby, animalhorse, horses => horse, animalsea lion, sea lions => seal, animal# sportsfootball => football, sportsurf => surf, sportmotorcycle => motorcycle, sportrace => race, sport# peoplefans, fan => fan, people# waterriver => river, waterlake => lake, water# stone, rockrock => rock, stonebrick, bricks => brick, stoneA.2 Query Expansion List# countriesUSA, United States of America, US# synonymchurch, churches, cathedral, cathedralsoxidised, rustyobserving, watch, seefootball, soccermatch, gameaccommodation, room# common senseswimming => swimming in the waterdrawings, drawing => drawing, line, petroglyphprize, prizes => prize, medalwater => water, sea

Building a diversity featured search system by

fusing existing tools

This paper describes our diversity featured retrieval system which are built for the task of ImageCLEFPhoto 2008. Two existing tools are used: Solr and Carrot2. We have experimented with different settings of the system to see how the performance changes. The results suggest that the system can indeed increase diversity of the retrieved results and keep the precision about the same. Categories and Subject Descriptor

Building a diversity featured search system by fusing existing tools

Abstract

Similar works

Full text

Available Versions

White Rose Research Online

CiteSeerX