This working notes paper describes our first participation in the ImageCLEF2010 Wikipedia Retrieval Task. In this task, we mainly test our Quantum Theory inspired retrieval function on cross media retrieval. Instead of heuristically combining the ranking scores independently from different media types, we develop a tensor product based model to represent textual and visual content features of an image as a non-separable composite system. Such system incorporates the statistical/semantic dependencies between certain features. Then the ranking scores of the images are computed in a way as quantum measurement does. Meanwhile, we also test a new local feature that we have developed for content based image retrieval

Kaliciak, Leszek

Song, Dawei

Wang, Jun

English

Open Research Online (The Open University)

Open Research OnlineThe Open University’s repository of research publicationsand other research outputsRGU at ImageCLEF2010 Wikipedia Retrieval TaskConference or Workshop ItemHow to cite:Wang, Jun; Song, Dawei and Kaliciak, Leszek (2010). RGU at ImageCLEF2010 Wikipedia Retrieval Task. In:CLEF 2010 LABs and Workshops, Notebook Papers, 22-23 Sep 2010, Padua, Italy.For guidance on citations see FAQs.c© 2010 The AuthorsVersion: Version of RecordLink(s) to article on publisher’s website:http://clef2010.org/resources/proceedings/clef2010labs submission 96.pdfCopyright and Moral Rights for the articles on this site are retained by the individual authors and/or other copyrightowners. For more information on Open Research Online’s data policy on reuse of materials please consult the policiespage.oro.open.ac.ukRGU at ImageCLEF2010 Wikipedia Retrieval TaskJun Wang, Dawei Song, and Leszek KaliciakSchool of Computing, The Robert Gordon University, Aberdeen, UK{j.wang3,d.song,l.kaliciak}@rgu.ac.ukAbstract. This working notes paper describes our first participation in the Im-ageCLEF2010 Wikipedia Retrieval Task[1]. In this task, we mainly test our Quan-tum Theory inspired retrieval function on cross media retrieval. Instead of heuris-tically combining the ranking scores independently from different media types,we develop a tensor product based model to represent textual and visual contentfeatures of an image as a non-separable composite system. Such system incor-porates the statistical/semantic dependencies between certain features. Then theranking scores of the images are computed in a way as quantum measurementdoes. Meanwhile, we also test a new local feature that we have developed forcontent based image retrieval.1 IntroductionIn our participation to ImageCLEF we have submitted runs in the Wikipedia RetrievalTask, which are based on our Quantum Theory inspired retrieval model. This modelapplies tensor product to represent the textual and visual features of an image as a n-order tensor in order to capture the non-separability of textual and visual features. Theorder of the tensor depends on the visual features that are going to be incorporated inthe image retrieval.We have tested this retrieval model on the ImageCLEF2007 data collection that has20,000 images in total, and achieved some promising results. Therefore we would liketo test it on a larger scale dataset.2 Tensor Based Retrieval ModelMathematically, the tensor product is used to construct a new vector space or a new ten-sor, where the relationship of the vector spaces can be expressed. In quantum mechan-ics, the tensor product can be used to expand Hilbert spaces or construct a compositesystem with separate systems.Suppose an image represented in a single feature space to be a single system Si, thenthat image can be constructed as a composite system by tensor product the separatedsystems from different feature spaces:S = S1 ⊗ S2 · · · ⊗ Sn (1)Where Si is a system in a single feature space.Next lets look at how we present a single system in the Hilbert space with Diracnotation. Here we will not introduce the detail of Dirac notation, readers who are inter-ested can refer to Van Rijsbergen’s book “The Geometry of Information Retrieval”[2].With quantum mechanics, traditionally vector based representation of documentsare represented as superposed states. The textual feature of an image is:|T 〉 =∑iwti |ti〉 (2)where∑i w2ti = 1, and the amplitude wti is proportional to the probability that thedocument is about the term ti. wti can be set up with any term weighting scheme, andwe adopted TF-IDF in our experiment.Similarly, the histogram of content feature can also be represented as a superposedstate in a content feature space:|F 〉 =∑iwfi |fi〉 (3)where∑i w2fi= 1, fi is the a particular feature bin, and wfi is proportional to thenumber of pixels falling into the corresponding bin of the feature space.When more than one features are used to represent an image, the representation willbe:|D〉 = |T 〉 ⊗ |F1〉 ⊗ |F2〉 · · · ⊗ |Fn〉 (4)In our pilot study, we only combine one content feature with the textual feature.|D〉 = |T 〉 ⊗ |F 〉 (5)=∑ijγij |ti ⊗ fj〉 (6)Here, i and j are the dimensionalities of textual and content features. When textualand content feature are completely independent, then γij = wti · wfj . However, thisdoes not hold generally. Extra operation is necessary to reflect the non-separability ofthe two features. It can be operationalised as the co-occurrence or correlation of thefeatures. The tensor product enables the expansion of feature spaces in a seamless wayand incorporates the correlations between the feature spaces.With superposed representation, the similarity of a document and a query can beviewed as the probability that the document projects onto the sub-space that is expandedby the query features.The probability that a document collapses to a state is:P (ti|d) = |〈ti|T 〉|2 = w2ti (7)It can be described as a projection onto a space spanned by |ti〉:P (ti|d) = 〈ti|ρd|ti〉 = w2ti (8)where ρd = |d〉〈d| is the density matrix of document d, and w2ti is the probability thatthe term ti appears in the document or the probability that the document is about theterm.With the textual and visual composite system, the density matrix of a document is:ρD = |D〉〈D| =∑ijγ2ij |tifj〉〈tifj | (9)Then the similarity between a document and a query is:sim(d, q) = tr(ρdρq) (10)For a visual feature, each of its dimensions can be treated as orthogonal to otherdimensions. Because the blue color pixel can never be counted as a red pixel. Whilefor textual feature, two terms can be semantically related, e.g. cup and mug may referto the same thing in one document. To represent the document with orthogonal textualbasis, we can use a transformation matrix to fulfill the requirement:ρd =∑iw2i |ti〉〈ti| (11)=∑iw2i∑jUij |ej〉∑k〈ek|Uik (12)=∑jkρjk|ej〉〈ek| (13)In the current experiment, we assume that all the terms are orthogonal to simplifythe calculation, then:sim(d, q) = tr(∑id2i |ti〉〈ti|Q) (14)= tr(∑id2i |ti〉〈ti|∑jq2j |tj〉〈tj |) (15)=∑id2i q2i (16)3 Experiment Settings and Results3.1 Text ProcessingWhen we associated texts with images, we not only used annotation documents, butalso used Wikipedia pages, hoping the original full text document can provide moresemantic information.Although in the Wikipedia task, the images may be annotated by more than onelanguage, due to our language expertise we only parse the English language. If theimages are not annotated by English, or do not appeared in English Wikipedia page,then this image will not be indexed. It also means that this image will never be retrievedin the runs using both text and content features as queries.For the annotation files, we parsed all the terms from name, description, commentand caption entries if they contain any term. For the Wikipedia dump files, We parsed allthe terms in the Wikipedia page, including the the link names within the page. Howeverwe do not go further into the linked page. The tags particular to the Wikipedia webpageare removed.Same for the queries, we only use their English titles during the retrieval stage.3.2 Content FeatureApart from the content features provided by the organizer, we also used our local fea-ture, which is based on the “bag of features”/“bag of visual words” approach. Thefeature extraction consists of following stages: image sampling, description of localpatches, feature vector construction, visual dictionary generation and histogram com-putation. The number of sample points is 900 per image and the sampling is purelyrandom. We open a square window - local patch (10 by 10 pixels wide) centred atthe sample point. Each local patch is represented in a form of three colour momentscomputed for individual colour channels in HSV colour space. Thus obtained vectorrepresentation of a local patch has 9 dimensions. We apply the K-means clusteringto the training set in order to obtain the codebook of visual words. Finally, we createa histogram of visual word counts by calculating manhattan distance between imagepatches and cluster centroids and generate a vector representation for each image fromthe collection. Thus obtained vector representation of an image has 40 dimensions.3.3 Experiment runs and ResultsWith the textual and visual feature available, we submitted the following runs, someof which are based on content feature only and some are content and textual featuremixed.– Text and content mixed retrieval• T+F L : retrieve on annotation first, then re-rank with our local feature• T+F C : retrieve on annotation first, then re-rank with cime• TXF L : quantum-like measurement on tensor product space of annotation vec-tor and our local feature vector• TXF C : quantum-like measurement on tensor product space of annotation vec-tor and feature cime vector• combine: T+F C based retrieval. When the length of result from T+F C is lessthan 1000, then the images from content retrieval are appended into the resultlist.• W+F C : retrieve on Wikipedia file first, then re-rank with cime– Content only retrieval• c leszek: city block distance with our new local feature• c add: city block distance with all content feature provided by ImageCLEForganiserFor the mixed retrieval, we did not run the mixed retrieval process on the whole col-lection due to the huge size. We ran the text retrieval first, then applied mixed retrievalto the re-ranking.From table 1, we can see that the retrieval result based on content feature has ex-tremely low MAP. The retrieval results from text and content mixed retrieval in Image-CLEF2010 is also considerably lower than our results from ImageCLEF2007 whoseMAPs that are around 14%.After further looking into the tensor product based experiments, we did find a bug inthe code. Because a “+” operator is missing, which resulted in that only the last textualand content feature dimension had been used to re-rank the image. This accounts thepoor performance of the submission runs.We have corrected the code and re-run the experiments. The text based result isMAP=0.0939 and P@10=0.3485; The tensor product based result with our local featureis MAP=0.0665 and P@10=0.2000. There is a slightly improvement after removing thebug from the experiment, but it is still a lot of worse than text based retrieval, whichneeds a further investigation.Run Modality field(s) MAP P@10combine Mixed TITLEIMG 0.0617 0.2271T+F L Mixed TITLEIMG 0.0617 0.2257TXF C Mixed TITLEIMG 0.0486 0.1443TXF L Mixed TITLEIMG 0.0484 0.1500T+F C Mixed TITLEIMG 0.0325 0.1143W+F C Mixed TITLEIMG 0.0031 0.0086c leszek Visual IMG 0.0069 0.0614c add Visual IMG 0.0003 0.0100Table 1. Our runs in ImageCLEF20104 Conclusions and Future WorksIn this notes paper, we reported our quantum theory inspired multimedia retrieval frame-work, which provides a formal and flexible way to expand the feature spaces, and seam-lessly integrate different features. The similarity measurement between query and doc-ument follows quantum measurement.We did not include the correlations between dimensions across different featurespaces in this year submission. We would like to investigate such issue in the continuingstudy, and further the study with entanglement concept in quantum mechanics.In the current experiments, we assumed each word is orthogonal, which does nothold and can be relaxed in the future. We can solve this problem with either dimension-ality reduction or utilize the thesaurus to remove the synonyms. This will also facilitatethe ranking computation.AcknowledgementsThis work is funded in part by the UK’s Engineering and Physical Sciences ResearchCouncil (grant number: EP/F014708/2).References1. Adrian Popescu, Theodora Tsikrika, and Jana Kludas. Overview of the wikipedia retrievaltask at imageclef 2010. In Working Notes of CLEF 2010, Padova, Italy, 2010.2. C. J. van Rijsbergen, editor. The Geometry of Information Retrieval. Cambridge UniversityPress, 2004.

RGU at ImageCLEF2010 Wikipedia Retrieval Task

http://oro.open.ac.uk/35335/1/clef2010labs_submission_96.pdf

RGU at ImageCLEF2010 Wikipedia Retrieval Task

Abstract

Similar works

Full text

Available Versions

Open Research Online (The Open University)