The holy grail of multimedia indexing and retrieval is developing algorithms capable of imitating human abilities in distinguishing and recognising semantic concepts within the content, so that retrieval can be based on ”real world” concepts that come naturally to users. In this paper, we discuss an approach to using segmented video objects as the midlevel connection between low-level features and semantic

concept description. In this paper, we consider a video object as a particular instance of a semantic concept and we

model the semantic concept as an average representation

of its instances. A system supporting object-based search

through a test corpus is presented that allows matching presegmented objects based on automatically extracted lowlevel features. In the system, relevance feedback is employed to drive the learning of the semantic model during

a regular search process

Murphy, Noel

O'Connor, Noel E.

Sav, Sorin Vasile

Smeaton, Alan F.

English

The holy grail of multimedia indexing and retrieval is developing algorithms capable of imitating human abilities in distinguishing and recognising semantic concepts within the content, so that retrieval can be based on ”real world” concepts that come naturally to users. In this paper, we discuss an approach to using segmented video objects as the midlevel connection between low-level features and semantic\ud
concept description. In this paper, we consider a video object as a particular instance of a semantic concept and we\ud
model the semantic concept as an average representation\ud
of its instances. A system supporting object-based search\ud
through a test corpus is presented that allows matching presegmented objects based on automatically extracted lowlevel features. In the system, relevance feedback is employed to drive the learning of the semantic model during\ud
a regular search process

Name not available

ASSOCIATING LOW-LEVEL FEATURES WITH SEMANTIC CONCEPTS USING VIDEOOBJECTS AND RELEVANCE FEEDBACKSorin Sav, Noel O’Connor, Alan Smeaton, Noel MurphyCentre for Digital Video Processing, Dublin City University, IrelandABSTRACTThe holy grail of multimedia indexing and retrieval is de-veloping algorithms capable of imitating human abilities indistinguishing and recognising semantic concepts within thecontent, so that retrieval can be based on ”real world” con-cepts that come naturally to users. In this paper, we discussan approach to using segmented video objects as the mid-level connection between low-level features and semanticconcept description. In this paper, we consider a video ob-ject as a particular instance of a semantic concept and wemodel the semantic concept as an average representationof its instances. A system supporting object-based searchthrough a test corpus is presented that allows matching pre-segmented objects based on automatically extracted low-level features. In the system, relevance feedback is em-ployed to drive the learning of the semantic model duringa regular search process.1. INTRODUCTIONThe continuous expansion of multimedia archives has re-sulted in an increasing demand for effective informationmanagement systems. Content-based information retrieval(CBIR) systems are required to assist humans in locatingand retrieving relevant content. Ideally, retrieval should beperformed using concepts and queries that come natural tohumans. However, retrieval systems often operate with low-level representations [1], more accessible for machine pro-cessing, whilst humans perceive multimedia at the seman-tic level [2]. Similarity measures constructed on low-levelfeatures [3] [4] [5], such as colour, texture and shape, willonly capture analogous features and to move beyond this,concept identification is required. However, unconstrainedobject recognition and subsequent concept identification isconsidered to remain far beyond the capabilities of the re-search community in the near future [6]. To make mattersworse, mapping between feature level descriptions and hu-man concepts is ambiguous due to multiple concepts shar-ing similar distributions of features.Relevance feedback has been proposed as an approachthat allows the user to implicitly construct a feature-conceptmapping by directly influencing the retrieval process. Orig-inally in image retrieval, the knowledge obtained from userinteraction was regarded as specific to each query and there-fore discarded at the end of the session. Recent approaches,however, attempt to further use this information by infer-ring the semantic space [7] driving the user’s actions. Fur-thermore, the inferred semantic space, could be later usedin building connections among the images in the archive [8]or for propagating annotation terms [9] in order to subse-quently aid retrieval for other users.Humans can effortlessly assess visual content in mid-level terms (colour, shape, texture), but find it difficult to de-tect and express variations in low-level features (e.g. colourmoments, shape moments, texture descriptors). In retrievalsystems that work only on low-level features, feedback fromusers is generally limited to the entire content of the im-age. In systems that divide the image into regions (auto-matically or by user interaction) feedback can be given foreach separate region in a partition. It has been argued thatfeedback effectiveness increases as the partitions employedmore closely express the semantics of the scene [10]. There-fore, in this paper we consider extracting concept descrip-tions based on low-level features directly related to humanperception of the objects present in an image.The remainder of this paper is organized as follows.Section 2 provides an overview of the overall approach.Object feature representation is described in Section 3. Theuse of relevance feedback is explained in Section 4 and ex-perimental results are presented in Section 5. We presentconclusions and ideas for future work in Section 6.2. ASSOCIATING OBJECTS WITH SEMANTICCONCEPTSWe approach retrieval from a combined perspective: as shortterm learning for immediate increase of retrieval performanceand as long term learning with the view of modelling thesemantics of the query. Short term improvement of the re-trieval performance means presenting more relevant itemsto the user, while for long term learning it is critical to ob-tain a large set of positive and negative training examples.The proposed system uses the semantic video objectsextracted from the keyframe associated to the video shots inthe archive. When new shots are added to the archive foreach keyframe the most representative video objects are ex-tracted using the interactive tool for segmentation describedin [11]. Each object is automatically described in terms ofcolour and shape (see Section 3). Colour and shape are usedas they are independent features directly related to humanperception of objects. For images with no obvious video ob-jects, the colour distribution of the entire image is used, astypically such images represent scenery with no foregroundobjects. In this case, shape information is not considered.During the retrieval process, clusters of features are as-sociated to relevant/irrelevant objects based on user interac-tion. The relevance feedback approach employed (see Sec-tion 4) creates clusters of low-level features. The clustersthat cover relevant objects are associated with the semanticconcept conveyed by the keyword used to identify the query.3. OBJECT FEATURE REPRESENTATION3.1. ColorTo represent colour we adopted the MPEG-7 dominant colourdescriptor (DCD) [1] as used in many retrieval systems. Inorder to adapt this feature to objects as oppose to imagesonly the interior of the object is considered.The recommended distance to be used with DCD is [12]:DDCD(Q, I) = N1∑i=1p21i +N2∑j=1p22j −N1∑i=1N2∑j=12α1i,2j1/2(1)where N is a set of colour vectors ci, and pi their per-centages. The similarity coefficient αk,l between two RGBcolor vectors ck and cl is calculated as:αk,l ={1− Dk,lDmax , Dk,l ≤ Td0, Dk,l > Td(2)where Dk,l =‖ ck − cl ‖ represents the Euclidian distancebetween two colour vectors, Td = 20, α = 1, and Dmax =αTd = 20 as suggested in [13].3.2. ShapeShape description and similarity is an extremely complexresearch topic – 2D projection on the image plane, elas-tic deformations of the object, and diversity of shapes inwhich instances of the same semantic object appear in thereal world are common problems that must be consideredfor shape similarity. In this initial work, we use a relativelysimple shape descriptor corresponding to the compactnessmoment γ [14]:γ =P24piA(3)where A is the area and P perimeter of the video object.This is a simple and robust descriptor that can indicate adegree of shape similarity.4. BUILDING FEATURE CLUSTERSSince the object descriptors are independent, the probabilitydistribution of a semantic concept over an image descriptorspace can be modeled as a Gaussian mixture model:p(ε) =N∑j=1αjpj(ε|j) (4)where∑Nj=1 αj = 1. Each mixture component is a Gaus-sian with mean µ and covariance matrix Σ:p(ε|j) = 12pi|Σj | 12exp−12 (ε−µj)TΣ−1j (ε−µj) (5)At the start of the retrieval session the user selects aquery image, and thus implicitly the extracted object asso-ciated with that image. The colour and shape of the queryobject are considered as the means µ of two independentGaussian clusters, one in the colour space, the other one inthe shape space. An arbitrary variance Σ is assumed for theinitial clusters. The mean µ and variance Σ are then sub-sequently estimated from user labelled examples. Imagescontaining objects similar to the query object are retrievedusing the Mahalanobis distance and presented to the user ex-plicitly rated as positive or negative instances relative to thequery. We argue that presenting rated results makes the usermore attentive to correcting by feedback erroneousness re-sults. Thus, within the list of relevant results we deliberatelyinsert low confidence items in order to elicit user feedback.In conformity with the Minimum Description Length(MDL) principle we assume the best semantic model to bethe Gaussian mixture of low-level features with the mini-mum number of components that correctly classifies the la-beled set of images. Intuitively, this occurs when the major-ity of items in a cluster are similarly labelled (positive/nega–tive) and the cluster contains no oppositely labelled items.Clusters of positively labelled items are assumed to gener-ate positive instances whilst clusters of negative items areassumed to generate negative instances. This is a weak as-sumption, however, as labelled items are sparsely distributedwithin the feature space. Therefore, we seek to verify thevalidity of the assumption by inserting negative sampleswithin the cluster as relevant results (low-confidence or neg-ative instances), expecting the user to label them appropri-ately. The Gaussian mixture is updated, when feedback isFig. 1. Retrieved relevant items against iterations for eachpredefined category of objectavailable, based on a estimation-maximisation (EM) tech-nique [15].The EM algorithm consists of two steps an expectationstep (E) and a maximisation step(M). In the E step a givennumber of clusters are build based on current estimate ofthe positive items distribution in the feature space. In the Mstep the underlying distributions of the positive items are re-estimated. These two steps are iterated until convergence.5. EXPERIMENTAL RESULTSFor experimental purposes, we created a test corpus of ob-jects culled from 2000 images from the Corel dataset and1137 keyframes extracted from 2 hours of TV broadcastdrawn from the TREC 2003 [16] corpus. In each imagea single dominant object was manually segmented. We de-fined five categories of objects – people, cars, tigers, planesand eagles – and manually classified all segmented objectsto one category in order to create a ground truth. Low-levelfeatures were extracted for all objects. Experiments wereperformed with an expert user selecting an initial query ob-ject and providing negative/positive feedback. Each querysession was stopped after 25 feedback iterations.Figure 1 shows the number of relevant items (as judgedagainst the ground truth) retrieved against the number of it-erations for each category. As can be seen the retrieval per-formance is better in the people category due to the sim-ilarities of object shape. In the eagles category there islarge variation in shape and consequently the retrieval per-formance is lower.Figures 2, 3 and 4 show the first 16 images retrieved forthe people, tigers and eagles categories with the segmentedobject indicated in each case. In each case, the image in theupper left corner is the initial query image.6. CONCLUSIONS AND FUTURE WORKIn this paper we presented an approach to automatically as-sociating low-level features extracted from segmented videoobjects to semantic concepts. The features used are thecolour and shape of the objects. We use video objects asthe mid-level link between low-level features and semanticconcepts whereby we consider a video object as a particu-lar instance of a semantic concept. Relevance feedback isemployed to model the semantic concept as an average rep-resentation of its instancesOur main efforts in the future will focus on using morefeatures other than those reported here and expanding therange of experiments and the size of the image corpus. Otherwork will concentrate on determining a criterion for select-ing the minimum number of feature clusters that satisfacto-rily cover the instances of a given concept.7. ACKNOWLEDGMENTSThe support of the Enterprise Ireland Informatics Initiativeis gratefully acknowledged. This material is based uponwork supported by the IST programme of the EU in theproject IST-2000-32795 SCHEMA.8. REFERENCES[1] P. Salambier and J.R. Smith, “MPEG-7 multimediadescriptions schemes,” IEEE Transactions Circuitsand Systems for Video Technology, vol. 11, pp. 748–759, June 2001.[2] J.R. Smith and S.F. Chang, “Visualseek: A fully au-tomated content-based query system,” Proccedingsof Fourth International Conference on Multimedia,ACM, pp. 87–92, 1996.[3] M. Flickner, H. Sawhney, W. Niblack, J. Ashley, Q.Huang, B. Dom, M. Gorkani, J. Hafner, D. Lee, D.Petkovic, D. Steele and P. Yanker, “Query by imageand video content: The QBIC system,” IEEE Com-puter, , no. 28, pp. 23–32, September 1995.[4] W.Y. Ma and B.S. Manjunath, “Netra: A toolbox fornavigating large image databases,” ACM MultimediaSystems, , no. 7, pp. 184–198, 1999.[5] Y. Rui, T.S. Huang, S. Methrotra and M. Ortega,“A relevance feedback architecture for content-basedmultimedia information retrieval system,” Proceed-ings of IEEE Workshop on Content-based Access ofImage and Video Libraries, pp. 82–89, 1997.[6] S. Santini and R. Jain, “Visual navigation in percep-tual databases,” Proccedings of International Confer-ence on Visual Information Systems, San Diego, CA,December 1997.[7] X. He, O. King, W.Y. Ma, M. Li and H.J. Zhang,“Learning a semantic space from user’s relevancefeedback for image retrieval,” IEEE Transactions onCircuits and Systems for Video Technology, vol. 13,no. 1, pp. 39–48, January 2003.[8] M. Lee, W.Y. Ma and H.J. Zhang, “Information em-bbeding based on user’s relevance feedback for imageretrieval,” Proceedings of SPIE Multimedia Storageand Archiving Systems IV, Boston, September 1999.[9] B. Li, K. Goh and E.Y. Chang, “Confidence-baseddynamic ensamble for image annotation and seman-tic discovery,” Proceeding of the 11th ACM Interna-tional Conference on Multimedia, Berkeley, CA, USA,November 2003.[10] S.F. Chang, W. Chen and H. Sundaram, “Semanticvisual templates: Linking visual features to seman-tics,” Proceeding of IEEE International Conferenceon Image Processing 1998, Chicago, IL, USA, Octo-ber 1998.[11] N. O’Connor, T. Adamek, S. Sav, N. Murphy and S.Marlow, “QIMERA: A software platform for videoobject segmentation and tracking,” Proceeding of the4th European Workshop on Image Analysis for Multi-media Interactive Services, London, UK, April 2003.[12] B. Manjunath, P. Salambier and T. Sikora, Introduc-tion to MPEG-7: Multimedia content description in-terface, Wiley, New York, USA, 2002.[13] A. Kushki, P. Androutsos, K.N. Plataniotis and A.N.Venetsanopoulos, “Query feedback for interactive im-age retrieval,” IEEE Transactions on Circuits and Sys-tems for Video Technology, vol. 14, no. 5, pp. 644–655,May 2004.[14] S. Theodoridis and K. Koutroumbas, Pattern Recog-nition, Academic Press, 1999.[15] T. K. Moon, “The Expectation-Maximisation Algo-rithm,” IEEE Signal Processing Magazine, pp. 47–60,November 1996.[16] TrecVid 2003 web site:http://www nlpir.nist.gov/projects/tv2003/tv2003.html.Fig. 2. Items retrieved in the people categoryFig. 3. Items retrieved in the tigers categoryFig. 4. Items retrieved in the eagles category

Associating low-level features with semantic concepts using video objects and relevance feedback

DCU Online Research Access Service

A relevance feedback architecture for content-based multimedia information retrieval system,”

Conﬁdence-based dynamic ensamble for image annotation and semantic discovery,”

Information embbeding based on user’s relevance feedback for image retrieval,”

Introduction to MPEG-7: Multimedia content description interface,

Learning a semantic space from user’s relevance feedback for image retrieval,”

multimedia descriptions schemes,”

Netra: A toolbox for navigating large image databases,”

Pattern Recognition,

QIMERA: A software platform for video object segmentation and tracking,”

Query by image and video content: The QBIC system,”

Query feedback for interactive image retrieval,”

Semantic visual templates: Linking visual features to semantics,”

The Expectation-Maximisation Algorithm,”

Visualseek: A fully automated content-based query system,”

The holy grail of multimedia indexing and retrieval is developing algorithms capable of imitating human abilities in distinguishing and recognising semantic concepts within the content, so that retrieval can be based on ”real world” concepts that come naturally to users. In this paper, we discuss an approach to using segmented video objects as the midlevel connection between low-level features and semantic
concept description. In this paper, we consider a video object as a particular instance of a semantic concept and we
model the semantic concept as an average representation
of its instances. A system supporting object-based search
through a test corpus is presented that allows matching presegmented objects based on automatically extracted lowlevel features. In the system, relevance feedback is employed to drive the learning of the semantic model during
a regular search process

O\u27Connor, Noel E.

Irish Universities

http://doras.dcu.ie/403/1/wiamis_2005_2.pdf

Associating low-level features with semantic concepts using video objects and relevance feedback

Abstract

Similar works

Full text

Available Versions

Name not available

DCU Online Research Access Service

Irish Universities