This demonstration presents the MediAssist prototype system for organisation of personal digital photo collections based on contextual information, such as time and location of image capture, and content-based analysis, such as face detection and recognition. This metadata is used directly for identification of photos which match specified attributes, and also to create text surrogates for photos, allowing for text-based queries of photo collections without relying on manual annotation. MediAssist illustrates our research into digital photo management, showing how a combination of automatically extracted context and content-based information, together with user annotation and

traditional text indexing techniques, facilitates efficient searching of personal photo collections

Cooray, Saman H.

Gurrin, Cathal

Jones, Gareth J.F.

Lee, Hyowon

Malobabić, Jovanka

O'Connor, Noel E.

O'Hare, Neil

Smeaton, Alan F.

Uscilowski, Bartlomiej

English

This demonstration presents the MediAssist prototype system for organisation of personal digital photo collections based on contextual information, such as time and location of image capture, and content-based analysis, such as face detection and recognition. This metadata is used directly for identification of photos which match specified attributes, and also to create text surrogates for photos, allowing for text-based queries of photo collections without relying on manual annotation. MediAssist illustrates our research into digital photo management, showing how a combination of automatically extracted context and content-based information, together with user annotation and\ud
traditional text indexing techniques, facilitates efficient searching of personal photo collections

Name not available

Automatic Text Searching For Personal PhotosNeil O’Hare, Hyowon Lee, Saman Cooray, Cathal Gurrin, Gareth J.F. Jones,Jovanka Malobabic, Noel E. O’Connor, Alan F. Smeaton, and Bartlomiej UscilowskiAbstract— This demonstration presents the MediAssist proto-type system for organisation of personal digital photo collectionsbased on contextual information, such as time and location ofimage capture, and content-based analysis, such as face detectionand recognition. This metadata is used directly for identificationof photos which match specified attributes, and also to create textsurrogates for photos, allowing for text-based queries of photocollections without relying on manual annotation. MediAssistillustrates our research into digital photo management, show-ing how a combination of automatically extracted context andcontent-based information, together with user annotation andtraditional text indexing techniques, facilitates efficient searchingof personal photo collections.Index Terms— Personal Photo Management, Text Search, Con-textI. INTRODUCTIONIn recent years digital photography has become increasinglypopular, resulting in the accumulation of large numbers ofpersonal digital photos. The MediAssist project [6] at theCentre for Digital Video Processing (CDVP) addresses thissituation by developing tools for the efficient searching ofphoto archives. The system uses both automatically generatedcontextual metadata (eg. time, location) and content-basedanalysis tools (eg. face detection and recognition). Semi-automatic annotation allows the user to interactively improvethe automatically generated annotations. Retrieval tools allowfor complex query formulation, in addition to the facility tocreate simple text queries, based on these features. In previouswork using context for photo management, Davis et al [1]utilised context to recommend recipients for sharing photostaken with a context-aware phone, although their system doesnot support retrieval. Naaman et al [4] use context-basedfeatures for photo management, but they do not use content-based analysis tools, or facilitate semi-automatic annotationor text-based searches. There is also a huge body of workon content-based image retrieval [10], but it has been shownthat users to not find this facility useful for personal photomanagement [9].II. CONTENT AND CONTEXT-AWARE PHOTOORGANISATIONThe MediAssist photo archive contains over 17,000location-stamped photos taken with a number of differentcamera models, including camera phones. Over 11,000 ofthese have been manually annotated for a number of concepts,including buildings, indoor/outdoor and the presence andAll authors are members of the Centre for Digital Video Processing, DublinCity University. Alan F. Smeaton and Noel E. O’Connor are members of theAdaptive Information Cluster, Dublin City University.email: nohare@computing.dcu.ieFig. 1. The MediAssist Photo Management Systemidentity of faces. This manually annotated dataset serves as aground truth for the evaluation of content-based analysis tools,and also can be used to bootstrap semi-automatic tools (whichdepend on a certain level of user annotation). All photos areindexed using both context and content-based analysis. Timeand location of photo capture are used to derive additionalcontextual information such as daylight status, weather andindoor/outdoor classification [7]. A face detection system isused to detect the presence of frontal-view faces [2]. Othercontent-based tools used include body patch (the area underthe face, modelling the clothes worn by the individual) fea-ture extraction [2], face recognition using ICA (IndependentComponent Analysis) and building detection based on thedistribution of edges in the image [7]. All of this informationcan prove very useful for searching photo collections.III. THE MEDIASSIST WEB DEMONSTRATOR SYSTEMThe MediAssist Web-based desktop interface allows usersto search through their personal photo collections using thecontextual and content-based features described above. TheMediAssist system interface is shown in Fig. 1. Our earlierversion of the MediAssist prototype supported filter-basedsearching using the photo metadata features [6]. The newversion presented here has been extended to include free-textranked information retrieval functionality.A. Filter-Based SearchThe system presents search options enabling a user to enterdetails of desired locations, times, and advanced options suchas people present, weather, light status, indoor/outdoor andbuilding/non-building. Semi-automatic person identificationrelies on a combination of automatic methods and manual an-notation as described below. Time filters enable powerful time-based queries, for example all photos taken in the evening, atthe weekend, during the summer or within certain date ranges.B. Text Search InterfaceFor text-based search the automatic context and content-based features are mined to construct text surrogates for allphotos, creating a textual equivalent of each feature (e.g. if thedate is October 21th 2006, the text ‘october autumn saturdayweekend 21 twenty-first 2006’ would form a surrogate textualdescription). So an image might have the text ‘dublin irelandseptember weekend afternoon person alan’ associated with it,representing the features location, time, face detection andperson annotation. We index the text document associated withan image using a conventional text search engine based onthe standard BM25 information retrieval model [8]. We alsocreate text surrogates for ‘events’ (see below) to allow for text-based searching of events in the ‘Event List’ view describedbelow. The system presents a text search box to allow forthe quick and easy formulation of text queries based on thecontent and context features described above. We will conductan evaluation of this search interface in future work.C. Collection BrowsingFour different views are available to present the results ofsearches. The default view, Event List, organises the filteredphotos into ‘events’ in which the photos are grouped togetherbased on time proximity, by detecting large temporal gapsbetween consecutively captured photos, similar to [3]. Eachevent is summarized by a label (location and date/time) andfive representative thumbnail photos selected based on thequery. Event Detail is composed of the full set of photos inan event, automatically organized into sub-events. IndividualPhoto List is an optional view where the thumbnail size photosare presented without any particular event grouping, but sortedby date/time. Photo Detail is an enlarged single photo viewpresented when the user selects one of the thumbnail size pho-tos in any of the above views. In all of the above presentationoptions, each photo is presented with its associated automaticannotation information.D. Semi-Automatic AnnotationMediAssist allows users to manually change or update anyof the automatically tagged information for a single photo orfor a group of photos. In Photo Detail view, the user canhighlight all detected faces in the photo and tidy up the resultsof the automatic detection by removing false detections oradding missed faces. The system uses a body patch feature (i.e.a feature modeling the clothes worn by a person) combinedwith a face recognition feature to suggest names for detectedfaces: the suggested name for an unknown face is the knownface with the most similar body patch and face [2]. The usercan confirm the system choice or choose from a shortlist ofsuggested names, again based on similarity. Other work hasshown effective methods of suggesting identities within photosusing context-based data [5]: in our ongoing research we areexploring the combination of this type of approach with bothface recognition and body-patch matching.IV. CONCLUSIONSWe have presented the MediAssist demonstrator system forcontext-aware management of personal digital photo collec-tions. Automatically extracted features are supplemented withsemi-automatic annotation which allows the user to correct oradd to the automatically generated annotations. The systemallows the user to formulate precise queries using content andcontext based features, or alternatively the user can formulatesimple text queries, which are enabled without the need formanual annotation. We plan to leverage context metadata toimprove on the performance of content analysis tools [7], andwe will use combined context and content-based approachesto identity annotation, based on face recognition, body-patchmatching and contextual information. We will also extend theintegration of person recognition to enable the user to query fora given individual, and the system will (in addition to returningthe photos with confirmed annotations) return a ranked list ofcandidate photos which should contain this person.ACKNOWLEDGMENTSThe MediAssist project is supported by Enterprise Irelandunder Grant No CFTD-03-216. This work is partly supportedby Science Foundation Ireland under Grant No 03/IN.3/I361REFERENCES[1] S. Ahern, S. King, and M. Davis. MMM2: mobile media metadatafor photo sharing. In ACM Multimedia, pages 267–268, Singapore,November 2005.[2] S. Cooray, N. O’Connor, C. Gurrin, G. Jones, N. O’Hare, and A. F.Smeaton. Identifying person re-occurrences for personal photo man-agement applications. In VIE 2006, pages 144–149, Bangalore, India,September 2006.[3] A. Graham, H. Garcia-Molina, A. Paepcke, and T. Winograd. Time asessence for photo browsing through personal digital libraries. In ACMJoint Conference on Digital Libraries, pages 326–335, Portland, USA,July 2002.[4] M. Naaman, S. Harada, Q. Wang, H. Garcia-Molina, and A. Paepcke.Context data in geo-referenced digital photo collections. In ACMMultimedia, pages 196–203, New York, USA, October 2004.[5] M. Naaman, R. B. Yeh, H. Garcia-Molina, and A. Paepcke. Leveragingcontext to resolve identity in photo albums. In ACM Joint Conferenceon Digital Libraries, pages 178–187, Denver, CO, USA, June 2005.[6] N. O’Hare, C. Gurrin, H. Lee, N. Murphy, A. F. Smeaton, and G. Jones.Digital photos: Where and when? In ACM Multimedia 2005, pages 261–262, Singapore, November 2005.[7] N. O’Hare, H. Lee, S. Cooray, C. Gurrin, G. Jones, J. Malobabic,N. O’Connor, A. F. Smeaton, and B. Uscilowski. Mediassist: Usingcontent-based analysis and context to manage personal photo collections.In CIVR 2006, pages 529–532, Tempe, AZ, July 2006.[8] S. E. Robertson, S. Walker, , S. Jones, M. M. Hancock-Beaulieu, andM. Gatford. Okapi at TREC-3. In Proceedings of the Third TextREtrieval Conference (TREC-3), pages 109–126, NIST, November 1995.[9] K. Rodden and K. R. Wood. How do people manage their digitalphotographs? In CHI 2003, pages 409–416, Florida, USA, April 2003.[10] A. Smeulders, M. Worring, S. Santini, A. Gupta, and A. Jain. Content-based image retrieval at the end of the early years. IEEE Transactionson Pattern Analysis and Machine Intelligence, 22(12):1349–1380, 2000.

Automatic text searching for personal photos

This demonstration presents the MediAssist prototype system for organisation of personal digital photo collections based on contextual information, such as time and location of image capture, and content-based analysis, such as face detection and recognition. This metadata is used directly for identification of photos which match specified attributes, and also to create text surrogates for photos, allowing for text-based queries of photo collections without relying on manual annotation. MediAssist illustrates our research into digital photo management, showing how a combination of automatically extracted context and content-based information, together with user annotation and
traditional text indexing techniques, facilitates efficient searching of personal photo collections

O\u27Hare, Neil

O\u27Connor, Noel E.

Irish Universities

DCU Online Research Access Service

Contentbased image retrieval at the end of the early years.

Context data in geo-referenced digital photo collections.

Digital photos: Where and when?

How do people manage their digital photographs?

Identifying person re-occurrences for personal photo management applications.

Leveraging context to resolve identity in photo albums.

Mediassist: Using content-based analysis and context to manage personal photo collections. In CIVR

MMM2: mobile media metadata for photo sharing.

Okapi at TREC-3.

Time as essence for photo browsing through personal digital libraries.

http://doras.dcu.ie/311/1/samt_2006_3.pdf

Automatic text searching for personal photos

Abstract

Similar works

Full text

Available Versions

Name not available

Irish Universities

DCU Online Research Access Service