Automatic text searching for personal photos

This demonstration presents the MediAssist prototype system for organisation of personal digital photo collections based on contextual information, such as time and location of image capture, and content-based analysis, such as face detection and recognition. This metadata is used directly for identification of photos which match specified attributes, and also to create text surrogates for photos, allowing for text-based queries of photo collections without relying on manual annotation. MediAssist illustrates our research into digital photo management, showing how a combination of automatically extracted context and content-based information, together with user annotation and traditional text indexing techniques, facilitates efficient searching of personal photo collections

Using text search for personal photo collections with the MediAssist system

The MediAssist system enables organisation and searching of personal digital photo collections based on contextual information, content-based analysis and semi-automatic annotation. One mode of user interaction uses automatically extracted features to create text surrogates for photos, which enables text search of photo collections without manual annotation. Our evaluation shows that this text search facility is effective for known-item search

Context-aware person identification in personal photo collections

Identifying the people in photos is an important need for users of photo management systems. We present MediAssist, one such system which facilitates browsing, searching and semi-automatic annotation of personal photos, using analysis of both image content and the context in which the photo is captured. This semi-automatic annotation includes annotation of the identity of people in photos. In this paper, we focus on such person annotation, and propose person identification techniques based on a combination of context and content. We propose language modelling and nearest neighbor approaches to context-based person identification, in addition to novel face color and image color content-based features (used alongside face recognition and body patch features). We conduct a comprehensive empirical study of these techniques using the real private photo collections of a number of users, and show that combining context- and content-based analysis improves performance over content or context alone

Combination of content analysis and context features for digital photograph retrieval.

In recent years digital cameras have seen an enormous rise in popularity, leading to a huge increase in the quantity of digital photos being taken. This brings with it the challenge of organising these large collections. The MediAssist project uses date/time and GPS location for the organisation of personal collections. However, this context information is not always sufficient to support retrieval when faced with a large, shared, archive made up of photos from a number of users. We present work in this paper which retrieves photos of known objects (buildings, monuments) using both location information and content-based retrieval tools from the AceToolbox. We show that for this retrieval scenario, where a user is searching for photos of a known building or monument in a large shared collection, content-based techniques can offer a significant improvement over ranking based on context (specifically location) alone

Adaptive Information Cluster at Dublin City University

The Adaptive Information Cluster (AIC) is a collaboration between Dublin City University and University College Dublin, and in the AIC at DCU, we investigate and develop as one stream of our research activities, various content analysis tools that can automatically index and structure video information. This includes movies or CCTV footage and the motivation is to support useful searching and browsing features for the envisaged end-users of such systems. We bring in the HCI perspective to this highly-technically-oriented research by brainstorming, generating scenarios, sketching and prototyping the user-interfaces to the resulting video retrieval systems we develop, and we conduct usability studies to better understand the usage and opinions of such systems so as to guide the future direction of our technological research

Integrating memory context into personal information re-finding

Personal information archives are emerging as a new challenge for information retrieval (IR) techniques. The user’s memory plays a greater role in retrieval from person archives than from other more traditional types of information collection (e.g. the Web), due to the large overlap of its content and individual human memory of the captured material. This paper presents a new analysis on IR of personal archives from a cognitive perspective. Some existing work on personal information management (PIM) has begun to employ human memory features into their IR systems. In our work we seek to go further, we assume that for IR in PIM system terms can be weighted not only by traditional IR methods, but also taking the user’s recall reliability into account. We aim to develop algorithms that combine factors from both the system side and the user side to achieve more effective searching. In this paper, we discuss possible applications of human memory theories for this algorithm, and present results from a pilot study and a proposed model of data structure for the HDMs achieves

Exploring Memory Cues to Aid Information Retrieval from Personal LifeLog Archives

The expansion of personal information archives and the emerging field of Personal Lifelogs (PLs) are creating new challenges for information retrieval (IR). While studies have demonstrated the difficulties of IR for these massive data collection [1], we should also think about how we can opportunities and benefits from integrating these data sources as a component of “digital memories” , considering their rich connections with the users‟ memory. We observed that most existing approaches to personal archive IR are mostly technology-driven. Although in recent years studies in Personal Information management (PIM) have claimed to make use of the human memory features, and many works have been reported as investigating well-remembered features of computer files (documents, email, photos). Yet, these explorations are usually confined to the attributes or feature that current computer file systems or technology have provided. I believe that there are important and potentially useful data attributes that these studies have ignored. In addition, current personal search interfaces provide searching options based on what is available in the system, e.g. require users to fill in the calendar date, regardless of the fact that people actually don‟t often encode „time‟ in such a way. My PhD project aims to explore what users actually tend to recall in different personal achieve information seeking tasks, how to present searching options to cater for the right type or format of information that users can recall, and how to exploit this information in an IR system for personal lifelog archives. In this paper, I discuss the limits and advantages of some related work, and present my current and proposed study, with an outlook of an interface that I plan to develop to explore my proposals

Físchlár-DiamondTouch: collaborative video searching on a table

In this paper we present the system we have developed for our participation in the annual TRECVid benchmarking activity, specically the system we have developed, Físchlár-DT, for participation in the interactive search task of TRECVid 2005. Our back-end search engine uses a combination of a text search which operates over the automatic speech recognised text, and an image search which uses low-level image features matched against video keyframes. The two novel aspects of our work are the fact that we are evaluating collaborative, team-based search among groups of users working together, and that we are using a novel touch-sensitive tabletop interface and interaction device known as the DiamondTouch to support this collaborative search. The paper summarises the backend search systems as well as presenting the interface we have developed, in detail

Collaborative video searching on a tabletop

Almost all system and application design for multimedia systems is based around a single user working in isolation to perform some task yet much of the work for which we use computers to help us, is based on working collaboratively with colleagues. Groupware systems do support user collaboration but typically this is supported through software and users still physically work independently. Tabletop systems, such as the DiamondTouch from MERL, are interface devices which support direct user collaboration on a tabletop. When a tabletop is used as the interface for a multimedia system, such as a video search system, then this kind of direct collaboration raises many questions for system design. In this paper we present a tabletop system for supporting a pair of users in a video search task and we evaluate the system not only in terms of search performance but also in terms of user–user interaction and how different user personalities within each pair of searchers impacts search performance and user interaction. Incorporating the user into the system evaluation as we have done here reveals several interesting results and has important ramifications for the design of a multimedia search system

MediAssist: Using content-based analysis and context to manage personal photo collections

We present work which organises personal digital photo collections based on contextual information, such as time and location, combined with content-based analysis such as face detection and other feature detectors. The MediAssist demonstration system illustrates the results of our research into digital photo management, showing how a combination of automatically extracted context and content-based information, together with user annotation, facilitates efficient searching of personal photo collections