Building and Tracking Hierarchical Geographical & Temporal Partitions for Image Collection Management on Mobile Devices

International audienceUsage of mobile devices (phones, digital cameras) raises the need for organizing large personal image collections. In accordance with studies on user needs, we propose a statistical criterion and an associated optimization technique, relying on geo-temporal image metadata, for building and tracking a hierarchical structure on the image collection. In a mixture model framework, particularities of the application and typical data sets are taken into account in the design of the scheme (incrementality, ability to cope with non-Gaussian data, with both small and large samples). Results are reported on real data sets

Proceedings of the ECIR2010 workshop on information access for personal media archives (IAPMA2010), Milton Keynes, UK, 28 March 2010

Towards e-Memories: challenges of capturing, summarising, presenting, understanding, using, and retrieving relevant information from heterogeneous data contained in personal media archives. This is the proceedings of the inaugural workshop on “Information Access for Personal Media Archives”. It is now possible to archive much of our life experiences in digital form using a variety of sources, e.g. blogs written, tweets made, social network status updates, photographs taken, videos seen, music heard, physiological monitoring, locations visited and environmentally sensed data of those places, details of people met, etc. Information can be captured from a myriad of personal information devices including desktop computers, PDAs, digital cameras, video and audio recorders, and various sensors, including GPS, Bluetooth, and biometric devices. In this workshop research from diverse disciplines was presented on how we can advance towards the goal of effective capture, retrieval and exploration of e-memories

Automatic Image Event Segmentation and Quality Screening for Albuming Applications

In this paper, a system for automatic albuming of consumer photographs is described, and its specific core components of event segmentation and screening of low quality images are discussed. A novel event segmentation algorithm was created to automatically cluster pictures into events and sub-events for albuming, based on date/time metadata information as well as color content of the pictures. A new quality-screening algorithm is developed based on object quality measures, to detect problematic images due to underexposure, low contrast, and camera defocus or movement. Performance testing of these algorithms was conducted using a database of real consumer photos and showed that these functions provide a useful firstcut album layout for typical rolls of consumer pictures. A first version of the automatic albuming application software was tested through a consumer trial in the United States from August to December 1999. I

IMAGE MANAGEMENT USING PATTERN RECOGNITION SYSTEMS

With the popular usage of personal image devices and the continued increase of computing power, casual users need to handle a large number of images on computers. Image management is challenging because in addition to searching and browsing textual metadata, we also need to address two additional challenges. First, thumbnails, which are representative forms of original images, require significant screen space to be represented meaningfully. Second, while image metadata is crucial for managing images, creating metadata for images is expensive. My research on these issues is composed of three components which address these problems. First, I explore a new way of browsing a large number of images. I redesign and implement a zoomable image browser, PhotoMesa, which is capable of showing thousands of images clustered by metadata. Combined with its simple navigation strategy, the zoomable image environment allows users to scale up the size of an image collection they can comfortably browse. Second, I examine tradeoffs of displaying thumbnails in limited screen space. While bigger thumbnails use more screen space, smaller thumbnails are hard to recognize. I introduce an automatic thumbnail cropping algorithm based on a computer vision saliency model. The cropped thumbnails keep the core informative part and remove the less informative periphery. My user study shows that users performed visual searches more than 18% faster with cropped thumbnails. Finally, I explore semi-automatic annotation techniques to help users make accurate annotations with low effort. Automatic metadata extraction is typically fast but inaccurate while manual annotation is slow but accurate. I investigate techniques to combine these two approaches. My semi-automatic annotation prototype, SAPHARI, generates image clusters which facilitate efficient bulk annotation. For automatic clustering, I present hierarchical event clustering and clothing based human recognition. Experimental results demonstrate the effectiveness of the semi-automatic annotation when applied on personal photo collections. Users were able to make annotation 49% and 6% faster with the semi-automatic annotation interface on event and face tasks, respectively

Organising and structuring a visual diary using visual interest point detectors

As wearable cameras become more popular, researchers are increasingly focusing on novel applications to manage the large volume of data these devices produce. One such application is the construction of a Visual Diary from an individual’s photographs. Microsoft’s SenseCam, a device designed to passively record a Visual Diary and cover a typical day of the user wearing the camera, is an example of one such device. The vast quantity of images generated by these devices means that the management and organisation of these collections is not a trivial matter. We believe wearable cameras, such as SenseCam, will become more popular in the future and the management of the volume of data generated by these devices is a key issue. Although there is a significant volume of work in the literature in the object detection and recognition and scene classification fields, there is little work in the area of setting detection. Furthermore, few authors have examined the issues involved in analysing extremely large image collections (like a Visual Diary) gathered over a long period of time. An algorithm developed for setting detection should be capable of clustering images captured at the same real world locations (e.g. in the dining room at home, in front of the computer in the office, in the park, etc.). This requires the selection and implementation of suitable methods to identify visually similar backgrounds in images using their visual features. We present a number of approaches to setting detection based on the extraction of visual interest point detectors from the images. We also analyse the performance of two of the most popular descriptors - Scale Invariant Feature Transform (SIFT) and Speeded Up Robust Features (SURF).We present an implementation of a Visual Diary application and evaluate its performance via a series of user experiments. Finally, we also outline some techniques to allow the Visual Diary to automatically detect new settings, to scale as the image collection continues to grow substantially over time, and to allow the user to generate a personalised summary of their data

A lifelogging system supporting multimodal access

Today, technology has progressed to allow us to capture our lives digitally such as taking pictures, recording videos and gaining access to WiFi to share experiences using smartphones. People’s lifestyles are changing. One example is from the traditional memo writing to the digital lifelog. Lifelogging is the process of using digital tools to collect personal data in order to illustrate the user’s daily life (Smith et al., 2011). The availability of smartphones embedded with different sensors such as camera and GPS has encouraged the development of lifelogging. It also has brought new challenges in multi-sensor data collection, large volume data storage, data analysis and appropriate representation of lifelog data across different devices. This study is designed to address the above challenges. A lifelogging system was developed to collect, store, analyse, and display multiple sensors’ data, i.e. supporting multimodal access. In this system, the multi-sensor data (also called data streams) is firstly transmitted from smartphone to server only when the phone is being charged. On the server side, six contexts are detected namely personal, time, location, social, activity and environment. Events are then segmented and a related narrative is generated. Finally, lifelog data is presented differently on three widely used devices which are the computer, smartphone and E-book reader. Lifelogging is likely to become a well-accepted technology in the coming years. Manual logging is not possible for most people and is not feasible in the long-term. Automatic lifelogging is needed. This study presents a lifelogging system which can automatically collect multi-sensor data, detect contexts, segment events, generate meaningful narratives and display the appropriate data on different devices based on their unique characteristics. The work in this thesis therefore contributes to automatic lifelogging development and in doing so makes a valuable contribution to the development of the field

Adaptive user interfaces for mobile map-based visualisation

Mobile devices today frequently serve as platforms for the visualisation of map-based data. Despite the obvious advantages, mobile map-based visualisation (MMV) systems are often difficult to design and use. Limited screen space, resource constraints and awkward interaction mechanisms are among the many problems with which designers and users have to contend. Adaptive user interfaces (AUIs), which adapt to the individual user, represent a possible means of addressing the problems of MMV. Adaptive MMV systems are, however, generally designed in an ad-hoc fashion, making the benefits achieved difficult to replicate. In addition, existing models for adaptive MMV systems are either conceptual in nature or only address a subset of the possible input variables and adaptation effects. The primary objective of this research was to develop and evaluate an adaptive MMV system using a model-based approach. The Proteus Model was proposed to support the design of MMV systems which adapt in terms of information, visualisation and user interface in response to the user‟s behaviour, tasks and context. The Proteus Model describes the architectural, interface, data and algorithm design of an adaptive MMV system. A prototype adaptive MMV system, called MediaMaps, was designed and implemented based on the Proteus Model. MediaMaps allows users to capture, location-tag, organise and visualise multimedia on their mobile phones. Information adaptation is performed through the use of an algorithm to assist users in sorting media items into collections based on time and location. Visualisation adaptation is performed by adapting various parameters of the map-based visualisations according to user preferences. Interface adaptation is performed through the use of adaptive lists. An international field study of MediaMaps was conducted in which participants were required to use MediaMaps on their personal mobile phones for a period of three weeks. The results of the field study showed that high levels of accuracy were achieved by both the information and interface adaptations. High levels of user satisfaction were reported, with participants rating all three forms of adaptation as highly useful. The successful implementation of MediaMaps provides practical evidence that the model-based design of adaptive MMV systems is feasible. The positive results of the field study clearly show that the adaptations implemented were highly accurate and that participants found these adaptations to be useful, usable and easy to understand. This research thus provides empirical evidence that the use of AUIs can provide significant benefits for the visualisation of map-based information on mobile devices