Building and Tracking Hierarchical Geographical & Temporal Partitions for Image Collection Management on Mobile Devices

International audienceUsage of mobile devices (phones, digital cameras) raises the need for organizing large personal image collections. In accordance with studies on user needs, we propose a statistical criterion and an associated optimization technique, relying on geo-temporal image metadata, for building and tracking a hierarchical structure on the image collection. In a mixture model framework, particularities of the application and typical data sets are taken into account in the design of the scheme (incrementality, ability to cope with non-Gaussian data, with both small and large samples). Results are reported on real data sets

IMAGE MANAGEMENT USING PATTERN RECOGNITION SYSTEMS

With the popular usage of personal image devices and the continued increase of computing power, casual users need to handle a large number of images on computers. Image management is challenging because in addition to searching and browsing textual metadata, we also need to address two additional challenges. First, thumbnails, which are representative forms of original images, require significant screen space to be represented meaningfully. Second, while image metadata is crucial for managing images, creating metadata for images is expensive. My research on these issues is composed of three components which address these problems. First, I explore a new way of browsing a large number of images. I redesign and implement a zoomable image browser, PhotoMesa, which is capable of showing thousands of images clustered by metadata. Combined with its simple navigation strategy, the zoomable image environment allows users to scale up the size of an image collection they can comfortably browse. Second, I examine tradeoffs of displaying thumbnails in limited screen space. While bigger thumbnails use more screen space, smaller thumbnails are hard to recognize. I introduce an automatic thumbnail cropping algorithm based on a computer vision saliency model. The cropped thumbnails keep the core informative part and remove the less informative periphery. My user study shows that users performed visual searches more than 18% faster with cropped thumbnails. Finally, I explore semi-automatic annotation techniques to help users make accurate annotations with low effort. Automatic metadata extraction is typically fast but inaccurate while manual annotation is slow but accurate. I investigate techniques to combine these two approaches. My semi-automatic annotation prototype, SAPHARI, generates image clusters which facilitate efficient bulk annotation. For automatic clustering, I present hierarchical event clustering and clothing based human recognition. Experimental results demonstrate the effectiveness of the semi-automatic annotation when applied on personal photo collections. Users were able to make annotation 49% and 6% faster with the semi-automatic annotation interface on event and face tasks, respectively

Building and Tracking Hierarchical Geographical & Temporal Partitions for Image Collection Management on Mobile Devices

International audienceUsage of mobile devices (phones, digital cameras) raises the need for organizing large personal image collections. In accordance with studies on user needs, we propose a statistical criterion and an associated optimization technique, relying on geo-temporal image metadata, for building and tracking a hierarchical structure on the image collection. In a mixture model framework, particularities of the application and typical data sets are taken into account in the design of the scheme (incrementality, ability to cope with non-Gaussian data, with both small and large samples). Results are reported on real data sets

An Experimental investigation of presentation medium–dependent differences of picture consumption by college-aged adults

While digital camera owners are taking more photos than ever before, most of them are not printing the photos. When they do, they only print in small quantities. The reason for this is that most users share their photos online or through e-mail. While fewer people print their images at the moment they are taking them, they are saving the digital files of the photos for future use. In conducting a literature review, a good overview was acquired of the current consumer photographer\u27s practices in taking, sharing, and saving pictures. In addition, a first experiment was set up involving college-aged young adults as the population. This first experiment focused on presentation medium-dependent differences in picture consumption, as well as consumer printing behavior regarding their own photographs. A following experiment took a second look at presentation medium-dependent differences in picture consumption. In addition, it provided a more complete picture of sharing and saving behavior, as well as an understanding of the value that observers place on conventional photographic images. The outcome of these experiments showed that most participants preferred printed images over on-screen images. Regardless of this finding, participants did not print images very often for a variety of reasons, including lack of time or money. In addition, results showed that the most commonly used printing tools included Kodak Gallery EasyShare, Shutterfly, and Flickr. Finally, participants cited Photoshop, Lightroom, and Picasa as the primary editing tools, with Facebook being mentioned as the main sharing tool

Event-centric management of personal photos

Since the last decade we have been observing a tremendous growth in the size of personal photo collections. For this reason, and due to the lack of proper automatic classification and annotation in standard album-centric photo software, users find it increasingly difficult to organise and make use of their photos. Although automatic annotation of media content can work to achieve more sophisticated multimedia classification and retrieval if its used in combination with rich knowledge representations, it still requires the availability of well-annotated training sets to produce the type of higher-level descriptions that would be of interest to casual users. Thus, the applicability of this approach is highly unlikely in the broad domain of personal photography. Recent developments in the media industry show an interest towards the organisation and structuring of media collections using an event-centric metaphor. This event-centric approach is inspired by strong research in psychology on how our autobiographical memory works to organise, recollect and share our life experiences. While this metaphor is backed by some early user studies, these were led before the large adoption of social media sharing services and there has been little recent research on how users actually use events digitally to organise and share their media. In this work we first present an updated study on what users are doing with their photos in current online platforms to support the suitability of an event-centric approach. Next, we introduce a simple framework for event-centric personal photo management focused on temporal and spatial aspects and through it we describe our techniques for automatic photo organisation and sharing. Finally, we propose a platform for personal photo management that makes use of these automatic techniques and present an evaluation of a prototypical implementation

Proceedings of the ECIR2010 workshop on information access for personal media archives (IAPMA2010), Milton Keynes, UK, 28 March 2010

Towards e-Memories: challenges of capturing, summarising, presenting, understanding, using, and retrieving relevant information from heterogeneous data contained in personal media archives. This is the proceedings of the inaugural workshop on “Information Access for Personal Media Archives”. It is now possible to archive much of our life experiences in digital form using a variety of sources, e.g. blogs written, tweets made, social network status updates, photographs taken, videos seen, music heard, physiological monitoring, locations visited and environmentally sensed data of those places, details of people met, etc. Information can be captured from a myriad of personal information devices including desktop computers, PDAs, digital cameras, video and audio recorders, and various sensors, including GPS, Bluetooth, and biometric devices. In this workshop research from diverse disciplines was presented on how we can advance towards the goal of effective capture, retrieval and exploration of e-memories

Valokuvaajasta katsojaan – mikä tekee toimituksellisista kuvista kiinnostavia?

The licentiate thesis focused on the editorial interestingness of photos. The term editorial interest was coined in the study; it refers to all factors that contribute to making a photograph editorially interesting in professional workflow. The thesis considered the whole process from taking, editing and selecting editorial photos to viewing the final, published photos. Three research questions were set for the study. The first concentrated on the factors that influ-ence the editorial interestingness of photos in the process of their creation and selection. The sec-ond question dealt with the factors, which affect the interestingness of editorial photos according to viewers. These two questions were approached through three case studies. The third research question analyzed what factors of editorial interest are overarching independent of the scenario. The two first cases examined the production perspective of news photos: the first with observa-tions and thematic interviews in a photo agency, the second through a survey integrated in the publishing system of four newspapers. The third case study was conducted as an interestingness evaluation with supporting interviews in a laboratory setting with amateur participants and maga-zine photos as material. All three cases had five factors of editorial interestingness in common; those factors were over-arching independent of test setting, photo genre, and role of evaluator. The factors were aesthetics, affect, novelty-complexity, semantics, and utility. The editorial interestingness of a photo was thus influenced by aesthetic criteria, the ability of the photo to evoke emotion, unexpectedness, con-tent-related criteria, and versatile possibilities of photo use. Viewers considered editorial interestingness to encompass two additional factors: the ability of the photo to attract and hold attention, and self-reference experienced by the viewer. These two were confirmed by literature, which suggested also one additional factor, so called coping potential, i.e. person’s ability to adapt and to cope with a novel, complex view.Lisensiaatintyön aiheena oli valokuvien toimituksellinen kiinnostavuus. Toimituksellinen kiinnostavuus asetettiin työssä kattotermiksi, jonka alle kuuluvat kaikki kuvan kiinnostavuuteen ammattilaistyönkulussa vaikuttavat tekijät. Työssä tarkasteltiin koko prosessia kuvan ottamisesta, käsittelystä ja valinnasta valmiin, julkaistun kuvan katseluun. Työlle asetettiin kolme tutkimuskysymystä. Ensimmäinen keskittyi tekijöihin, jotka vaikuttavat toimituksellisten kuvien kiinnostavuuteen niiden luonti- ja valintavaiheessa. Toinen kysymys selvitti, mitkä tekijät vaikuttavat toimituksellisten kuvien kiinnostavuuteen katsojan näkökulmasta. Näitä kahta tutkimuskysymystä lähestyttiin kolmen tapaustutkimuksen kautta. Kolmas kysymys analysoi, mitkä toimituksellisen kiinnostavuuden tekijät ovat yhdistäviä tilanteesta riippumatta. Kaksi ensimmäistä tutkimusta selvittelivät tuotantonäkökulmaa uutiskuvien puolella: ensimmäinen havainnointien ja teemahaastattelujen kautta kuvatoimistossa ja jälkimmäinen toimitusjärjestelmään integroidulla kyselyllä neljässä sanomalehdessä. Kolmas tapaustutkimus toteutettiin aikakauslehtikuvien kiinnostavuusarviointeina ja tarkentavina haastatteluina laboratoriossa, amatööreillä koehenkilöillä. Kaikissa kolmessa tutkimuksessa nousi esiin viisi toimituksellisten kuvien kiinnostavuuteen liittyvää osatekijää, jotka yhdistivät kaikkia tapauksia koeympäristöstä, kuvagenrestä, arvioijan ammattimaisuudesta riippumatta. Nämä tekijät olivat: affektiivisuus, estetiikka, käyttökelpoisuus, semantiikka ja uutuus-kompleksisuus. Kuvan toimitukselliseen kiinnostavuuteen siis vaikuttivat kuvan kyky herättää tunteita, esteettiset tekijät, monipuolinen käytettävyys, sisältöön liittyvät semanttiset tekijät, sekä ennennäkemättömyyteen ja monimutkaisuuteen liittyvät attribuutit. Katsojien näkökulmasta kuvan kiinnostavuudella oli kaksi muutakin tekijää: kuvan kyky kiinnittää ja ylläpitää katsojan huomio, sekä katsojan kokema omakohtaisuus. Nämä vahvistuivat kirjallisuuskatsauksessa, kirjallisuudesta löytyi lisäksi yksi tekijä, nk. selviytymispotentiaali, eli katsojan kyky sopeutua ja selvitä katselemastaan uudesta ja monimutkaisesta asiasta

Event Based Media Indexing

Multimedia data, being multidimensional by its nature, requires appropriate approaches for its organizing and sorting. The growing number of sensors for capturing the environmental conditions in the moment of media creation enriches data with context-awareness. This unveils enormous potential for eventcentred multimedia processing paradigm. The essence of this paradigm lies in using events as the primary means for multimedia integration, indexing and management. Events have the ability to semantically encode relationships of different informational modalities. These modalities can include, but are not limited to: time, space, involved agents and objects. As a consequence, media processing based on events facilitates information perception by humans. This, in turn, decreases the individual’s effort for annotation and organization processes. Moreover events can be used for reconstruction of missing data and for information enrichment. The spatio-temporal component of events is a key to contextual analysis. A variety of techniques have recently been presented to leverage contextual information for event-based analysis in multimedia. The content-based approach has demonstrated its weakness in the field of event analysis, especially for the event detection task. However content-based media analysis is important for object detection and recognition and can therefore play a role which is complementary to that of event-driven context recognition. The main contribution of the thesis lies in the investigation of a new eventbased paradigm for multimedia integration, indexing and management. In this dissertation we propose i) a novel model for event based multimedia representation, ii) a robust approach for mining events from multimedia and iii) exploitation of detected events for data reconstruction and knowledge enrichment

Face age estimation using wrinkle patterns

Face age estimation is a challenging problem due to the variation of craniofacial growth, skin texture, gender and race. With recent growth in face age estimation research, wrinkles received attention from a number of research, as it is generally perceived as aging feature and soft biometric for person identification. In a face image, wrinkle is a discontinuous and arbitrary line pattern that varies in different face regions and subjects. Existing wrinkle detection algorithms and wrinkle-based features are not robust for face age estimation. They are either weakly represented or not validated against the ground truth. The primary aim of this thesis is to develop a robust wrinkle detection method and construct novel wrinkle-based methods for face age estimation. First, Hybrid Hessian Filter (HHF) is proposed to segment the wrinkles using the directional gradient and a ridge-valley Gaussian kernel. Second, Hessian Line Tracking (HLT) is proposed for wrinkle detection by exploring the wrinkle connectivity of surrounding pixels using a cross-sectional profile. Experimental results showed that HLT outperforms other wrinkle detection algorithms with an accuracy of 84% and 79% on the datasets of FORERUS and FORERET while HHF achieves 77% and 49%, respectively. Third, Multi-scale Wrinkle Patterns (MWP) is proposed as a novel feature representation for face age estimation using the wrinkle location, intensity and density. Fourth, Hybrid Aging Patterns (HAP) is proposed as a hybrid pattern for face age estimation using Facial Appearance Model (FAM) and MWP. Fifth, Multi-layer Age Regression (MAR) is proposed as a hierarchical model in complementary of FAM and MWP for face age estimation. For performance assessment of age estimation, four datasets namely FGNET, MORPH, FERET and PAL with different age ranges and sample sizes are used as benchmarks. Results showed that MAR achieves the lowest Mean Absolute Error (MAE) of 3.00 ( 4.14) on FERET and HAP scores a comparable MAE of 3.02 ( 2.92) as state of the art. In conclusion, wrinkles are important features and the uniqueness of this pattern should be considered in developing a robust model for face age estimation

Organising and structuring a visual diary using visual interest point detectors

As wearable cameras become more popular, researchers are increasingly focusing on novel applications to manage the large volume of data these devices produce. One such application is the construction of a Visual Diary from an individual’s photographs. Microsoft’s SenseCam, a device designed to passively record a Visual Diary and cover a typical day of the user wearing the camera, is an example of one such device. The vast quantity of images generated by these devices means that the management and organisation of these collections is not a trivial matter. We believe wearable cameras, such as SenseCam, will become more popular in the future and the management of the volume of data generated by these devices is a key issue. Although there is a significant volume of work in the literature in the object detection and recognition and scene classification fields, there is little work in the area of setting detection. Furthermore, few authors have examined the issues involved in analysing extremely large image collections (like a Visual Diary) gathered over a long period of time. An algorithm developed for setting detection should be capable of clustering images captured at the same real world locations (e.g. in the dining room at home, in front of the computer in the office, in the park, etc.). This requires the selection and implementation of suitable methods to identify visually similar backgrounds in images using their visual features. We present a number of approaches to setting detection based on the extraction of visual interest point detectors from the images. We also analyse the performance of two of the most popular descriptors - Scale Invariant Feature Transform (SIFT) and Speeded Up Robust Features (SURF).We present an implementation of a Visual Diary application and evaluate its performance via a series of user experiments. Finally, we also outline some techniques to allow the Visual Diary to automatically detect new settings, to scale as the image collection continues to grow substantially over time, and to allow the user to generate a personalised summary of their data