The SenseCam is a wearable camera that passively captures images. Therefore, it requires no conscious eﬀort by a user in taking a photo. A Visual Diary from such a source could prove to be a valuable tool in assisting the elderly, individuals with neurodegenerative diseases, or other traumas. One issue with Visual Lifelogs is the large volume of image data generated. In previous work we spit a day's worth of images into more manageable segments, i.e. into distinct events or activities. However, each event coud stil consist of 80-100 images. thus, in this paper we propose a novel approach to selecting the key images within an event using a combination of MPEG-7 and Scale Invariant Feature Transform (SIFT) features

Blighe, Michael

Doherty, Aiden R.

O'Connor, Noel E.

Smeaton, Alan F.

English

Crossref

Keyframe detection in visual lifelogs

The SenseCam is a wearable camera that passively captures images. Therefore, it requires no conscious eﬀort by a user in taking a photo. A Visual Diary from such a source could prove to be a valuable tool in assisting the elderly, individuals with neurodegenerative diseases, or other traumas. One issue with Visual Lifelogs is the large volume of image data generated. In previous work we spit a day\u27s worth of images into more manageable segments, i.e. into distinct events or activities. However, each event coud stil consist of 80-100 images. thus, in this paper we propose a novel approach to selecting the key images within an event using a combination of MPEG-7 and Scale Invariant Feature Transform (SIFT) features

O\u27Connor, Noel E.

Irish Universities

Name not available

Keyframe Detection in Visual LifelogsMichael Blighe, Aiden Doherty, Alan F. Smeaton and Noel E. O’ConnorCentre for Digital Video Processing, Adaptive Information ClusterDublin City University, Ireland{blighem, oconnorn}@eeng.dcu.ie, {adoherty, alan.smeaton}@computing.dcu.ieABSTRACTThe SenseCam is a wearable camera that passively capturesimages. Therefore, it requires no conscious effort by a userin taking a photo. A Visual Diary from such a source couldprove to be a valuable tool in assisting the elderly, individu-als with neurodegenerative diseases, or other traumas. Oneissue with Visal Lifelogs is the large volume of image datagenerated. In previous work, we split a day’s worth of imagesinto more manageable segments, i.e. into distinct events oractivities. However, each event could still consist of 80-100images, thus, in this paper we propose a novel approach toselecting the key images within an event using a combinationof MPEG-7 and Scale Invariant Feature Transform (SIFT)features.KeywordsVisual Diary, Health Management, Keyframe Selection1. INTRODUCTIONAn initial report from Microsoft has demonstrated howthe SenseCam can be used in research to assist people withshort term memory loss [3]. The most widespread neurode-generative diseases are Alzheimer’s and Parkinson’s. Alzheimer’sdisease is an irreversible neurodegenerative disorder thatprogressively degrades the brains ability to maintain normalexecutive, attention, and memory functions. A treatmentthat could delay the onset of Alzheimer’s by 5 years wouldreduce the number of sufferers by 50% in 50 years.Besides assisting with neurodegenerative diseases, there isa growing belief that technology can be used to address theproblem of in-home care for the elderly. We believe that theuse of a Visual Diary could lead to significant improvementsin the health and quality of life of elderly people withintheir own homes. We propose the use of passive capturedevices, such as the SenseCam, to assist in this area, wherea user wears the camera around their neck and the cameratakes pictures continuously throughout the day. However,Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.Copyright 200X ACM X-XXXXX-XX-X/XX/XX ...$5.00.the management of the larger volume of image data gen-erated by such devices remains a challenging problem. Inprevious work, we successfully split each day’s worth of im-ages into distinct events or activities [1, 2].2. RELATED RESEARCHIn health management, an analysis of behavioural factorsplays a critical role. For example, previous research hasintroduced photography into diabetes self-management rou-tines to help patients make their behaviours explicit and towork with physicians to see possible correlations between selfmedication and long-term health [5]. Past approaches havemanually collected images, however, using SenseCam thisprocess could be automated, giving a better understandingof how to improve the diagnoses and treatment of illnessesthat are highly influenced by behavioural routines. In ad-dition, many home monitoring technologies have been pro-posed to detect health crises, support aging-in-place, andimprove medical care [4]. The potential costs, and fears overbreaches of privacy amongst health professionals and mem-bers of the public, mean that these technologies have hada limited impact to date. However, there is some evidencethat these systems may be more readily adopted if they aredeveloped as tools for personalised use, thus helping userslearn about the conditions and variables which affect theirphysical health.One of the issues associated with visual recording of alifelog is selecting single images, or keyframes, which ap-propriately represent real life events. Techniques to addressthis will require the use of image processing algorithms suchas SIFT. The SIFT descriptor is a gradient orientation his-togram robust to illumination and viewpoint changes. In[1], SIFT descriptors are used to detect important settingsin SenseCam images. In [2], MPEG-7 features were used inconjuction with sensor data to structure collections of Sense-Cam images into events. Given that each event typicallyconsists of up to 80-100 images, in this paper we introducea novel concept for selecting a representative image throughthe use of SIFT features.3. APPROACH AND PRELIMINARY RESULTSIn this investigation, we examined examples of three typi-cal events taken from the collections of two SenseCam users.Details of how images are segmented into these distinctevents can be found in [2]. The three scenarios investigatedare known as Static Scene, Random Scene and Return Scene.A Static Scene is one where the user is relatively stationarywhilst wearing the SenseCam. An example of such a sceneis when the user is sitting watching TV for an extended pe-riod of time. There may be some small movements to theleft or right, but essentially the scene remains the same.A Random Scene occurs where the user is walking aroundwearing the camera. All the images taken are of randomobjects and places, depending on where the user is going.A Return Scene is one where the user is in one location,moves temporarily to somewhere else, but then returns tothe original location. An example could be someone walk-ing around their kitchen, looking in a cupboard or fridge,walking away, but then returning to the original location foranother look. This process may be repeated several times ina typical kitchen activity like preparing a meal.Given the low quality of SenseCam images [1], we de-termine the quality of each SenseCam image based on itscontrast and salience properties [2]. Given an image qual-ity score for each image we then apply the Kapur adaptivethresholding technique to select a subset of images from theevent that are of a sufficiently high image quality.Once this process has been completed, SIFT features areextracted from the remaining images in each event. In orderto match features between images, the distance ratio testwas used [1]. To examine whether a point from the 1st imagehas a match in the 2nd, it’s two most similar descriptors inthe 2nd image are found. If the ratio of the nearest distanceto the second nearest distance is less than 0.7, a match isdeclared. The number of matches between an image and allother images in the event are summed, and then the averagenumber of matches is calculated. The image which has thehighest average is deemed to be the most similar to all otherimages in the event and, hence, is selected as the keyframefor that event.In order to evaluate this approach, results were comparedto the more traditional approach of selecting the middle im-ages as the keyframe in SenseCam events (see Table 1). Foreach of the three types of scene, two users selected five ex-amples of each scene from their collections. This gave a totalof 30 different scenes, consisting of 3,179 images. After re-moval of low-quality images, the SIFT keypoints from 2,178images were analysed from which to select a keyframe im-age for each event. For this preliminary work, the resultswere qualitatively analysed by both users. For the StaticScene, our approach selects images captured by taking theentire scene into consideration (as opposed to, for exam-ple, an image showing just a small portion of a computerscreen). There is no one definitive image that should be akeyframe image in the Random Scene. In general though,for images from the Random Scene or Static Scene, therewas little difference in the performance of both techniques.Generally, the quality or semantic meaning of the selectedkeyframe influenced which approach was deemed most suc-cessful. However, the Return Scene did produce a discrep-ancy between our approach and simply selecting the middleimage from an event. An example from one scene is shownin Figures 1(a) & 1(b). Both images were taken in a similarlocation, however, the image selected using our approach issemantically more meaningful to the user than that selectedusing the middle keyframe. Both images show the user onhis regular walk along the river side, however, Figure 1(b)shows detail of a particular fishing event which occurred onthe river. Figure 1(a) could have been taken on any dayas there is nothing in the image to tie it to this particularevent.Approach Static Scene Random Scene Return SceneMiddle Keyframe 0 1 1New Approach 1 3 5Table 1: This table show’s the number of event’swhich were judged to be better using the two ap-proaches for each type of scene. There were 30scene’s in total; 10 for each type.(a) (b)Figure 1: (a) Middle image selected; (b) Image se-lected using our novel approach4. CONCLUSIONSNumerous strategies exist for selecting representative im-ages from a collection and we intend to perform a muchmore detailed set of experiments to compare our approachto other techniques. In addition, we also intend to explorethe benefits this technology may offer in improving the qual-ity of life of those with memory difficulties or those requiringin-home care.AcknowledgmentsThe research leading to this paper was supported by theIrish Research Concil for Science, Engineering, and Technol-ogy; Microsoft Research under grant 2007-056; and ScienceFoundation Ireland under grant number 03/IN.3/I361.5. REFERENCES[1] M. Blighe, N. O’Connor, H. Rehatschek, andG. Kienast. Identifying different settings in a visualdiary. In 9th International Workshop on ImageAnalysis for Multimedia Interactive Services, May 2008.[2] A. Doherty, D. Byrne, A. Smeaton, G. Jones, andM. Hughes. Investigating keyframe selection methods inthe novel domain of passively captured visual lifelogs.In ACM International Conference on Image and VideoRetrieval, Niagara Falls, Canada, July 2008.[3] S. Hodges, L. Williams, E. Berry, S. Izadi,J. Srinivasan, A. Butler, G. Smyth, N. Kapur, andK. Wood. Sensecam: A retrospective memory aid. InEighth International Conference on UbiquitousComputing, September 2006.[4] S. Intille, K. Larson, J. Beaudin, E. M. Tapia,P. Kaushik, J. Nawyn, and T. McLeish. The placelab: alive-in laboratory for pervasive computing research. InPervasive 2005 Video Program, May 2005.[5] B. Smith, J. Frost, and M. A. R. Sudhakar. Improvingdiabetes self-management with glucometers and digitalphotography. Personal and Ubiquitous Computing,2006.

DCU Online Research Access Service

Identifying diﬀerent settings in a visual diary.

Improving diabetes self-management with glucometers and digital photography. Personal and Ubiquitous Computing,

Investigating keyframe selection methods in the novel domain of passively captured visual lifelogs.

Sensecam: A retrospective memory aid.

The placelab: a live-in laboratory for pervasive computing research. In Pervasive

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.148.2429

Keyframe detection in visual lifelogs

Abstract

Similar works

Full text

Available Versions

Crossref

Irish Universities

Name not available

DCU Online Research Access Service