In this work, we describe an interactive lifelog search engine developed for the LSC 2018 search challenge at ACM ICMR 2018. The paper introduces the four-step process required to support lifelog search engines and describes the source data for the search engine as well as the approach to ranking chosen for the iterative search engine. Finally the interface used is introduced before we highlight the limits of the current prototype and suggest opportunities for future work.Peer ReviewedPostprint (published version

Alsina, Adrià

Giró Nieto, Xavier

Gurrin, Cathal

Adrià Alsina

Xavier Giró

Cathal Gurrin

Crossref

An Interactive Lifelog Search Engine for LSC2018

Irish Universities

An interactive lifelog search engine for LSC2018

Giró-i-Nieto, Xavier

DCU Online Research Access Service

UPCommons. Portal del coneixement obert de la UPC

An Interactive Lifelog Search Engine for LSC2018Adrià Alsina & Xavier GiróUniversitat Politècnica de CatalunyaBarcelona, Spainadria.alsinatorra2@mail.dcu.ie,xavier.giro@upc.eduCathal GurrinDublin City UniversityDublin, Irelandcathal.gurrin@dcu.ieABSTRACTIn this work, we describe an interactive lifelog search engine devel-oped for the LSC 2018 search challenge at ACM ICMR 2018. Thepaper introduces the four-step process required to support lifelogsearch engines and describes the source data for the search engineas well as the approach to ranking chosen for the iterative searchengine. Finally the interface used is introduced before we highlightthe limits of the current prototype and suggest opportunities forfuture work.CCS CONCEPTS• Information systems→Digital libraries and archives; Searchinterfaces;KEYWORDSLifelogging, Interactive Search Engine, Information RetrievalACM Reference Format:Adrià Alsina & Xavier Giró and Cathal Gurrin. 2018. An Interactive LifelogSearch Engine for LSC2018. In LSC’18: The Lifelog Search Challenge, June 11,2018, Yokohama, Japan. ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/3210539.32105461 INTRODUCTIONLifelogging is a new concept and has only recently attracted theattention of the research community [7]. One standard definitionof lifelogging states that it is "the action or practice of making a con-tinuous record of one’s daily activities by means of a digital device orcomputer application". The growing research interest in lifelogginghas been facilitated by the market-availability of a range of lifelog-ging and quantified self devices that can digitally capture a widerange of life activities, from wearable cameras for all-of-lifeloggingto themore targeted health andwellness devices commonly referredto as quantified self devices [9].With such a range of lifelogging devices, one can easily gather, inan automated manner, a wide range of data about the life experienceof the individual, such as image data from smartphones or wearablecameras, audio recordings, location data, biometric data, to nameonly a few.Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.LSC’18, June 11, 2018, Yokohama, Japan© 2018 Association for Computing Machinery.ACM ISBN 978-1-4503-5796-8/18/06. . . $15.00https://doi.org/10.1145/3210539.3210546The increase of interest in lifelogging has resulted in many differ-ent research challenges being developed, such as the NTCIR lifelogtask Semantic Access Task (LSAT) of the the NTCIR-12 challenge[6] and the ImageCLEF [8] lifelog task. This LSC lifelog task posedunique challenges due to the interactive nature of the task, hencewe felt that our retrieval approaches would be suitable for this task.The interactive nature of the challenge means that any system de-veloped for the LSC would need to be optimised to select or locaterelevant content from a comparatively small archive of lifelog data,which in this case, consisted of multimedia data from wearablecameras, biometric data from smartwatches, human activity datafrom smartphones and computer usage data to identify informationaccess activities.Without there having been similar tasks in the past, there are notmany interactive retrieval engines that were developed for lifelogdata previously. The seminal MyLifeBits [5] project at Microsoftalso developed a lifelog retrieval engine based on an underlyingdatabase system, which is generally regarded as the first lifelogretrieval system. The LEMoRe [2], an interactive lifelog retrieval en-gine, developed in the context of the Lifelog Semantic Access Task(LSAT) of the the NTCIR-12 challenge [6]. LEMoRe integrated clas-sical image descriptors with high-level semantic concepts extractedby Convolutional Neural Networks [13], powered by a graphicaluser interface that uses natural language processing to process auser’s query. Finally, Doherty et al. [4], developed an interactiveevent-driven lifelog browser for visual lifelog data that segmenteddays into events, based on analysis of visual and sensor data, organ-ising and linking events together in a single diary-style interface.Extending this work, an interactive faceted lifelog search engine[3] was developed that allowed the user to select a number of con-textual factors in order to find an event of interest from an archivethat spanned a number of years.Building on experiences from this past work, we have developedthe experimental search engine which forms the main contributionof this paper. The general approach taken for this prototype is todevelop an indexing and filtering tool, utilising an SQL database [5],with a customised interface that is designed to support fast free-textand faceted search (faceted search based on [3]). The differencebetween this search engine and the preceding efforts is that freetext search, different facets, and the interface presentation thatconsiders the temporal context of the lifelogger when presentingranked or filtered results.2 A FOUR-STEP APPROACH TOINTERACTIVE LIFELOG RETRIEVALIn designing our lifelog retrieval engine, we separate the task intofour challenges:Oral Session LSC’18, June 11, 2018, Yokohama, Japan30• Data Selection and Enrichment. Lifelogging typically in-volves multiple sensors, as did the LSC dataset. The firsttask is to choose, process and align these data files so thatthey are temporally arranged.• Multi-faceted query interface. The multi-modal nature ofthe LSC dataset naturally lends itself to multi-faceted querygeneration, so we have develop a query mechanism thatallows a user to enter queries based on free-text or facets,such as date/time and place.• Ranking Engine. For retrieving potentially relevant contentfor the user using an appropriate ranking technique.• Presentation of a Result List. Given the temporal nature oflifelog data, it is sensible to consider temporal organisationof the result list, since the ranking employed will hugelyinfluence how effective the interactive retrieval system canbe.Wewill now explain howwe solved each of these four challenges.2.1 Data Selection and EnrichmentThe LSC lifelog dataset consists of a wide array of multimodaldata of an individual’s life experience captured via wearable andsoftware sensors. The dataset was gathered over a period of 27 daysby a single individual and it includes:• Multimedia data: Represents about 1,500 images per daytaken by a wearable camera. So, if we have 27 days of data itmeans that we have a database of images about 40,500 images.These images are accompanied by the output of a conceptdetector which identifies the dominant visual concepts inevery image. In addition, the music listening history of thelifelogger is included, though we did not make use of it inthis prototype.• Biometric data: It is formed for biometric data like heartrate, galvanic skin response, calorie burn and steps storedalmost every minute of the day. Also we have data aboutblood pressure and blood sugar daily. In this search engine,we utilised the heart rate data and segmented it into threecategories (resting, normal, and physically active).• Human activity data: Consists in information about the se-mantic locations visited, the physical activities, the dailymood and a diet log made of manual logging of photos. Wefocused on the semantic locations and physical activities inthe search engine.• Computer Usage: Represents a vector of every minute storeddata filtered using blacklist, anonymised and then stemmed.We did not use this data in this work.In addition, each data item described above is timestamped andthis was used to time-align the data for indexing. We also utilisedthis time information to create a semantic time annotation for everymoment in the collection, such as day-of-the-week or time-of-the-day (morning, afternoon, evening).The image formed the main unit of retrieval for this system andevery image was annotated with the appropriate enriched semanticcontent from the dataset. These images were later selected fordisplay to the user during the interactive retrieval process.2.2 Multi-faceted query interfaceGiven the interactive nature of the LSC exercise, the design of theinterface, both for rapid querying and efficient browsing of resultsets becomes the most important task. Borrowing from the standardWWW-interface for faceted search systems (e.g. hotel booking orfight booking), we designed the interface with two sections, as inFigure 1. On the left side the query panel is displayed which containsthe faceted and free-text query elements. On the right side is theresult display panel.Given the fact that the number of unique annotated terms inthe collection could be very large (especially in terms of visualconcepts and locations), it was decided to replace the typical drop-down list with a free-text search box on the query panel. Thissearch box matched against content in any of the underlying datasources and is hoped to simplify the use of the system for the user.Given a limited lexicon of available search terms, the search boxoffered auto-completion of input terms, which assists the user informulating free text queries. In additional to the free-text search,somemore conventional faceted search filter options were available:• Day-of-the-Week selector to filter content by the day theevent represented in the topic occurred.• Calendar selector to choose and exact date.• Moment-of-the-Day selector to filter content that occurs atcertain times of the day.• Place selector to allow filtering into indoor/outdoor events.• Heart-rate selector to choose the various degrees of physicalactivity of the lifelogger.Figure 1: Display of the images.In terms of technologies the client interface is developed usingthe JavaScript library React.js. The server-side data storage is han-dled by SQL Server and communication is via the python libraryFlask. The technologies are chosen so as to optimise the speedof handing every faceted query change and the system has beendesigned to be responsive to any facet updates.2.3 Ranking EngineThe ranking engine used in this interactive lifelog search tool inte-grates TF-IDF ranking for the free-text search as well as the facetedOral Session LSC’18, June 11, 2018, Yokohama, Japan31filtering mechanism. TF-IDF is a statistical approach to informationretrieval that is intended to reflect how important a word is in ourdatabase, so the words that appear less in our database have a higherscore than words that appear more often. The free text search im-plements standard enhancements, such as stopword removal andterm stemming for the English language. Since the annotations forthe LSC dataset are generated automatically by content analysistools, the natural variability in human text does not occur in theLSC collection. Hence, concepts such as term frequency weightingof terms are unlikely to play any significant role in the rankingprocess. However, the inverse document frequency concept frominformation retrieval is important in that it allows for the higherranking of concepts and our use of TF-IDF ranking includes theinverse document frequency weighing in the IDF component.This ranked list from the free-text search is filtered by the otherdata facets, such as time of day, day of week, or location, as iscommonly employed for interactive retrieval systems [14]. Theresult is a ranked list of filtered images for presentation to the user.One enhancement to the ranked list is that the context of theimage is also considered in the scoring process. The previous threeimages and the following three images contribute (on a sliding scale)to the overall score of the main image. This is based on previoussuccessful deployments in interactive video retrieval systems [1].2.4 Result List PresentationUpon submission of a query, the system generates a ranked list ofimages. The images are displayed as you can see in the Figure 1 onthe result-display panel to the right of the query panel. Rather thansimply display a ranked list of images, the system aims to providesome context to the user by integrating two enhancements:• Providing context to the main image by highlighting meta-data regarding the image, such as temporal, location andactivity data.• Highlighting the visual context of the image by borrowingfrom concepts previously used in interactive video retrievalsystems [1], the visual context of each ranked image is dis-played by the highlighting of 1 or 2 (1 shown in Figure 1)previously occurring and following images. Given the pas-sive nature of lifelog data capture, a naive approach wouldbe to simply select the previous occurring image(s) to beshown on screen. However, lifelog data tends to be veryrepetitive, so a judicial approach is used which only showsthe previews and following images if they are sufficiently vi-sually different from the ranked image in terms of the visualcontent and the concept annotations. We include this func-tionality because the development topics contained sometemporal topics that require knowledge about the precedingor succeeding activities of the lifelogger.Upon finding a potentially relevant image for a topic, the userselects an image which submits it to the LSC server for validation.3 CONCLUSIONIn this paper, we describe the data and the task from the LSC 2018lifelog search challenge. We describe the prototype interactivesearch engine that we built for the challenge and we presentedthe interface and extensions to the basic search engine.We have selected a number of facets for the query interface (asdescribed above) and we are limited by the nature of the LSC col-lection, as well as the available screen estate for the query-entryprocess. Knowing that integrating more facets are likely to increaseperformance [10], future work will seek to enhance the types andrichness of the facets available to the user. In addition, the imple-mentation of free-text search in this first generation lifelog searchengine does not include many of the standard text retrieval en-hancements such as relevance feedback [12] or optimised rankingalgorithms (e.g. BM25) [11]. For future work, consideration shouldbe given to optimising the system components.ACKNOWLEDGMENTSThe authors acknowledge the financial support of Science Founda-tion Ireland (SFI) under grant number SFI/12/RC/2289.REFERENCES[1] Eddie Cooke, Paul Ferguson, Georgina Gaughan, Cathal Gurrin, Gareth J.F. Jones,Herve Le Borgne, Hyowon Lee, Sean Marlow, Kieran McDonald, Mike McHugh,Noel Murply, Noel E. O Connor, Neil O Hare, Sandra Rothwell, Peter Wilkins, andAlan F. Smeaton. 2004. TRECVID 2004 experiments in Dublin City University.TRECVID 2004 experiments in Dublin City University (2004). arXiv:1404.7828http://arxiv.org/abs/1404.7828[2] Gabriel de Oliveira-Barra, X. Giró-i Nieto, Alejandro Cartas-Ayala, and PetiaRadeva. 2016. LEMoRe: A Lifelog Engine for Moments Retrieval at the NTCIR-Lifelog LSAT Task. In The 12th NTCIR Conference, Evaluation of InformationAccess Technologies. National Institute of Informatics (NII), National Instituteof Informatics (NII), Tokyo, Japan. http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings12/NTCIR/toc_ntcir.html$#$Lifelog[3] Aiden R. Doherty, Kata Pauly-Takacs Niamh Caprani, Cathal Gurrin, ChrisMoulin, Noel O Connor, and Alan F. Smeaton. 2012. Experiences of AidingAutobiographical Memory Using the SenseCam. Human-Computer Interaction27, 1-2 (2012), 151–174.[4] Aidan R. Doherty and Alan F. Smeaton. 2008. Automatically segmenting lifelogdata into events. In Ninth International IEEE Workshop on Image Analysis forMultimedia Interactive Services. IEEE, 20–23.[5] Jim Gemmell, Gordon Bell, and Roger Lueder. 2006. MyLifeBits: a personaldatabase for everything. Commun. ACM 49, 1 (2006), 88–95.[6] Cathal Gurrin, Hideo Joho, Frank Hopfgartner, Liting Zhou, and Rami Albatal.2016. Overview of NTCIR-12 Lifelog Task. , 354–360 pages. http://eprints.gla.ac.uk/131460/[7] Cathal Gurrin, Alan F. Smeaton, and Aiden R. Doherty. 2014. LifeLogging:Personal Big Data. Foundations and Trends in Information Retrieval 8, 1 (2014),1–125. https://doi.org/10.1561/1500000033[8] Bogdan Ionescu, HenningMüller, Mauricio Villegas, Helbert Arenas, Giulia Boato,Duc-Tien Dang-Nguyen, Yashin Dicente Cid, Carsten Eickhoff, Alba G. Seco deHerrera, Cathal Gurrin, Bayzidul Islam, Vassili Kovalev, Vitali Liauchuk, JosianeMothe, Luca Piras, Michael Riegler, and Immanuel Schwall. 2017. Overview ofImageCLEF 2017: Information Extraction from Images. In Experimental IR MeetsMultilinguality, Multimodality, and Interaction, Gareth J.F. Jones, Séamus Lawless,Julio Gonzalo, Liadh Kelly, Lorraine Goeuriot, Thomas Mandl, Linda Cappellato,and Nicola Ferro (Eds.). Springer International Publishing, Cham, 315–337.[9] Jochen Meyer, Steven Simske, Katie Siek, Cathal Gurrin, and Hermie J Hermens.2014. Beyond quantified self: data for wellbeing. In Extended Abstracts on HumanFactors in Computing Systems. http://dl.acm.org/citation.cfm?id=2560469[10] Mor Naaman, Susumu Harada, QianYing Wang, Hector Garcia-Molina, andAndreas Paepcke. 2004. Context data in geo-referenced digital photo collections.In ACM Multimedia.[11] Stephen E. Robertson and Karen Sparck Jones. 1994. Simple, proven approachesto text retrieval. University of Cambridge Technical Report (1994). https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-356.pdf[12] J. J. Rocchio. 1971. Relevance Feedback in Information Retrieval. Prentice Hall,Englewood, Cliffs, New Jersey. http://www.is.informatik.uni-duisburg.de/bib/docs/Rocchio_71.html[13] Jürgen Schmidhuber. 2014. Deep Learning in Neural Networks: An Overview.CoRR abs/1404.7828 (2014). arXiv:1404.7828 http://arxiv.org/abs/1404.7828[14] Daniel Tunkelang. 2009. Faceted Search. Synthesis Lectureson Information Concepts, Retrieval, and Services 1, 1 (2009),1–80. https://doi.org/10.2200/S00190ED1V01Y200904ICR005arXiv:https://doi.org/10.2200/S00190ED1V01Y200904ICR005Oral Session LSC’18, June 11, 2018, Yokohama, Japan32

http://doras.dcu.ie/25863/1/ICMR_LSC_2018_Adria.pdf

An interactive lifelog search engine for LSC2018

Abstract

Similar works

Full text

Available Versions

Crossref

Irish Universities

DCU Online Research Access Service

UPCommons. Portal del coneixement obert de la UPC