Towards activity recommendation from lifelogs

With the increasing availability of passive, wearable sensor devices, digital lifelogs can now be captured for individuals. Lifelogs contain a digital trace of a person’s life, and are characterised by large quantities of rich contextual data. In this paper, we propose a content-based recommender sys- tem to leverage such lifelogs to suggest activities to users. We model lifelogs as timelines of chronological sequences of activity objects, and describe a recommendation framework in which a two-level distance metric is proposed to measure the similarity between current and past timelines. An ini- tial evaluation of our activity recommender performed using a real-world lifelog dataset demonstrates the utility of our approach

Detecting Physical Activity within Lifelogs towards Preventing Obesity and Aid Ambient Assisted Living

Obesity is a global health issue that affects 2.1 billion people worldwide and has an economic impact of approximately $2 trillion. It is a disease that can make the aging process worse by impairing physical function, which can lead to people becoming more frail and immobile. Nevertheless, it is envisioned that technology can be used to aid in motivating behavioural changes to combat this preventable condition. The ubiquitous presence of wearable and mobile devices has enabled a continual stream of quantifiable data (e.g. physiological signals) to be collected about ourselves. This data can then be used to monitor physical activity to aid in self-reflection and motivation to alter behaviour. However, such information is susceptible to noise interference, which makes processing and extracting knowledge from such data challenging. This paper posits our approach that collects and processes physiological data that has been collected from tri-axial accelerometers and a heart-rate monitor, to detect physical activity. Furthermore, an end-user use case application has also been proposed that integrates these findings into a smartwatch visualisation. This provides a method of visualising the results to the user so that they are able to gain an overview of their activity. The goal of the paper has been to evaluate the performance of supervised machine learning in distinguishing physical activity. This has been achieved by (i) focusing on wearable sensors to collect data and using our methodology to process this raw lifelogging data so that features can be extracted/selected. (ii) Undertaking an evaluation between ten supervised learning classifiers to determine their accuracy in detecting human activity. To demonstrate the effectiveness of our method, this evaluation has been performed across a baseline method and two other methods. (iii) Undertaking an evaluation of the processing time of the approach and the smartwatch battery and network cost analysis between transferring data from the smartwatch to the phone. The results of the classifier evaluations indicate that our approach shows an improvement on existing studies, with accuracies of up to 99% and sensitivities of 100%

Communicating with your E-memory: finding and refinding in personal lifelogs

The rapid development of technology enables the digital capture and storage of our life experiences in an “E-Memory” (electronic–memory) or personal lifelog (PLL). This offers the potential for people to store the details of their life in a permanent archive, so that the information is still available even when its physical existence has vanished and when memory traces of it have faded away. A major challenge for PLLs is enabling people to access information when it is needed. Many people may also want to share or transfer some of their memory to their friends and descendants, so that their experiences can be appreciated and their knowledge can be kept even after they have passed away. This thesis further explores people’s potential needs from their own PLLs, discuss the possible methods people may use and potential problems that they may encounter while accessing their PLLs, and hypothesize that better support of users’ own memory can provide better user experience and improved efficiency for accessing their E-memories (or PLLs). As part of a larger project, three lifeloggers collected their own prototype lifelog collection for about 20 months’ time. To complete this study, the author developed a prototype PLL system, called the iCLIPS Lifelog Archive Browser (LAB), based on the author’s theoretical exploration and empirical studies, and evaluated it using our prototype lifelog collections through a user study with the three lifeloggers. The results of this study provide promising evidence which support the hypothesis. The end of this thesis also discusses the issues that the lifeloggers encountered in using their lifelogs and future technologies that are desirable based the studies in this thesis

The role of context in image annotation and recommendation

With the rise of smart phones, lifelogging devices (e.g. Google Glass) and popularity of image sharing websites (e.g. Flickr), users are capturing and sharing every aspect of their life online producing a wealth of visual content. Of these uploaded images, the majority are poorly annotated or exist in complete semantic isolation making the process of building retrieval systems difficult as one must firstly understand the meaning of an image in order to retrieve it. To alleviate this problem, many image sharing websites offer manual annotation tools which allow the user to “tag” their photos, however, these techniques are laborious and as a result have been poorly adopted; Sigurbjörnsson and van Zwol (2008) showed that 64% of images uploaded to Flickr are annotated with < 4 tags. Due to this, an entire body of research has focused on the automatic annotation of images (Hanbury, 2008; Smeulders et al., 2000; Zhang et al., 2012a) where one attempts to bridge the semantic gap between an image’s appearance and meaning e.g. the objects present. Despite two decades of research the semantic gap still largely exists and as a result automatic annotation models often offer unsatisfactory performance for industrial implementation. Further, these techniques can only annotate what they see, thus ignoring the “bigger picture” surrounding an image (e.g. its location, the event, the people present etc). Much work has therefore focused on building photo tag recommendation (PTR) methods which aid the user in the annotation process by suggesting tags related to those already present. These works have mainly focused on computing relationships between tags based on historical images e.g. that NY and timessquare co-exist in many images and are therefore highly correlated. However, tags are inherently noisy, sparse and ill-defined often resulting in poor PTR accuracy e.g. does NY refer to New York or New Year? This thesis proposes the exploitation of an image’s context which, unlike textual evidences, is always present, in order to alleviate this ambiguity in the tag recommendation process. Specifically we exploit the “what, who, where, when and how” of the image capture process in order to complement textual evidences in various photo tag recommendation and retrieval scenarios. In part II, we combine text, content-based (e.g. # of faces present) and contextual (e.g. day-of-the-week taken) signals for tag recommendation purposes, achieving up to a 75% improvement to precision@5 in comparison to a text-only TF-IDF baseline. We then consider external knowledge sources (i.e. Wikipedia & Twitter) as an alternative to (slower moving) Flickr in order to build recommendation models on, showing that similar accuracy could be achieved on these faster moving, yet entirely textual, datasets. In part II, we also highlight the merits of diversifying tag recommendation lists before discussing at length various problems with existing automatic image annotation and photo tag recommendation evaluation collections. In part III, we propose three new image retrieval scenarios, namely “visual event summarisation”, “image popularity prediction” and “lifelog summarisation”. In the first scenario, we attempt to produce a rank of relevant and diverse images for various news events by (i) removing irrelevant images such memes and visual duplicates (ii) before semantically clustering images based on the tweets in which they were originally posted. Using this approach, we were able to achieve over 50% precision for images in the top 5 ranks. In the second retrieval scenario, we show that by combining contextual and content-based features from images, we are able to predict if it will become “popular” (or not) with 74% accuracy, using an SVM classifier. Finally, in chapter 9 we employ blur detection and perceptual-hash clustering in order to remove noisy images from lifelogs, before combining visual and geo-temporal signals in order to capture a user’s “key moments” within their day. We believe that the results of this thesis show an important step towards building effective image retrieval models when there lacks sufficient textual content (i.e. a cold start)

Ethics of lifelog technology

In a lifelog, data from different digital sources are combined and processed to form a unified multimedia archive containing information about the quotidian activities of an individual. This dissertation aims to contribute to a responsible development of lifelog technology used by members of the general public for private reasons. Lifelog technology can benefit, but also harm lifeloggers and their social environment. The guiding idea behind this dissertation is that if the ethical challenges can be met and the opportunities realised, the conditions will be optimised for a responsible development and application of the technology. To achieve this, it is important to reflect on these concerns at an early stage of development before the existing rudimentary forms of lifelogs develop into more sophisticated devices with a broad societal application. For this research, a normative framework based on prima facie principles is used. Lifelog technology in its current form is a relatively novel invention and a consensus about its definition is still missing. Therefore the author aims to clarify the characteristics of lifelog technology. Next, the ethical challenges and opportunities of lifelogs are analysed, as they have been discussed in the scholarly literature on the ethics of lifelog technology. Against this backdrop, ethical challenges and opportunities are identified and elaborated. The normative analysis concentrates on two areas of concern, namely (1) the ethical challenges and opportunities that result from the use of lifelog technology, and (2) the conditions under which one becomes a lifelogger. For the first, three sets of key issues are discussed, namely issues to do with (a) privacy, (b) autonomy, and (c) beneficence. For the second, one key set of issues is examined, namely issues to do with autonomy. The discussion of each set of issues is concluded with recommendations designed to tackle the challenges and realise the opportunities

The role of context in human memory augmentation

Technology has always had a direct impact on what humans remember. In the era of smartphones and wearable devices, people easily capture on a daily basis information and videos, which can help them remember past experiences and attained knowledge, or simply evoke memories for reminiscing. The increasing use of such ubiquitous devices and technologies produces a sheer volume of pictures and videos that, in combination with additional contextual information, could potentially significantly improve one’s ability to recall a past experience and prior knowledge. Calendar entries, application use logs, social media posts, and activity logs comprise only a few examples of such potentially memory-supportive additional information. This work explores how such memory-supportive information can be collected, filtered, and eventually utilized, for generating memory cues, fragments of past experience or prior knowledge, purposed for triggering one’s memory recall. In this thesis, we showcase how we leverage modern ubiquitous technologies as a vessel for transferring established psychological methods from the lab into the real world, for significantly and measurably augmenting human memory recall in a diverse set of often challenging contexts. We combine experimental evidence garnered from numerous field and lab studies, with knowledge amassed from an extensive literature review, for substantially informing the design and development of future pervasive memory augmentation systems. Ultimately, this work contributes to the fundamental understanding of human memory and how today’s modern technologies can be actuated for augmenting it

Context-sensitive memory augmentation using recorded everyday life data

The recent rise of life-logging technologies and wearable computing gadgets allows the recording of data from our daily lives. Experiences make people what they are. The omnipresent tracking devices and their sensors experience the same things as their owners, thus creating e-memories and surrogate brains. Such life-logs or e-memories contain everything we can sense or our environment senses, like images, heart rates or locations. With this increase of digital personal data we explore challenges and solutions how to use this vast amount of data with the goal to support human memory. To do this, we used a user-centered approach. In the first step we conducted a series of focus groups and an online survey with the goal of understanding the requirements of life-logging tools. The results of the requirement analysis led to the development of a holistic concept of a digital life assistant. Our initial prototype leverages life-log data in form of a smart alarm clock, which provides an automatic morning briefing about the past and the upcoming day via audio and bedside projection. The prototype was finally evaluated in the field in a small-scale pilot study with the focus on the different presentation modes.Die aktuelle Entwicklung von Life-Logging-Technologien und tragbaren Computern ermöglicht die Aufzeichnung von Daten aus dem täglichen Leben. Erfahrungen machen Menschen zu dem was sie sind. Die allgegenwärtigen Aufnahmegeräte erleben dasselbe, wie ihre Besitzer und schaffen damit elektronische Erinnerungen und einen stellvertretenden Verstand. Diese Life-Logs oder elektronischen Erinnerungen beinhalten alles was deren Besitzer oder deren Umgebungen wahrnehmen, wie z. B. Bilder, Herzfrequenzen oder Standorte. Mit diesem Anstieg von digitalen persönlichen Daten erforschen wir Herausforderungen und Lösungen, wie diese gewaltige Datenmenge nutzbar gemacht und das menschliche Gedächtnis unterstützt werden kann. Daher haben wir einen nutzerorientierten Ansatz gewählt. Im ersten Schritt haben wir eine Serie von Fokusgruppen und eine Online-Umfrage durchgeführt, um die Anforderungen von Life-Logging Werkzeugen zu verstehen. Das Ergebnis der Anforderungsanalyse führte zu der Entwicklung eines ganzheitlichen Konzepts eines digitalen persönlichen Assistentens. Unser initialer Prototyp macht sich Life-Logging-Daten in Form eines intelligenten Weckers zu Nutze. Der Assistent bereitet automatisiert ein morgendliches Briefing über die Vergangenheit und den bevorstehenden Tag vor und präsentiert dieses mittels Sprache und einer bettseitigen Projektion. Schließlich wurde der Prototyp im praktischen Einsatz in einer kleinen Pilotstudie mit dem Fokus auf die verschiedenen Präsentationsmodi untersucht

Tracking in the wild: exploring the everyday use of physical activity trackers

As the rates of chronical diseases, such as obesity, cardiovascular disease and diabetes continue to increase, the development of tools that support people in achieving healthier habits is becoming ever more important. Personal tracking systems, such as activity trackers, have emerged as a promising class of tools to support people in managing their everyday health. However, for this promise to be fulfilled, these systems need to be well designed, not only in terms of how they implement specific behavior change techniques, but also in how they integrate into people’s daily lives and address their daily needs. My dissertations provides evidence that accounting for people’s daily practices and needs can help to design activity tracking systems that help people get more value from their tracking practices. To understand how people derive value from their activity tracking practices, I have conducted two inquiries into people’s daily uses of activity tracking systems. In a fist attempt, I led a 10-month study of the adoption of Habito, our own activity tracking mobile app. Habito logged not only users’ physical activity, but also their interactions with the app. This data was used to acquire an estimate of the adoption rate of Habito, and understanding of how adoption is affected by users’ ‘readiness’, i.e., their attitude towards behavior change. In a follow-up study, I turned to the use of video methods and direct, in-situ observations of users’ interactions to understand what motivates people to engage with these tools in their everyday life, and how the surrounding environment shapes their use. These studies revealed some of the complexities of tracking, while extending some of the underlying ideas of behavior change. Among key results: (1) people’s use of activity trackers was found to be predominantly impulsive, where they simultaneously reflect, learn and change their behaviors as they collect data; (2) people’s use of trackers is deeply entangled with their daily routines and practices, and; (3) people use of trackers often is not in line with the traditional vision of these tools as mediators of change – trackers are also commonly used to simply learn about behaviors and engage in moments of self-discovery. Examining how to design activity tracking interfaces that best support people’s different needs , my dissertation further describes an inquiry into the design space of behavioral feedback interfaces. Through a iterative process of synthesis and analysis of research on activity tracking, I devise six design qualities for creating feedback that supports people in their interactions with physical activity data. Through the development and field deployment of four concepts in a field study, I show the potential of these displays for highlighting opportunities for action and learning.À medida que a prevalência de doenças crónicas como a obesidade, doenças cardiovasculares e diabetes continua a aumentar, o desenvolvimento de ferramentas que suportam pessoas a atingir mudanças de comportamento tem-se tornado essencial. Ferramentas de monitorização de comportamentos, tais como monitores de atividade física, têm surgido com a promessa de encorajar um dia a dia mais saudável. Contudo, para que essa promessa seja cumprida, torna-se essencial que estas ferramentas sejam bem concebidas, não só na forma como implementam determinadas estratégias de mudança de comportamento, mas também na forma como são integradas no dia-a-dia das pessoas. A minha dissertação demonstra a importância de considerar as necessidades e práticas diárias dos utilizadores destas ferramentas, de forma a ajudá-las a tirar melhor proveito da sua monitorização de atividade física. De modo a entender como é que os utilizadores destas ferramentas derivam valor das suas práticas de monitorização, a minha dissertação começa por explorar as práticas diárias associadas ao uso de monitores de atividade física. A minha dissertação contribui com duas investigações ao uso diário destas ferramentas. Primeiro, é apresentada uma investigação da adoção de Habito, uma aplicação para monitorização de atividade física. Habito não só registou as instâncias de atividade física dos seus utilizadores, mas também as suas interações com a própria aplicação. Estes dados foram utilizados para adquirir uma taxa de adopção de Habito e entender como é que essa adopção é afetada pela “prontidão” dos utilizadores, i.e., a sua atitude em relação à mudança de comportamento. Num segundo estudo, recorrendo a métodos de vídeo e observações diretas e in-situ da utilização de monitores de atividade física, explorei as motivações associadas ao uso diário destas ferramentas. Estes estudos expandiram algumas das ideias subjacentes ao uso das ferramentas para mudanças de comportamento. Entre resultados principais: (1) o uso de monitores de atividade física é predominantemente impulsivo, onde pessoas refletem, aprendem e alteram os seus comportamentos à medida que recolhem dados sobe estes mesmos comportamentos; (2) o uso de monitores de atividade física está profundamente interligado com as rotinas e práticas dos seus utilizadores, e; (3) o uso de monitores de atividade física nem sempre está ligado a mudanças de comportamento – estas ferramentas também são utilizadas para divertimento e aprendizagem. A minha dissertação contribui ainda com uma exploração do design de interfaces para a monitorização de atividade física. Através de um processo iterativo de síntese e análise de literatura, seis qualidades para a criação de interfaces são derivadas. Através de um estudo de campo, a minha dissertação demonstro o potencial dessas interfaces para ajudar pessoas a aprender e gerir a sua saúde diária

Evaluating Information Retrieval and Access Tasks

This open access book summarizes the first two decades of the NII Testbeds and Community for Information access Research (NTCIR). NTCIR is a series of evaluation forums run by a global team of researchers and hosted by the National Institute of Informatics (NII), Japan. The book is unique in that it discusses not just what was done at NTCIR, but also how it was done and the impact it has achieved. For example, in some chapters the reader sees the early seeds of what eventually grew to be the search engines that provide access to content on the World Wide Web, today’s smartphones that can tailor what they show to the needs of their owners, and the smart speakers that enrich our lives at home and on the move. We also get glimpses into how new search engines can be built for mathematical formulae, or for the digital record of a lived human life. Key to the success of the NTCIR endeavor was early recognition that information access research is an empirical discipline and that evaluation therefore lay at the core of the enterprise. Evaluation is thus at the heart of each chapter in this book. They show, for example, how the recognition that some documents are more important than others has shaped thinking about evaluation design. The thirty-three contributors to this volume speak for the many hundreds of researchers from dozens of countries around the world who together shaped NTCIR as organizers and participants. This book is suitable for researchers, practitioners, and students—anyone who wants to learn about past and present evaluation efforts in information retrieval, information access, and natural language processing, as well as those who want to participate in an evaluation task or even to design and organize one