Multiple multimodal mobile devices: Lessons learned from engineering lifelog solutions

For lifelogging, or the recording of one’s life history through digital means, to be successful, a range of separate multimodal mobile devices must be employed. These include smartphones such as the N95, the Microsoft SenseCam – a wearable passive photo capture device, or wearable biometric devices. Each collects a facet of the bigger picture, through, for example, personal digital photos, mobile messages and documents access history, but unfortunately, they operate independently and unaware of each other. This creates significant challenges for the practical application of these devices, the use and integration of their data and their operation by a user. In this chapter we discuss the software engineering challenges and their implications for individuals working on integration of data from multiple ubiquitous mobile devices drawing on our experiences working with such technology over the past several years for the development of integrated personal lifelogs. The chapter serves as an engineering guide to those considering working in the domain of lifelogging and more generally to those working with multiple multimodal devices and integration of their data

MyPlaces: detecting important settings in a visual diary

We describe a novel approach to identifying specific settings in large collections of passively captured images corresponding to a visual diary. An algorithm developed for setting detection should be capable of detecting images captured at the same real world locations (e.g. in the dining room at home, in front of the computer in the office, in the park, etc.). This requires the selection and implementation of suitable methods to identify visually similar backgrounds in images using their visual features. We use a Bag of Keypoints approach. This method is based on the sampling and subsequent vector quantization of multiple image patches. The image patches are sampled and described using Scale Invariant Feature Transform (SIFT) features. We compare two different classifiers, K Nearest Neighbour and Multiclass Linear Perceptron, and present results for classifying ten different settings across one week’s worth of images. Our results demonstrate that the method produces good classification accuracy even without exploiting geometric or context based information. We also describe an early prototype of a visual diary browser that integrates the classification results

From lifelog to diary: a timeline view for memory reminiscence

As digital recording sensors and lifelogging devices become more prevalent, the suitability of lifelogging tools to act as a reminiscence supporting tool has become an important research challenge. This paper aims to describe a rst- generation memory reminiscence tool that utilises lifelog- ging sensors to record a digital diary of user activities and presents it as a narrative description of user activities. The automatically recognised daily activities are shown chronologically in the timeline view

Life editing: Third-party perspectives on lifelog content

Lifelog collections digitally capture and preserve personal experiences and can be mined to reveal insights and understandings of individual significance. These rich data sources also offer opportunities for learning and discovery by motivated third parties. We employ a custom-designed storytelling application in constructing meaningful lifelog summaries from third-party perspectives. This storytelling initiative was implemented as a core component in a university media-editing course. We present promising results from a preliminary study conducted to evaluate the utility and potential of our approach in creatively interpreting a unique experiential dataset

Augmenting human memory using personal lifelogs

Memory is a key human facility to support life activities, including social interactions, life management and problem solving. Unfortunately, our memory is not perfect. Normal individuals will have occasional memory problems which can be frustrating, while those with memory impairments can often experience a greatly reduced quality of life. Augmenting memory has the potential to make normal individuals more effective, and those with significant memory problems to have a higher general quality of life. Current technologies are now making it possible to automatically capture and store daily life experiences over an extended period, potentially even over a lifetime. This type of data collection, often referred to as a personal life log (PLL), can include data such as continuously captured pictures or videos from a first person perspective, scanned copies of archival material such as books, electronic documents read or created, and emails and SMS messages sent and received, along with context data of time of capture and access and location via GPS sensors. PLLs offer the potential for memory augmentation. Existing work on PLLs has focused on the technologies of data capture and retrieval, but little work has been done to explore how these captured data and retrieval techniques can be applied to actual use by normal people in supporting their memory. In this paper, we explore the needs for augmenting human memory from normal people based on the psychology literature on mechanisms about memory problems, and discuss the possible functions that PLLs can provide to support these memory augmentation needs. Based on this, we also suggest guidelines for data for capture, retrieval needs and computer-based interface design. Finally we introduce our work-in-process prototype PLL search system in the iCLIPS project to give an example of augmenting human memory with PLLs and computer based interfaces

Experiences of aiding autobiographical memory Using the SenseCam

Human memory is a dynamic system that makes accessible certain memories of events based on a hierarchy of information, arguably driven by personal significance. Not all events are remembered, but those that are tend to be more psychologically relevant. In contrast, lifelogging is the process of automatically recording aspects of one's life in digital form without loss of information. In this article we share our experiences in designing computer-based solutions to assist people review their visual lifelogs and address this contrast. The technical basis for our work is automatically segmenting visual lifelogs into events, allowing event similarity and event importance to be computed, ideas that are motivated by cognitive science considerations of how human memory works and can be assisted. Our work has been based on visual lifelogs gathered by dozens of people, some of them with collections spanning multiple years. In this review article we summarize a series of studies that have led to the development of a browser that is based on human memory systems and discuss the inherent tension in storing large amounts of data but making the most relevant material the most accessible

Exploring Memory Cues to Aid Information Retrieval from Personal LifeLog Archives

The expansion of personal information archives and the emerging field of Personal Lifelogs (PLs) are creating new challenges for information retrieval (IR). While studies have demonstrated the difficulties of IR for these massive data collection [1], we should also think about how we can opportunities and benefits from integrating these data sources as a component of “digital memories” , considering their rich connections with the users‟ memory. We observed that most existing approaches to personal archive IR are mostly technology-driven. Although in recent years studies in Personal Information management (PIM) have claimed to make use of the human memory features, and many works have been reported as investigating well-remembered features of computer files (documents, email, photos). Yet, these explorations are usually confined to the attributes or feature that current computer file systems or technology have provided. I believe that there are important and potentially useful data attributes that these studies have ignored. In addition, current personal search interfaces provide searching options based on what is available in the system, e.g. require users to fill in the calendar date, regardless of the fact that people actually don‟t often encode „time‟ in such a way. My PhD project aims to explore what users actually tend to recall in different personal achieve information seeking tasks, how to present searching options to cater for the right type or format of information that users can recall, and how to exploit this information in an IR system for personal lifelog archives. In this paper, I discuss the limits and advantages of some related work, and present my current and proposed study, with an outlook of an interface that I plan to develop to explore my proposals

What are people's responses to thermal discomfort? Sensing clothing and activity levels using senseCam

Recent international agreements on reducing energy consumption have led to a series of interventions in residential buildings, from modifying the building fabric to upgrading operating systems. To date, these attempts have met with limited success. One reason for this has been identified as the ‘rebound effect’, where the occupants’ respond to their home thermal environment change in unexpected ways after interventions. Often people decide to turn up the heating, to leave it on for longer, or to increase the average spatial temperature by heating more rooms. Although much of the research on heating patterns in dwellings has focused on identifying methods to predict and to assess thermal sensation, less is understood about the way occupants form their responses. Research presented in this paper focuses on mapping householders thermal discomfort responses. Empirical methods, drawn from the social and cognitive sciences, were used in a several studies, which monitored a small sample of UK households during winter of 2010. One of the tools used, the SenseCam, facilitates an automatic electronic diary collection by logging occupants’ responses in a systematic approach.SenseCam results enabled the mapping of participants’ activities in their home, in particular the estimation of clothing and activity level throughout the record period. The preliminary monitoring results show that different householders are interacting with their home thermal comfort systems in very different ways, and that their responses diverge from the current predictive models. Further analysisexamines the factors influencing responses to thermal discomfort and thereby energy consumption of individual in dwellings

Experiences of aiding autobiographical memory using the sensecam

Human memory is a dynamic system that makes accessible certain memories of events based on a hierarchy of information, arguably driven by personal significance. Not all events are remembered, but those that are tend to be more psychologically relevant. In contrast, lifelogging is the process of automatically recording aspects of one's life in digital form without loss of information. In this article we share our experiences in designing computer-based solutions to assist people review their visual lifelogs and address this contrast. The technical basis for our work is automatically segmenting visual lifelogs into events, allowing event similarity and event importance to be computed, ideas that are motivated by cognitive science considerations of how human memory works and can be assisted. Our work has been based on visual lifelogs gathered by dozens of people, some of them with collections spanning multiple years. In this review article we summarize a series of studies that have led to the development of a browser that is based on human memory systems and discuss the inherent tension in storing large amounts of data but making the most relevant material the most accessible