Augmenting human memory using personal lifelogs

Memory is a key human facility to support life activities, including social interactions, life management and problem solving. Unfortunately, our memory is not perfect. Normal individuals will have occasional memory problems which can be frustrating, while those with memory impairments can often experience a greatly reduced quality of life. Augmenting memory has the potential to make normal individuals more effective, and those with significant memory problems to have a higher general quality of life. Current technologies are now making it possible to automatically capture and store daily life experiences over an extended period, potentially even over a lifetime. This type of data collection, often referred to as a personal life log (PLL), can include data such as continuously captured pictures or videos from a first person perspective, scanned copies of archival material such as books, electronic documents read or created, and emails and SMS messages sent and received, along with context data of time of capture and access and location via GPS sensors. PLLs offer the potential for memory augmentation. Existing work on PLLs has focused on the technologies of data capture and retrieval, but little work has been done to explore how these captured data and retrieval techniques can be applied to actual use by normal people in supporting their memory. In this paper, we explore the needs for augmenting human memory from normal people based on the psychology literature on mechanisms about memory problems, and discuss the possible functions that PLLs can provide to support these memory augmentation needs. Based on this, we also suggest guidelines for data for capture, retrieval needs and computer-based interface design. Finally we introduce our work-in-process prototype PLL search system in the iCLIPS project to give an example of augmenting human memory with PLLs and computer based interfaces

What do people want from their lifelogs?

The practice of lifelogging potentially consists of automatically capturing and storing a digital record of every piece of information that a person (lifelogger) encounters in their daily experiences. Lifelogging has become an increasingly popular area of research in recent years. Most current lifeloggiing research focuses on techniques for data capture or processing. Current applications of lifelogging technology are usually driven by new technology inventions, creative ideas of researchers, or the special needs of a particular user group, e.g. individuals with memory impairment. To the best of our knowledge, little work has explored potential lifelogs applications from the perspective of the desires of the general public. One of the difficulties of carrying out such a study is the balancing of the information given to the subject regarding lifelog technology to enable them to generate realistic ideas without limiting or directing their imaginations by providing too much specific information. We report a study in which we take a progressive approach where we introduce lifelogging in three stages, and collect the ideas and opinions of a volunteer group of general public participants on techniques for lifelog capture, and applications and functionality

Experiences of aiding autobiographical memory Using the SenseCam

Human memory is a dynamic system that makes accessible certain memories of events based on a hierarchy of information, arguably driven by personal significance. Not all events are remembered, but those that are tend to be more psychologically relevant. In contrast, lifelogging is the process of automatically recording aspects of one's life in digital form without loss of information. In this article we share our experiences in designing computer-based solutions to assist people review their visual lifelogs and address this contrast. The technical basis for our work is automatically segmenting visual lifelogs into events, allowing event similarity and event importance to be computed, ideas that are motivated by cognitive science considerations of how human memory works and can be assisted. Our work has been based on visual lifelogs gathered by dozens of people, some of them with collections spanning multiple years. In this review article we summarize a series of studies that have led to the development of a browser that is based on human memory systems and discuss the inherent tension in storing large amounts of data but making the most relevant material the most accessible

Experiences of aiding autobiographical memory using the sensecam

Human memory is a dynamic system that makes accessible certain memories of events based on a hierarchy of information, arguably driven by personal significance. Not all events are remembered, but those that are tend to be more psychologically relevant. In contrast, lifelogging is the process of automatically recording aspects of one's life in digital form without loss of information. In this article we share our experiences in designing computer-based solutions to assist people review their visual lifelogs and address this contrast. The technical basis for our work is automatically segmenting visual lifelogs into events, allowing event similarity and event importance to be computed, ideas that are motivated by cognitive science considerations of how human memory works and can be assisted. Our work has been based on visual lifelogs gathered by dozens of people, some of them with collections spanning multiple years. In this review article we summarize a series of studies that have led to the development of a browser that is based on human memory systems and discuss the inherent tension in storing large amounts of data but making the most relevant material the most accessible

Creating human digital memories with the aid of pervasive mobile devices

The abundance of mobile and sensing devices, within our environment, has led to a society in which any object, embedded with sensors, is capable of providing us with information. A human digital memory, created with the data from these pervasive devices, produces a more dynamic and data rich memory. Information such as how you felt, where you were and the context of the environment can be established. This paper presents the DigMem system, which utilizes distributed mobile services, linked data and machine learning to create such memories. Along with the design of the system, a prototype has also been developed, and two case studies have been undertaken, which successfully create memories. As well as demonstrating how memories are created, a key concern in human digital memory research relates to the amount of data that is generated and stored. In particular, searching this set of big data is a key challenge. In response to this, the paper evaluates the use of machine learning algorithms, as an alternative to SPARQL, and treats searching as a classification problem. In particular, supervised machine learning algorithms are used to find information in semantic annotations, based on probabilistic reasoning. Our approach produces good results with 100% sensitivity, 93% specificity, 93% positive predicted value, 100% negative predicted value, and an overall accuracy of 97%

LifeLogging: personal big data

We have recently observed a convergence of technologies to foster the emergence of lifelogging as a mainstream activity. Computer storage has become significantly cheaper, and advancements in sensing technology allows for the efficient sensing of personal activities, locations and the environment. This is best seen in the growing popularity of the quantified self movement, in which life activities are tracked using wearable sensors in the hope of better understanding human performance in a variety of tasks. This review aims to provide a comprehensive summary of lifelogging, to cover its research history, current technologies, and applications. Thus far, most of the lifelogging research has focused predominantly on visual lifelogging in order to capture life details of life activities, hence we maintain this focus in this review. However, we also reflect on the challenges lifelogging poses to an information retrieval scientist. This review is a suitable reference for those seeking a information retrieval scientist’s perspective on lifelogging and the quantified self

Creating human digital memories with the aid of pervasive mobile devices

The abundance of mobile and sensing devices, within our environment, has led to a society in which any object, embedded with sensors, is capable of providing us with information. A human digital memory, created with the data from these pervasive devices, produces a more dynamic and data rich memory. Information such as how you felt, where you were and the context of the environment can be established. This paper presents the DigMem system, which utilizes distributed mobile services, linked data and machine learning to create such memories. Along with the design of the system, a prototype has also been developed, and two case studies have been undertaken, which successfully create memories. As well as demonstrating how memories are created, a key concern in human digital memory research relates to the amount of data that is generated and stored. In particular, searching this set of big data is a key challenge. In response to this, the paper evaluates the use of machine learning algorithms, as an alternative to SPARQL, and treats searching as a classification problem. In particular, supervised machine learning algorithms are used to find information in semantic annotations, based on probabilistic reasoning. Our approach produces good results with 100% sensitivity, 93% specificity, 93% positive predicted value, 100% negative predicted value, and an overall accuracy of 97%. Crown Copyright © 2013 Published by Elsevier B.V. All rights reserved

Information access tasks and evaluation for personal lifelogs

Emerging personal lifelog (PL) collections contain permanent digital records of information associated with individuals’ daily lives. This can include materials such as emails received and sent, web content and other documents with which they have interacted, photographs, videos and music experienced passively or created, logs of phone calls and text messages, and also personal and contextual data such as location (e.g. via GPS sensors), persons and objects present (e.g. via Bluetooth) and physiological state (e.g. via biometric sensors). PLs can be collected by individuals over very extended periods, potentially running to many years. Such archives have many potential applications including helping individuals recover partial forgotten information, sharing experiences with friends or family, telling the story of one’s life, clinical applications for the memory impaired, and fundamental psychological investigations of memory. The Centre for Digital Video Processing (CDVP) at Dublin City University is currently engaged in the collection and exploration of applications of large PLs. We are collecting rich archives of daily life including textual and visual materials, and contextual context data. An important part of this work is to consider how the effectiveness of our ideas can be measured in terms of metrics and experimental design. While these studies have considerable similarity with traditional evaluation activities in areas such as information retrieval and summarization, the characteristics of PLs mean that new challenges and questions emerge. We are currently exploring the issues through a series of pilot studies and questionnaires. Our initial results indicate that there are many research questions to be explored and that the relationships between personal memory, context and content for these tasks is complex and fascinating

Exploring Memory Cues to Aid Information Retrieval from Personal LifeLog Archives

The expansion of personal information archives and the emerging field of Personal Lifelogs (PLs) are creating new challenges for information retrieval (IR). While studies have demonstrated the difficulties of IR for these massive data collection [1], we should also think about how we can opportunities and benefits from integrating these data sources as a component of “digital memories” , considering their rich connections with the users‟ memory. We observed that most existing approaches to personal archive IR are mostly technology-driven. Although in recent years studies in Personal Information management (PIM) have claimed to make use of the human memory features, and many works have been reported as investigating well-remembered features of computer files (documents, email, photos). Yet, these explorations are usually confined to the attributes or feature that current computer file systems or technology have provided. I believe that there are important and potentially useful data attributes that these studies have ignored. In addition, current personal search interfaces provide searching options based on what is available in the system, e.g. require users to fill in the calendar date, regardless of the fact that people actually don‟t often encode „time‟ in such a way. My PhD project aims to explore what users actually tend to recall in different personal achieve information seeking tasks, how to present searching options to cater for the right type or format of information that users can recall, and how to exploit this information in an IR system for personal lifelog archives. In this paper, I discuss the limits and advantages of some related work, and present my current and proposed study, with an outlook of an interface that I plan to develop to explore my proposals