Using machine learning for real-time activity recognition and estimation of energy expenditure

Thesis (Ph. D.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2008.This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.Includes bibliographical references (p. 481-493).Obesity is now considered a global epidemic and is predicted to become the number one preventive health threat in the industrialized world. Presently, over 60% of the U.S. adult population is overweight and 30% is obese. This is of concern because obesity is linked to leading causes of death, such as heart and pulmonary diseases, stroke, and type 2 diabetes. The dramatic rise in obesity rates is attributed to an environment that provides easy access to high caloric food and drink and promotes low levels of physical activity. Unfortunately, many people have a poor understanding of their own daily energy (im)balance: the number of calories they consume from food compared with what they expend through physical activity. Accelerometers offer promise as an objective measure of physical activity. In prior work they have been used to estimate energy expenditure and activity type. This work further demonstrates how wireless accelerometers can be used for real-time automatic recognition of physical activity type, intensity, and duration and estimation of energy expenditure. The parameters of the algorithms such as type of classifier/regressor, feature set, window length, signal preprocessing, sensor set utilized and their placement on the human body are selected by performing a set of incremental experiments designed to identify sets of parameters that may balance system usability with robust, real-time performance in low processing power devices such as mobile phones. The algorithms implemented are evaluated using a dataset of examples of 52 activities collected from 20 participants at a gymnasium and a residential home. The algorithms presented here may ultimately allow for the development of mobile phone-based just-in-time interventions to increase self-awareness of physical activity patterns and increases in physical activity levels in real-time during free-living that scale to large populations.(cont.) KEYWORDS: Activity recognition, context awareness, energy expenditure, physical activity, wearable sensors, obesity, mobile phone, pattern recognition, machine learning, ubiquitous, pervasive, just-in-time.by Emmanuel Munguia Tapia.Ph.D

Acquiring in situ training data for context-aware ubiquitous computing applications

Ubiquitous, context-aware computer systems may ultimately enable computer applications that naturally and usefully respond to a user's everyday activity. Although new algorithms that can automatically detect context from wearable and environmental sensor systems show promise, many of the most flexible and robust systems use probabilistic detection algorithms that require extensive libraries of training data with labeled examples. In this paper, we describe the need for such training data and some challenges we have identified when trying to collect it while testing three contextdetection systems for ubiquitous computing and mobile applications. Author Keywords Context-aware, ubiquitous, computing, supervised learning, experience sampling, user interface design ACM Classification Keywords H5.m Information interfaces and presentation (e.g. HCI): Miscellaneous

Mfingerprint: Privacy-preserving user modeling with multimodal mobile device footprints

Abstract. The dramatic increase of daily usage of mobile devices generates massive digital footprints of users. Such footprints come from physical sensing such as GPS, WiFi, and Bluetooth, as well as social behavior sensing, e.g., call logs, application usage, etc. Many existing studies apply the mobile device footprints to infer daily activities like sitting/standing and social contexts such as personality traits and emotional states. In this paper, we propose a different approach to explore multimodal mobile footprints and build a novel user modeling framework called mFingerprint that can effectively and uniquely depict users. mFingerprint does not expose raw sensitive information from mobile device, e.g., the exact location, WiFi access points, or apps installed, but computes privacy-preserving statistical features to model the user discriminatively. These descriptive features protect sensitive information, thus can be shared, transmitted, and reused with less privacy concerns. By testing on 22 users' mobile phone data collected over 2 months, we demonstrate the effectiveness of mFingerprint in user modeling and identification. In particular, our conditional entropy footprint statistics can achieve 81% accuracy across all 22 users while evaluating over 10-day intervals

Mfingerprint: Privacy-preserving user modeling with multimodal mobile device footprints

Abstract. Mobile devices collect a variety of information about their environments, recording "digital footprints" about the locations and activities of their human owners. These footprints come from physical sensors such as GPS, WiFi, and Bluetooth, as well as social behavior logs like phone calls, application usage, etc. Existing studies analyze mobile device footprints to infer daily activities like driving/running/walking, etc. and social contexts such as personality traits and emotional states. In this paper, we propose a different approach that uses multimodal mobile sensor and log data to build a novel user modeling framework called mFingerprint that can effectively and uniquely depict users. mFingerprint does not expose raw sensitive information from the mobile device, e.g., the exact location, WiFi access points, or apps installed, but computes privacy-preserving statistical features to model the user. These descriptive features obscure sensitive information, and thus can be shared, transmitted, and reused with fewer privacy concerns. By testing on 22 users' mobile phone data collected over 2 months, we demonstrate the effectiveness of mFingerprint in user modeling and identification, with our proposed statistics achieving 81% accuracy across 22 users over 10-day intervals

Spectrum-Guided Adversarial Disparity Learning

It has been a significant challenge to portray intraclass disparity precisely in the area of activity recognition, as it requires a robust representation of the correlation between subject-specific variation for each activity class. In this work, we propose a novel end-to-end knowledge directed adversarial learning framework, which portrays the class-conditioned intraclass disparity using two competitive encoding distributions and learns the purified latent codes by denoising learned disparity. Furthermore, the domain knowledge is incorporated in an unsupervised manner to guide the optimization and further boosts the performance. The experiments on four HAR benchmark datasets demonstrate the robustness and generalization of our proposed methods over a set of state-of-the-art. We further prove the effectiveness of automatic domain knowledge incorporation in performance enhancement

Ubiquitous Sensors

In this work, a system for recognizing activities in the home setting that uses a set of small and simple state-change sensors, machine learning algorithms, and electronic experience sampling is introduced. The sensors are designed to be “tape on and forget” devices that can be quickly and ubiquitously installed in home environments. The proposed sensing system presents an alternative to sensors that are sometimes perceived as invasive, such as cameras and microphones. Since temporal information is an important component of activities, a new algorithm for recognizing activities that extends the naive Bayes classifier to incorporate low-order temporal relationships was created. Unlike prior work, the system was deployed in multiple residential environments with non-researcher occupants. Preliminary results show that it is possible to recognize activities of interest to medical professionals such as toileting, bathing

Activity recognition in the home setting using simple and ubiquitous sensors

Thesis (S.M.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2003.Includes bibliographical references (p. 127-136).During the past several years, researchers have demonstrated that when new wireless sensors are placed in the home environment, data collected from them can be used by software to automatically infer context, such as the activities of daily living. This context-inference can then be exploited in novel applications for healthcare, communication, education, and entertainment. Prior work on automatic context-inference has cleared the way to a reduction in costs associated with manufacturing the sensor technologies and computing resources required by these systems. However, this prior work does not specifically address another major expense of wide-scale deployment of the proposed systems: the expense of expert installation of the sensor devices. To date, most of the context-detection algorithms proposed assume that an expert carefully installs the home sensors and that an expert is involved in acquiring the necessary training examples. End-user sensor installation may offer several advantages over professional sensor installations: 1.) It may greatly reduces the high cost of time required for an expert installation, especially if large numbers of sensors are required for an application, 2.) The process of installing the sensors may give the users a greater sense of control over the technology in their homes, and 3.) End-User Installations also may improve algorithmic performance by leveraging the end-user's domain expertise. An end-user installation method is proposed using "stick on" wireless object usage sensors. The method is then evaluated employing two in-situ, exploratory user studies, where volunteers live in a home fitted with an audio-visual monitoring system. Each participant was given a phone-based tool to help him or her self-install the object usage sensors. They each lived with the sensors for over a week. They were also asked to provide some training data on their everyday activities using multiple methods. Data collected from the two studies is used to qualitatively compare the end-user installation with two professional installation methods. Based on the two exploratory experiments, design guidelines for user self-installation of home sensors are proposed.by Emmanuel Munguia Tapia.S.M

CrowdSignals: A Call to Crowdfund the Community's Largest Mobile Dataset

Abstract Researchers from diverse backgrounds critically depend on mobile datasets. From training and testing activity recognition models, to verifying hypotheses in social science, mobile data is indispensable. Unfortunately, mobile data collection requires significant time and budget for infrastructure as well as subject recruiting, screening, training, legal agreements, equipment, and compensation. We estimate up to 70% of the resources in a study may be spent on data collection. Moreover, this massive investment can combine with institutional, legal, and political issues to create a disincentive to sharing with the community. In this paper, we propose and justify a crowdfunded and crowdsourced methodology for longitudinal mobile data collection that cuts researcher costs by orders of magnitude, removes barriers to data sharing, and boosts data value for all stakeholders. We also present CrowdSignals, a first instantiation which will generate the largest labeled mobile dataset available to the community

AltarNation: interface design for meditative communities

AltarNation allows physically isolated individuals to participate in communities of meditation and tailor their own meditative practices. By lighting candles, users enter a shared virtual community of users represented by a field of stars, each associated with a sound sample of a prayer, song, joy, or concern of another user. Existing practices of individual meditation and candlelight vigils inform this work. This paper describes implementation and design approaches of the AltarNation system

Lessons Learned Using Ubiquitous Sensors for Data Collection in Real Homes

Interface design for the home requires a realistic understanding of the complexity and richness of the human activities that go on there; it is our goal to develop tools that enable HCI investigation in actual home environments. We have developed a kit of ubiquitous sensing devices and over the past year have conducted a series of studies installing a large number of sensors, of diverse types, in multiple homes of participants not affiliated with the research team. As we deployed our portable kit outside the laboratory, we encountered unanticipated study design and technology requirements that will affect the continued development of the kit itself. We offer practical tips we have learned from our experience and describe how we are applying them to the design of our next generation of sensors