Recognition of quotidian activities in support of independent living using a single wrist-worn inertial measurement unit

The field of Ambient Assisted Living (AAL) is gaining increasing attention from the research community in recent years with the rapid present and future ageing of the population worldwide. This problem has been widely recognised as has the need for it to be addressed both from an economic and societal perspective. Assisted living environments incorporate technological solutions to create a better condition of life for older adults. However, in order to create a better condition of life, it is crucial to understand the specific needs of each individual. To this regard, self-assessment of daily activities has shown to be subjective and variable, presenting important discrepancies with those performed by clinicians. The above challenges have fostered the search for alternative monitoring solutions, increasing the research efforts upon the field of Human Activity Recognition (HAR). A vast array of sensing devices, including ambient sensors, video cameras and wearable devices, has been employed for the automatic monitoring of a person in a home environment. However, the research focus is shifting towards wearable solutions, which avoid the privacy concerns related to the use of video cameras in a home environment while providing more intrinsic information about the user than ambient devices. The focus of this research is the investigation of signal processing and machine learning techniques for the recognition of quotidian activities concerning self-neglect (a behavioural condition in which individuals, generally older people, disregard the attention, intentionally or un intentionally, of their basic needs). More precisely, the aimed group of activities include those concerning personal hygiene, namely handswashing and teeth brushing, as well as those directly related to dietary behaviour, namely eating and drinking. The work undertaken in this thesis is divided into three different stages. First, given the continuous quasi-periodic behaviour of hands washing and teeth brushing, these are studied alongside a group of other quotidian activities which also exhibit continuity during their performance. These studies include the investigation of informative features for activity recognition as well as relevant classification models and signal processing techniques. In addition, a novel multi-level refinement approach is proposed as a way to improve the classification rate of those activities with lower inter-activity classification rate. Second, a novel framework for fluid and food intake gesture recognition is developed. As opposed to the above activities, the nature of eating and drinking activities is neither static nor quasi-periodic. Instead, they are composed of sparsely occurring motions or gestures in continuous data streams. Given this characteristic, a novel signal segmentation technique, namely the Crossings-based Adaptive Segmentation Technique (CAST), is proposed to identify potential eating and drinking gestures while filtering out the remaining unwanted segments of the signals. In addition, various feature descriptors, namely a Soft Dynamic Time Warping (DTW) gesture discrepancy measure and time series to image encoding techniques, as well as various deep learning architectures are explored to overcome the notable existing similarity between eating and drinking gestures. The third stage of the work aims at the identification of meal periods through the analysis of the distribution of eating gestures along time using low-computational cost signal processing techniques, including a moving average and an entropy measure. The novel computational solutions and the results presented in this thesis, demonstrate a significant contribution towards the recognition of quotidian activities in support of independent living

A deep learning based wearable system for food and drink intake recognition

Eating difficulties and the subsequent need for eating assistance are a prevalent issue within the elderly population. Besides, a poor diet is considered a confounding factor for developing chronic diseases and functional limitations. Driven by the above issues, this paper proposes a wrist-worn tri-axial accelerometer based food and drink intake recognition system. First, an adaptive segmentation technique is employed to identify potential eating and drinking gestures from the continuous accelerometer readings. A posteriori, a study upon the use of Convolutional Neural Networks for the recognition of eating and drinking gestures is carried out. This includes the employment of three time series to image encoding frameworks, namely the signal spectrogram, the Markov Transition Field and the Gramian Angular Field, as well as the development of various multi-input multi-domain networks. The recognition of the gestures is then tackled as a 3-class classification problem (‘Eat’, ‘Drink’ and ‘Null’), where the ‘Null’ class is composed of all the irrelevant gestures included in the post-segmentation gesture set. An average per-class classification accuracy of 97.10% was achieved by the proposed system. When compared to similar work, such accurate classification performance signifies a great contribution to the field of assisted living

Detecting Periods of Eating in Everyday Life by Tracking Wrist Motion — What is a Meal?

Eating is one of the most basic activities observed in sentient animals, a behavior so natural that humans often eating without giving the activity a second thought. Unfortunately, this often leads to consuming more calories than expended, which can cause weight gain - a leading cause of diseases and death. This proposal describes research in methods to automatically detect periods of eating by tracking wrist motion so that calorie consumption can be tracked. We first briefly discuss how obesity is caused due to an imbalance in calorie intake and expenditure. Calorie consumption and expenditure can be tracked manually using tools like paper diaries, however it is well known that human bias can affect the accuracy of such tracking. Researchers in the upcoming field of automated dietary monitoring (ADM) are attempting to track diet using electronic methods in an effort to mitigate this bias. We attempt to replicate a previous algorithm that detects eating by tracking wrist motion electronically. The previous algorithm was evaluated on data collected from 43 subjects using an iPhone as the sensor. Periods of time are segmented first, and then classified using a naive Bayesian classifier. For replication, we describe the collection of the Clemson all-day data set (CAD), a free-living eating activity dataset containing 4,680 hours of wrist motion collected from 351 participants - the largest of its kind known to us. We learn that while different sensors are available to log wrist acceleration data, no unified convention exists, and this data must thus be transformed between conventions. We learn that the performance of the eating detection algorithm is affected due to changes in the sensors used to track wrist motion, increased variability in behavior due to a larger participant pool, and the ratio of eating to non-eating in the dataset. We learn that commercially available acceleration sensors contain noise in their reported readings which affects wrist tracking specifically due to the low magnitude of wrist acceleration. Commercial accelerometers can have noise up to 0.06g which is acceptable in applications like automobile crash testing or pedestrian indoor navigation, but not in ones using wrist motion. We quantify linear acceleration noise in our free-living dataset. We explain sources of noise, a method to mitigate it, and also evaluate the effect of this noise on the eating detection algorithm. By visualizing periods of eating in the collected dataset we learn that that people often conduct secondary activities while eating, such as walking, watching television, working, and doing household chores. These secondary activities cause wrist motions that obfuscate wrist motions associated with eating, which increases the difficulty of detecting periods of eating (meals). Subjects reported conducting secondary activities in 72% of meals. Analysis of wrist motion data revealed that the wrist was resting 12.8% of the time during self-reported meals, compared to only 6.8% of the time in a cafeteria dataset. Walking motion was found during 5.5% of the time during meals in free-living, compared to 0% in the cafeteria. Augmenting an eating detection classifier to include walking and resting detection improved the average per person accuracy from 74% to 77% on our free-living dataset (t[353]=7.86, p\u3c0.001). This suggests that future data collections for eating activity detection should also collect detailed ground truth on secondary activities being conducted during eating. Finally, learning from this data collection, we describe a convolutional neural network (CNN) to detect periods of eating by tracking wrist motion during everyday life. Eating uses hand-to-mouth gestures for ingestion, each of which lasts appx 1-5 sec. The novelty of our new approach is that we analyze a much longer window (0.5-15 min) that can contain other gestures related to eating, such as cutting or manipulating food, preparing foods for consumption, and resting between ingestion events. The context of these other gestures can improve the detection of periods of eating. We found that accuracy at detecting eating increased by 15% in longer windows compared to shorter windows. Overall results on CAD were 89% detection of meals with 1.7 false positives for every true positive (FP/TP), and a time weighted accuracy of 80%

Investigation of Low-Cost Wearable Internet of Things Enabled Technology for Physical Activity Recognition in the Elderly

Technological advances in mobile sensing technologies has produced new opportunities for the monitoring of the elderly in uncontrolled environments by researchers. Sensors have become smaller, cheaper and can be worn on the body, potentially creating a network of sensors. Smart phones are also more common in the average household and can also provide some behavioural analysis due to the built-in sensors. As a result of this, researchers are able to monitor behaviours in a more naturalistic setting, which can lead to more contextually meaningful data. For those suffering with a mental illness, non-invasive and continuous monitoring can be achieved. Applying sensors to real world environments can aid in improving the quality of life of an elderly person with a mental illness and monitor their condition through behavioural analysis. In order to achieve this, selected classifiers must be able to accurately detect when an activity has taken place. In this thesis we aim to provide a framework for the investigation of activity recognition in the elderly using low-cost wearable sensors, which has resulted in the following contributions: 1. Classification of eighteen activities which were broken down into three disparate categories typical in a home setting: dynamic, sedentary and transitional. These were detected using two Shimmer3 IMU devices that we have located on the participants’ wrist and waist to create a low-cost, contextually deployable solution for elderly care monitoring. 2. Through the categorisation of performed Extracted time-domain and frequency-domain features from the Shimmer devices accelerometer and gyroscope were used as inputs, we achieved a high accuracy classification from a Convolutional Neural Network (CNN) model applied to the data set gained from participants recruited to the study through Join Dementia Research. The model was evaluated by variable adjustments to the model, tracking changes in its performance. Performance statistics were generated by the model for comparison and evaluation. Our results indicate that a low epoch of 200 using the ReLu activation function can display a high accuracy of 86% on the wrist data set and 85% on the waist data set, using only two low-cost wearable devices

Designing Efficient and Accurate Behavior-Aware Mobile Systems

The proliferation of sensors on smartphones, tablets and wearables has led to a plethora of behavior classification algorithms designed to sense various aspects of individual user\u27s behavior such as daily habits, activity, physiology, mobility, sleep, emotional and social contexts. This ability to sense and understand behaviors of mobile users will drive the next generation of mobile applications providing services based on the users\u27 behavioral patterns. In this thesis, we investigate ways in which we can enhance and utilize the understanding of user behaviors in such applications. In particular, we focus on identifying the key challenges in the following three aspects of behavior-aware applications: detection, understanding, and prediction of user behaviors; and present systems and techniques developed to address these challenges. In this thesis, we first demonstrate the utility of wristbands equipped with inertial sensors in real-time detection of health-related behaviors such as smoking and eating. Our approach detects these behaviors in a passive manner without any explicit user interaction and does not require use of any cumbersome device. Our results show that we can detect smoking with 95% accuracy, 91% precision and 81% recall in the natural environment. Second, we design a context-query engine for sensing multiple user contexts continuously, accurately and efficiently on mobile devices; the key necessity for understanding and analyzing behaviors. Our context-query engine performs information fusion of contexts for an individual user to enable optimizations like i) energy-efficient sensing, and ii) accurate context inference. Our results show that we can improve accuracy of a context classifier by up to 42% and reduce the number of classifiers required to observe the user state by 33%. Finally, we demonstrate the utility of predicting app usage behavior, in improving the freshness of mobile apps such as Facebook that present users with the latest content fetched from remote servers. We present an app prediction algorithm that utilizes user contexts to predict the app a user is likely to use and pre-fetches the data over the network for the predicted app. We show that our proposed algorithm delivers application content to the user that is on an average fresh within 3 minutes

Early Abstraction of Inertial Sensor Data for Long-Term Deployments

Advances in microelectronics over the last decades have led to miniaturization of computing devices and sensors. A driving force to use these in various application scenarios is the desire to grasp physical phenomena from the environment, objects and living entities. We investigate sensing in two particularly challenging applications: one where small sensor modules are worn by people to detect their activities, and one where wirelessly networked sensors observe events over an area. This thesis takes a data-driven approach, focusing on human motion and vibrations caused by trains that are captured by accelerometer sensors as time series and shall be analyzed for characteristic patterns. For both, the acceleration sensor must be sampled at relatively high rates in order to capture the essence of the phenomena, and remain active for long stretches of time. The large amounts of gathered sensor data demand novel approaches that are able to swiftly process the data while guaranteeing accurate classification results. The following contributions are made in particular: * A data logger that would suit the requirements of long-term deployments is designed and evaluated. In a power profiling study both hardware components and firmware parameters are thoroughly tested, revealing that the sensor is able to log acceleration data at a sampling rate of 100 Hertz for up to 14 full days on a single battery charge. * A technique is proposed that swiftly and accurately abstracts an original signal with a set of linear segments, thus preserving its shape, while being twice as fast as a similar method. This allows for more efficient pattern matching, since for each pattern only a fraction of data points must be considered. A second study shows that this algorithm can perform data abstraction directly on a data logger with limited resources. * The railway monitoring scenario requires streaming vibration data to be analyzed for particular sparse and complex events directly on the sensor node, extracting relevant information such as train type or length from the shape of the vibration footprint. In a study conducted on real-world data, a set of efficient shape features is identified that facilitates train type prediction and length estimation with very high accuracies. * To achieve fast and accurate activity recognition for long-term bipolar patients monitoring scenarios, we present an approach that relies on the salience of motion patterns (motifs) that are characteristic for the target activity. These motifs are accumulated by using a symbolic abstraction that encodes the shape of the original signal. A large-scale study shows that a simple bag-of-words classifier trained with extracted motifs is on par with traditional approaches regarding the accuracy, while being much faster. * Some activities are hard to predict from acceleration data alone with the aforementioned approach. We argue that human-object interactions, captured as human motion and grasped objects through RFID, are an ideal supplement. A custom bracelet-like antenna to detect objects from up to 14 cm is proposed, along with a novel benchmark to evaluate such wearable setups. By aiming for wearable and wirelessly networked sensor systems, these contributions apply for particularly challenging applications that require long-term deployments of miniature sensors in general. They form the basis of a framework towards efficient event detection that relies heavily on early data abstraction and shape-based features for time series, while focusing less on the classification techniques

Activity recognition in naturalistic environments using body-worn sensors

Phd ThesisThe research presented in this thesis investigates how deep learning and feature learning can address challenges that arise for activity recognition systems in naturalistic, ecologically valid surroundings such as the private home. One of the main aims of ubiquitous computing is the development of automated recognition systems for human activities and behaviour that are sufficiently robust to be deployed in realistic, in-the-wild environments. In most cases, the targeted application scenarios are people’s daily lives, where systems have to abide by practical usability and privacy constraints. We discuss how these constraints impact data collection and analysis and demonstrate how common approaches to the analysis of movement data effectively limit the practical use of activity recognition systems in every-day surroundings. In light of these issues we develop a novel approach to the representation and modelling of movement data based on a data-driven methodology that has applications in activity recognition, behaviour imaging, and skill assessment in ubiquitous computing. A number of case studies illustrate the suitability of the proposed methods and outline how study design can be adapted to maximise the benefit of these techniques, which show promising performance for clinical applications in particular.SiDE research hu

Multi-Dimensional Task Recognition for Human-Robot Teaming

Human-robot teams involve humans and robots collaborating to achieve tasks under various environmental conditions. Successful teaming requires robots to adapt autonomously in real-time to a human teammate's state. An important element of such adaptation is the ability for the robot to infer the tasks performed by their human teammates. Human-robot teams often perform a wide variety of tasks, involving multiple activity components, and may even perform two or more tasks concurrently. A robot’s ability to recognize the human’s composite tasks that occur concurrently is a key requirement for realizing successful collaboration. Existing task recognition algorithms are not viable for human-robot teams, as they only detect tasks from a subset of activity components and rarely detect concurrent, composite tasks. This dissertation developed a multi-dimensional task recognition algorithm capable of detecting concurrent, composite tasks across the cognitive, speech, auditory, visual, gross motor, fine-grained motor, and tactile components by incorporating metrics that are sensitive, versatile, and suitable across human-robot teaming paradigms. The developed algorithm addresses a foundational problem of understanding an individual's task engagement state in human-robot teams operating in dynamic, unstructured environments

SHELDON Smart habitat for the elderly.

An insightful document concerning active and assisted living under different perspectives: Furniture and habitat, ICT solutions and Healthcare