Towards Clustering of Mobile and Smartwatch Accelerometer Data for Physical Activity Recognition

Mobile and wearable devices now have a greater capability of sensing human activity ubiquitously and unobtrusively through advancements in miniaturization and sensing abilities. However, outstanding issues remain around the energy restrictions of these devices when processing large sets of data. This paper presents our approach that uses feature selection to refine the clustering of accelerometer data to detect physical activity. This also has a positive effect on the computational burden that is associated with processing large sets of data, as energy efficiency and resource use is decreased because less data is processed by the clustering algorithms. Raw accelerometer data, obtained from smartphones and smartwatches, have been preprocessed to extract both time and frequency domain features. Principle component analysis feature selection (PCAFS) and correlation feature selection (CFS) have been used to remove redundant features. The reduced feature sets have then been evaluated against three widely used clustering algorithms, including hierarchical clustering analysis (HCA), k-means, and density-based spatial clustering of applications with noise (DBSCAN). Using the reduced feature sets resulted in improved separability, reduced uncertainty, and improved efficiency compared with the baseline, which utilized all features. Overall, the CFS approach in conjunction with HCA produced higher Dunn Index results of 9.7001 for the phone and 5.1438 for the watch features, which is an improvement over the baseline. The results of this comparative study of feature selection and clustering, with the specific algorithms used, has not been performed previously and provides an optimistic and usable approach to recognize activities using either a smartphone or smartwatch

A Two-Level Approach to Characterizing Human Activities from Wearable Sensor Data

International audienceThe rapid emergence of new technologies in recent decades has opened up a world of opportunities for a better understanding of human mobility and behavior. It is now possible to recognize human movements, physical activity and the environments in which they take place. And this can be done with high precision, thanks to miniature sensors integrated into our everyday devices. In this paper, we explore different methodologies for recognizing and characterizing physical activities performed by people wearing new smart devices. Whether it's smartglasses, smartwatches or smartphones, we show that each of these specialized wearables has a role to play in interpreting and monitoring moments in a user's life. In particular, we propose an approach that splits the concept of physical activity into two sub-categories that we call micro-and macro-activities. Micro-and macro-activities are supposed to have functional relationship with each other and should therefore help to better understand activities on a larger scale. Then, for each of these levels, we show different methods of collecting, interpreting and evaluating data from different sensor sources. Based on a sensing system we have developed using smart devices, we build two data sets before analyzing how to recognize such activities. Finally, we show different interactions and combinations between these scales and demonstrate that they have the potential to lead to new classes of applications, involving authentication or user profiling

Instructor Activity Recognition Using Smartwatch and Smartphone Sensors

During a classroom session, an instructor performs several activities, such as writing on the board, speaking to the students, gestures to explain a concept. A record of the time spent in each of these activities could be valuable information for the instructors to virtually observe their own style of instruction. It can help in identifying activities that engage the students more, thereby enhancing teaching effectiveness and efficiency. In this work, we present a preliminary study on profiling multiple activities of an instructor in the classroom using smartwatch and smartphone sensor data. We use 2 benchmark datasets to test out the feasibility of classifying the activities. Comparing multiple machine learning techniques, we finally propose a hybrid deep recurrent neural network based approach that performs better than the other techniques

An IoT based Virtual Coaching System (VSC) for Assisting Activities of Daily Life

Nowadays aging of the population is becoming one of the main concerns of theworld. It is estimated that the number of people aged over 65 will increase from 461million to 2 billion in 2050. This substantial increment in the elderly population willhave significant consequences in the social and health care system. Therefore, in thecontext of Ambient Intelligence (AmI), the Ambient Assisted Living (AAL) has beenemerging as a new research area to address problems related to the aging of the population. AAL technologies based on embedded devices have demonstrated to be effectivein alleviating the social- and health-care issues related to the continuous growing of theaverage age of the population. Many smart applications, devices and systems have beendeveloped to monitor the health status of elderly, substitute them in the accomplishment of activities of the daily life (especially in presence of some impairment or disability),alert their caregivers in case of necessity and help them in recognizing risky situations.Such assistive technologies basically rely on the communication and interaction be-tween body sensors, smart environments and smart devices. However, in such contextless effort has been spent in designing smart solutions for empowering and supportingthe self-efficacy of people with neurodegenerative diseases and elderly in general. Thisthesis fills in the gap by presenting a low-cost, non intrusive, and ubiquitous VirtualCoaching System (VCS) to support people in the acquisition of new behaviors (e.g.,taking pills, drinking water, finding the right key, avoiding motor blocks) necessary tocope with needs derived from a change in their health status and a degradation of theircognitive capabilities as they age. VCS is based on the concept of extended mind intro-duced by Clark and Chalmers in 1998. They proposed the idea that objects within theenvironment function as a part of the mind. In my revisiting of the concept of extendedmind, the VCS is composed of a set of smart objects that exploit the Internet of Things(IoT) technology and machine learning-based algorithms, in order to identify the needsof the users and react accordingly. In particular, the system exploits smart tags to trans-form objects commonly used by people (e.g., pillbox, bottle of water, keys) into smartobjects, it monitors their usage according to their needs, and it incrementally guidesthem in the acquisition of new behaviors related to their needs. To implement VCS, thisthesis explores different research directions and challenges. First of all, it addresses thedefinition of a ubiquitous, non-invasive and low-cost indoor monitoring architecture byexploiting the IoT paradigm. Secondly, it deals with the necessity of developing solu-tions for implementing coaching actions and consequently monitoring human activitiesby analyzing the interaction between people and smart objects. Finally, it focuses on the design of low-cost localization systems for indoor environment, since knowing theposition of a person provides VCS with essential information to acquire information onperformed activities and to prevent risky situations. In the end, the outcomes of theseresearch directions have been integrated into a healthcare application scenario to imple-ment a wearable system that prevents freezing of gait in people affected by Parkinson\u2019sDisease

SmartEAR: Smartwatch-based Unsupervised Learning for Multi-modal Signal Analysis in Opportunistic Sensing Framework

Wrist-bands such as smartwatches have become an unobtrusive interface for collecting physiological and contextual data from users. Smartwatches are being used for smart healthcare, telecare, and wellness monitoring. In this paper, we used data collected from the AnEAR framework leveraging smartwatches to gather and store physiological data from patients in naturalistic settings. This data included temperature, galvanic skin response (GSR), acceleration, and heart rate (HR). In particular, we focused on HR and acceleration, as these two modalities are often correlated. Since the data was unlabeled we relied on unsupervised learning for multi-modal signal analysis. We propose using k-means clustering, GMM clustering, and Self-Organizing maps based on Neural Networks for group the multi-modal data into homogeneous clusters. This strategy helped in discovering latent structures in our data

Multimodal Sensor Data Integration for Indoor Positioning in Ambient-Assisted Living Environments

A reliable Indoor Positioning System (IPS) is a crucial part of the Ambient-Assisted Living (AAL) concept. The use of Wi-Fi fingerprinting techniques to determine the location of the user, based on the Received Signal Strength Indication (RSSI) mapping, avoids the need to deploy a dedicated positioning infrastructure but comes with its own issues. Heterogeneity of devices and RSSI variability in space and time due to environment changing conditions pose a challenge to positioning systems based on this technique. The primary purpose of this research is to examine the viability of leveraging other sensors in aiding the positioning system to provide more accurate predictions. In particular, the experiments presented in this work show that Inertial Motion Units (IMU), which are present by default in smart devices such as smartphones or smartwatches, can increase the performance of Indoor Positioning Systems in AAL environments. Furthermore, this paper assesses a set of techniques to predict the future performance of the positioning system based on the training data, as well as complementary strategies such as data scaling and the use of consecutive Wi-Fi scanning to further improve the reliability of the IPS predictions. This research shows that a robust positioning estimation can be derived from such strategies

A 'one-size-fits-most' walking recognition method for smartphones, smartwatches, and wearable accelerometers

The ubiquity of personal digital devices offers unprecedented opportunities to study human behavior. Current state-of-the-art methods quantify physical activity using 'activity counts,' a measure which overlooks specific types of physical activities. We proposed a walking recognition method for sub-second tri-axial accelerometer data, in which activity classification is based on the inherent features of walking: intensity, periodicity, and duration. We validated our method against 20 publicly available, annotated datasets on walking activity data collected at various body locations (thigh, waist, chest, arm, wrist). We demonstrated that our method can estimate walking periods with high sensitivity and specificity: average sensitivity ranged between 0.92 and 0.97 across various body locations, and average specificity for common daily activities was typically above 0.95. We also assessed the method's algorithmic fairness to demographic and anthropometric variables and measurement contexts (body location, environment). Finally, we have released our method as open-source software in MATLAB and Python.Comment: 39 pages, 4 figures (incl. 1 supplementary), and 5 tables (incl. 2 supplementary

EFFICIENT AND SECURE ALGORITHMS FOR MOBILE CROWDSENSING THROUGH PERSONAL SMART DEVICES.

The success of the modern pervasive sensing strategies, such as the Social Sensing, strongly depends on the diffusion of smart mobile devices. Smartwatches, smart- phones, and tablets are devices capable of capturing and analyzing data about the user’s context, and can be exploited to infer high-level knowledge about the user himself, and/or the surrounding environment. In this sense, one of the most relevant applications of the Social Sensing paradigm concerns distributed Human Activity Recognition (HAR) in scenarios ranging from health care to urban mobility management, ambient intelligence, and assisted living. Even though some simple HAR techniques can be directly implemented on mo- bile devices, in some cases, such as when complex activities need to be analyzed timely, users’ smart devices should be able to operate as part of a more complex architecture, paving the way to the definition of new distributed computing paradigms. The general idea behind these approaches is to move early analysis to- wards the edge of the network, while relying on other intermediate (fog) or remote (cloud) devices for computations of increasing complexity. This logic represents the main core of the fog computing paradigm, and this thesis investigates its adoption in distributed sensing frameworks. Specifically, the conducted analysis focused on the design of a novel distributed HAR framework in which the heavy computation from the sensing layer is moved to intermediate devices and then to the cloud. Smart personal devices are used as processing units in order to guarantee real-time recognition, whereas the cloud is responsible for maintaining an overall, consistent view of the whole activity set. As compared to traditional cloud-based solutions, this choice allows to overcome processing and storage limitations of wearable devices while also reducing the overall bandwidth consumption. Then, the fog-based architecture allowed the design and definition of a novel HAR technique that combines three machine learning algorithms, namely k-means clustering, Support Vector Machines (SVMs), and Hidden Markov Models (HMMs), to recognize complex activities modeled as sequences of simple micro- activities. The capability to distribute the computation over the different entities in the network, allowing the use of complex HAR algorithms, is definitely one of the most significant advantages provided by the fog architecture. However, because both of its intrinsic nature and high degree of modularity, the fog-based system is particularly prone to cyber security attacks that can be performed against every element of the infrastructure. This aspect plays a main role with respect to social sensing since the users’ private data must be preserved from malicious purposes. Security issues are generally addressed by introducing cryptographic mechanisms that improve the system defenses against cyber attackers while, at the same time, causing an increase of the computational overhead for devices with limited resources. With the goal to find a trade-off between security and computation cost, the de- sign and definition of a secure lightweight protocol for social-based applications are discussed and then integrated into the distributed framework. The protocol covers all tasks commonly required by a general fog-based crowdsensing application, making it applicable not only in a distributed HAR scenario, discussed as a case study, but also in other application contexts. Experimental analysis aims to assess the performance of the solutions described so far. After highlighting the benefits the distributed HAR framework might bring in smart environments, an evaluation in terms of both recognition accuracy and complexity of data exchanged between network devices is conducted. Then, the effectiveness of the secure protocol is demonstrated by showing the low impact it causes on the total computational overhead. Moreover, a comparison with other state-of-art protocols is made to prove its effectiveness in terms of the provided security mechanisms

Improving the Security of Smartwatch Payment with Deep Learning

Making contactless payments using a smartwatch is increasingly popular, but this payment medium lacks traditional biometric security measures such as facial or fingerprint recognition. In 2022, Sturgess et al. proposed WatchAuth, a system for authenticating smartwatch payments using the physical gesture of reaching towards a payment terminal. While effective, the system requires the user to undergo a burdensome enrolment period to achieve acceptable error levels. In this dissertation, we explore whether applications of deep learning can reduce the number of gestures a user must provide to enrol into an authentication system for smartwatch payment. We firstly construct a deep-learned authentication system that outperforms the current state-of-the-art, including in a scenario where the target user has provided a limited number of gestures. We then develop a regularised autoencoder model for generating synthetic user-specific gestures. We show that using these gestures in training improves classification ability for an authentication system. Through this technique we can reduce the number of gestures required to enrol a user into a WatchAuth-like system without negatively impacting its error rates.Comment: Master's thesis, 74 pages. 32 figure