Detection of bimanual gestures everywhere: why it matters, what we need and what is missing

Bimanual gestures are of the utmost importance for the study of motor coordination in humans and in everyday activities. A reliable detection of bimanual gestures in unconstrained environments is fundamental for their clinical study and to assess common activities of daily living. This paper investigates techniques for a reliable, unconstrained detection and classification of bimanual gestures. It assumes the availability of inertial data originating from the two hands/arms, builds upon a previously developed technique for gesture modelling based on Gaussian Mixture Modelling (GMM) and Gaussian Mixture Regression (GMR), and compares different modelling and classification techniques, which are based on a number of assumptions inspired by literature about how bimanual gestures are represented and modelled in the brain. Experiments show results related to 5 everyday bimanual activities, which have been selected on the basis of three main parameters: (not) constraining the two hands by a physical tool, (not) requiring a specific sequence of single-hand gestures, being recursive (or not). In the best performing combination of modeling approach and classification technique, five out of five activities are recognized up to an accuracy of 97%, a precision of 82% and a level of recall of 100%.Comment: Submitted to Robotics and Autonomous Systems (Elsevier

Uncertainty Investigation for Personalised Lifelogging Physical Activity Intensity Pattern Assessment with Mobile Devices

Lifelogging physical activity (PA) assessment is crucial to healthcare technologies and studies for the purpose of treatments and interventions of chronic diseases. Traditional lifelogging PA monitoring is conducted in non-naturalistic settings by means of wearable devices or mobile phones such as fixed placements, controlled durations or dedicated sensors. Although they achieved satisfactory outcomes for healthcare studies, the practicability become the key issues. Recent advance of mobile devices make lifelogging PA tracking for healthy or unhealthy individuals possible. However, owning to diverse physical characteristics, immaturity of PA recognition techniques, different settings from manufactories and a majority of uncertainties in real life, the results of PA measurement is leading to be inapplicable for PA pattern detection in a long range, especially hardly exploited in the wellbeing monitoring or behaviour changes. This paper investigates and compares uncertainties of existing mobile devices for individual’s PA tracking. Irregular uncertainties (IU) are firstly removed by exploiting Ellipse fitting model, and then monthly density maps that contain regular uncertainties (RU) are constructed based on metabolic equivalents (METs) of different activity types. Five months of four subjects PA intensity changes using the mobile app tracker Moves [1] and Google Fit app on wearable device Samsung wear S2 are carried out from a mobile personalised healthcare platform MHA [2]. The result indicates that uncertainty of PA intensity monitored by mobile phone is 90% lower than wearable device, where the datasets tend to be further explored by healthcare/fitness studies. Whilst PA activity monitoring by mobile phone is still a challenging issue by far due to much more uncertainties than wearable devices

Computational Approaches for Remote Monitoring of Symptoms and Activities

We now have a unique phenomenon where significant computational power, storage, connectivity, and built-in sensors are carried by many people willingly as part of their life style; two billion people now use smart phones. Unique and innovative solutions using smart phones are motivated by rising health care cost in both the developed and developing worlds. In this work, development of a methodology for building a remote symptom monitoring system for rural people in developing countries has been explored. Design, development, deployment, and evaluation of e-ESAS is described. The system’s performance was studied by analyzing feedback from users. A smart phone based prototype activity detection system that can detect basic human activities for monitoring by remote observers was developed and explored in this study. The majority voting fusion technique, along with decision tree learners were used to classify eight activities in a multi-sensor framework. This multimodal approach was examined in details and evaluated for both single and multi-subject cases. Time-delay embedding with expectation-maximization for Gaussian Mixture Model was explored as a way of developing activity detection system using reduced number of sensors, leading to a lower computational cost algorithm. The systems and algorithms developed in this work focus on means for remote monitoring using smart phones. The smart phone based remote symptom monitoring system called e-ESAS serves as a working tool to monitor essential symptoms of patients with breast cancer by doctors. The activity detection system allows a remote observer to monitor basic human activities. For the activity detection system, the majority voting fusion technique in multi-sensor architecture is evaluated for eight activities in both single and multiple subjects cases. Time-delay embedding with expectation-maximization algorithm for Gaussian Mixture Model was studied using data from multiple single sensor cases

Computational Approaches for Remote Monitoring of Symptoms and Activities

We now have a unique phenomenon where significant computational power, storage, connectivity, and built-in sensors are carried by many people willingly as part of their life style; two billion people now use smart phones. Unique and innovative solutions using smart phones are motivated by rising health care cost in both the developed and developing worlds. In this work, development of a methodology for building a remote symptom monitoring system for rural people in developing countries has been explored. Design, development, deployment, and evaluation of e-ESAS is described. The system’s performance was studied by analyzing feedback from users. A smart phone based prototype activity detection system that can detect basic human activities for monitoring by remote observers was developed and explored in this study. The majority voting fusion technique, along with decision tree learners were used to classify eight activities in a multi-sensor framework. This multimodal approach was examined in details and evaluated for both single and multi-subject cases. Time-delay embedding with expectation-maximization for Gaussian Mixture Model was explored as a way of developing activity detection system using reduced number of sensors, leading to a lower computational cost algorithm. The systems and algorithms developed in this work focus on means for remote monitoring using smart phones. The smart phone based remote symptom monitoring system called e-ESAS serves as a working tool to monitor essential symptoms of patients with breast cancer by doctors. The activity detection system allows a remote observer to monitor basic human activities. For the activity detection system, the majority voting fusion technique in multi-sensor architecture is evaluated for eight activities in both single and multiple subjects cases. Time-delay embedding with expectation-maximization algorithm for Gaussian Mixture Model was studied using data from multiple single sensor cases

Smartphone based human activity prediction

Tese de Mestrado Integrado. Bioengenharia. Faculdade de Engenharia. Universidade do Porto. 201

Towards Clustering of Mobile and Smartwatch Accelerometer Data for Physical Activity Recognition

Mobile and wearable devices now have a greater capability of sensing human activity ubiquitously and unobtrusively through advancements in miniaturization and sensing abilities. However, outstanding issues remain around the energy restrictions of these devices when processing large sets of data. This paper presents our approach that uses feature selection to refine the clustering of accelerometer data to detect physical activity. This also has a positive effect on the computational burden that is associated with processing large sets of data, as energy efficiency and resource use is decreased because less data is processed by the clustering algorithms. Raw accelerometer data, obtained from smartphones and smartwatches, have been preprocessed to extract both time and frequency domain features. Principle component analysis feature selection (PCAFS) and correlation feature selection (CFS) have been used to remove redundant features. The reduced feature sets have then been evaluated against three widely used clustering algorithms, including hierarchical clustering analysis (HCA), k-means, and density-based spatial clustering of applications with noise (DBSCAN). Using the reduced feature sets resulted in improved separability, reduced uncertainty, and improved efficiency compared with the baseline, which utilized all features. Overall, the CFS approach in conjunction with HCA produced higher Dunn Index results of 9.7001 for the phone and 5.1438 for the watch features, which is an improvement over the baseline. The results of this comparative study of feature selection and clustering, with the specific algorithms used, has not been performed previously and provides an optimistic and usable approach to recognize activities using either a smartphone or smartwatch

Energy expenditure prediction via a footwear-based physical activity monitor: accuracy and comparison to other devices

2011 Summer.Includes bibliographical references.Accurately estimating free-living energy expenditure (EE) is important for monitoring or altering energy balance and quantifying levels of physical activity. The use of accelerometers to monitor physical activity and estimate physical activity EE is common in both research and consumer settings. Recent advances in physical activity monitors include the ability to identify specific activities (e.g. stand vs. walk) which has resulted in improved EE estimation accuracy. Recently, a multi]sensor footwear-based physical activity monitor that is capable of achieving 98% activity identification accuracy has been developed. However, no study has compared the EE estimation accuracy for this monitor and compared this accuracy to other similar devices. PURPOSE: To determine the accuracy of physical activity EE estimation of a footwear-based physical activity monitor that uses an embedded accelerometer and insole pressure sensors and to compare this accuracy against a variety of research and consumer physical activity monitors. METHODS: Nineteen adults (10 male, 9 female), mass: 75.14 (17.1) kg, BMI: 25.07(4.6) kg/m2 (mean (SD)), completed a four hour stay in a room calorimeter. Participants wore a footwear-based physical activity monitor, as well as three physical activity monitoring devices used in research: hip]mounted Actical and Actigraph accelerometers and a multi-accelerometer IDEEA device with sensors secured to the limb and chest. In addition, participants wore two consumer devices: Philips DirectLife and Fitbit. Each individual performed a series of randomly assigned and ordered postures/activities including lying, sitting (quietly and using a computer), standing, walking, stepping, cycling, sweeping, as well as a period of self-selected activities. We developed branched (i.e. activity specific) linear regression models to estimate EE from the footwear-based device, and we used the manufacturer's software to estimate EE for all other devices. RESULTS: The shoe-based device was not significantly different than the mean measured EE (476(20) vs. 478(18) kcal) (Mean(SE)), respectively, and had the lowest root mean square error (RMSE) by two]fold (29.6 kcal (6.19%)). The IDEEA (445(23) kcal) and DirecLlife (449(13) kcal) estimates of EE were also not different than the measured EE. The Actigraph, Fitbit and Actical devices significantly underestimated EE (339 (19) kcal, 363(18) kcal and 383(17) kcal, respectively (p<.05)). Root mean square errors were 62.1 kcal (14%), 88.2 kcal(18%), 122.2 kcal (27%), 130.1 kcal (26%), and 143.2 kcal (28%) for DirectLife, IDEEA, Actigraph, Actical and Fitbit respectively. CONCLUSIONS: The shoe based physical activity monitor was able to accurately estimate EE. The research and consumer physical activity monitors tested have a wide range of accuracy when estimating EE. Given the similar hardware of these devices, these results suggest that the algorithms used to estimate EE are primarily responsible for their accuracy, particularly the ability of the shoe-based device to estimate EE based on activity classifications

MonitorMe: sistema de reconhecimento de atividades baseado em Android

Mestrado em Engenharia de Computadores e TelemáticaA monitorização de uma pessoa pode ser importante em várias situações do dia-a-dia. Um modo de monitorização é a identificação de atividades realizadas. Atualmente, vários sensores potencialmente úteis para o reconhecimento de atividades, são integrados em dispositivos móveis, o que os torna particularmente interessantes para este tipo de monitorização. Uma forma complementar de monitorização é a utilização da gravação de um vídeo do ambiente que rodeia a pessoa a ser monitorizada. No entanto, dado o tamanho elevado dos vídeos para transmissão por canais sem fios ou mesmo para gravação no dispositivo, torna-se necessário atuar na compressão e redução da informação associada. Uma forma de o conseguir é adaptar a cadência de imagens adquiridas à velocidade da pessoa que está ser monitorizada. Nesta dissertação é proposto um sistema de monitorização online, chamado MonitorMe, que permite o reconhecimento de atividades e a gravação de um vídeo do ambiente envolvente de uma pessoa. Este sistema inclui um smartphone Android, mantido num bolso de camisa, e um módulo MARG (Magnetic, Angular Rate and Gravity), colocado num bolso das calças. Foi desenvolvida uma aplicação para o smartphone, que obtém dados dos sensores integrados em ambos os dispositivos para a realização do reconhecimento online de 6 atividades diferentes (em pé, sentado, deitado, andar, correr e queda). Este reconhecimento é conseguido utilizando um algoritmo de baixo custo computacional, cujo desenvolvimento teve em consideração as restrições relativas à capacidade de processamento e à duração da bateria dos telemóveis. Paralelamente ao reconhecimento de atividades, a câmara do smartphone captura imagens com uma cadência que varia com a velocidade do utilizador, esta última estimada a partir dos dados dos sensores processados para o reconhecimento de atividades. Demonstra-se assim a possibilidade de, com baixo custo computacional, diminuir a largura de banda de transmissão ou o armazenamento no dispositivo móvel. O sistema MonitorMe foi treinado e depois testado com dados obtidos em duas experiências envolvendo 10 pessoas, num total de 440 eventos diferentes com uma duração total de 45 minutos (2/3 usados para treino e 1/3 para teste). Os resultados globais obtidos mostraram uma sensibilidade superior a 93% e uma especificidade superior a 98% para o reconhecimento de atividades, e um erro médio relativo de 8.6% para a estimativa de velocidade.The monitoring of a given person can be important in different day-to-day scenarios. Monitoring can be performed by detecting activities while being carried out. Presently, various sensors with potential for activity recognition are being included in mobile devices, so they are particularly interesting for this type of monitoring. A complementary way of monitoring consists in the use of a video recording of the subject’s surrounding environment. However, given the large size of the videos for transmission through wireless links or even for storage in the device, it is necessary to compress and reduce the corresponding information. This can be achieved by adapting the frame rate of the captured images to the speed of the user being monitored. In this dissertation an online monitoring system, MonitorMe, which performs activity recognition and video recording of the surrounding environment of a subject, is proposed. This system includes an Android smartphone, inserted in a shirt pocket, and an MARG (Magnetic, Angular Rate and Gravity) module, placed in a pants pocket. A smartphone application was developed, which collects data from the sensors integrated in both devices to perform the online recognition of 6 different activities (standing, sitting, lying, walking, running and fall). This was achieved by using an algorithm of low computational cost, which took into account the existing restrictions regarding processing power and battery life of mobile phones. In parallel with activity recognition, the smartphone camera captures images with a frame rate that varies with the user speed, the latter estimated from sensor data processed for activity recognition. This demonstrates the possibility of reducing the required transmission bandwidth or the storage in the mobile device, with a low computational cost. The MonitorMe system was trained and then tested using data collected in two experiments with a participation of 10 subjects, which resulted in a total of 440 different events with a total duration of 45 minutes (2/3 used for training and 1/3 for testing). The overall results have shown a sensibility greater than 93% and a specificity greater than 98% for activity recognition, and an average relative error of 8.6% for speed estimation

Sampling frequency optimization and training model selection for physical activity classification with single triaxial accelerometer

Ambulatory monitoring system with accelerometers can provide a reliable, continuous, unsupervised and objective monitoring of human physical activities. The system can in many cases recognize the type of activity being performed, and calculate the duration and intensity. This kind of information can be utilized to help people to follow up their physical activities and remind people to be more active, because physical inactivity can cause some health problems. However, especially for mobile devices continuous sam-pling, signal processing and activity recognition rapidly depletes the system’s energy, which is a critically constrained resource. In this thesis work, several methods for reducing energy consumption in physical activi-ty recognition were reviewed and discussed, i.e., 1) reducing the number of sensors used; 2) selecting low power sensors; 3) reducing the number of axes; 4) decreasing the sampling frequency; 5) adopting an adaptive sampling strategy. In this thesis, a single tri-axial accelerometer was utilized for sensing the accelerations, and sampling frequency was optimized in order to lower the energy consumption. The physical activity recognition was performed with different sampling frequencies and training strategies, with the target to reach good classification accuracies and low energy consumption. Based on the obtained classification results, several conclusions were drawn. Firstly, personal models did not always achieve better classification accuracies over impersonal and hybrid models. However, personal models performed much better for some activi-ties, e.g., biking, lying, and rowing. Secondly, there was no uniform optimal sampling frequency for all activities. Sampling frequencies no larger than 10 Hz were enough to classify all activities. To further optimize the energy consumption, adaptive sampling rate logic was designed and implemented. It adaptively used 1 Hz when sampling the accelerations from lying activity and 10 Hz for other activities. The results showed it worked effectively and efficiently

Data Analysis for Physical Activity Monitoring

Master's thesis in Computer SciencePhysical activity is essential for humans for maintaining a healthy and comfortable lifestyle. With science and technological advancements, there comes various guidelines for the amount of physical activity a person should perform. Monitoring the physical activity enables us to follow those guidelines and be aware of own activity. Wearable computing is allowing us to track and monitor our own performed physical activities by mostly intrinsic (minimal) interaction. Physical activity monitoring is an emerging research area in wearable computing. Our thesis is about identifying and classifying which activity is being performed. We have used various classifiers and evaluation metrics to validate our classifier models