Summary of the Sussex-Huawei locomotion-transportation recognition challenge 2020

In this paper we summarize the contributions of participants to the third Sussex-Huawei Locomotion-Transportation (SHL) Recognition Challenge organized at the HASCAWorkshop of UbiComp/ISWC 2020. The goal of this machine learning/data science challenge is to recognize eight locomotion and transportation activities (Still, Walk, Run, Bike, Bus, Car, Train, Subway) from the inertial sensor data of a smartphone in a user-independent manner with an unknown target phone position. The training data of a “train” user is available from smartphones placed at four body positions (Hand, Torso, Bag and Hips). The testing data originates from “test” users with a smartphone placed at one, but unknown, body position. We introduce the dataset used in the challenge and the protocol of the competition. We present a meta-analysis of the contributions from 15 submissions, their approaches, the software tools used, computational cost and the achieved results. Overall, one submission achieved F1 scores above 80%, three with F1 scores between 70% and 80%, seven between 50% and 70%, and four below 50%, with a latency of maximum of 5 seconds

Summary of the Sussex-Huawei Locomotion-Transportation Recognition Challenge 2019

In this paper we summarize the contributions of participants to the third Sussex-Huawei Locomotion-Transportation (SHL) Recognition Challenge organized at the HASCAWorkshop of UbiComp/ISWC 2020. The goal of this machine learning/data science challenge is to recognize eight locomotion and transportation activities (Still, Walk, Run, Bike, Bus, Car, Train, Subway) from the inertial sensor data of a smartphone in a user-independent manner with an unknown target phone position. The training data of a “train” user is available from smartphones placed at four body positions (Hand, Torso, Bag and Hips). The testing data originates from “test” users with a smartphone placed at one, but unknown, body position. We introduce the dataset used in the challenge and the protocol of the competition. We present a meta-analysis of the contributions from 15 submissions, their approaches, the software tools used, computational cost and the achieved results. Overall, one submission achieved F1 scores above 80%, three with F1 scores between 70% and 80%, seven between 50% and 70%, and four below 50%, with a latency of maximum of 5 seconds

Transportation mode recognition fusing wearable motion, sound and vision sensors

We present the first work that investigates the potential of improving the performance of transportation mode recognition through fusing multimodal data from wearable sensors: motion, sound and vision. We first train three independent deep neural network (DNN) classifiers, which work with the three types of sensors, respectively. We then propose two schemes that fuse the classification results from the three mono-modal classifiers. The first scheme makes an ensemble decision with fixed rules including Sum, Product, Majority Voting, and Borda Count. The second scheme is an adaptive fuser built as another classifier (including Naive Bayes, Decision Tree, Random Forest and Neural Network) that learns enhanced predictions by combining the outputs from the three mono-modal classifiers. We verify the advantage of the proposed method with the state-of-the-art Sussex-Huawei Locomotion and Transportation (SHL) dataset recognizing the eight transportation activities: Still, Walk, Run, Bike, Bus, Car, Train and Subway. We achieve F1 scores of 79.4%, 82.1% and 72.8% with the mono-modal motion, sound and vision classifiers, respectively. The F1 score is remarkably improved to 94.5% and 95.5% by the two data fusion schemes, respectively. The recognition performance can be further improved with a post-processing scheme that exploits the temporal continuity of transportation. When assessing generalization of the model to unseen data, we show that while performance is reduced - as expected - for each individual classifier, the benefits of fusion are retained with performance improved by 15 percentage points. Besides the actual performance increase, this work, most importantly, opens up the possibility for dynamically fusing modalities to achieve distinct power-performance trade-off at run time

Sampling Strategies for Tackling Imbalanced Data in Human Activity Recognition

Human activity recognition (HAR) using wearable sensors is a topic that is being actively researched in machine learning. Smart, sensor-embedded devices, such as smartphones, fitness trackers, or smart watches that collect detailed data on movement, are widely available now. HAR may be applied in areas such as healthcare, physiotherapy, and fitness to assist users of these smart devices in their daily lives. However, one of the main challenges facing HAR, particularly when it is used in supervised learning, is how balanced data may be obtained for algorithm optimisation and testing. Because users engage in some activities more than others, e.g. walking more than running, HAR datasets are typically imbalanced. The lack of dataset representation from minority classes, therefore, hinders the ability of HAR classifiers to sufficiently capture new instances of those activities. Inspired by the concept of data fusion, this thesis will introduce three new hybrid sampling methods. Thus, the diversity of the synthesised samples will be enhanced by combining output from separate sampling methods into three hybrid approaches. The advantage of the hybrid method is that it provides diverse synthetic data that can increase the size of the training data from different sampling approaches. This leads to improvements in the generalisation of a learning activity recognition model. The first strategy, known as the (DBM), combines synthetic minority oversampling techniques (SMOTE) with Random_SMOTE, both of which are built around the k-nearest neighbours algorithm. The second technique, called the noise detection-based method (NDBM), combines Tomek links (SMOTE_Tomeklinks) and the modified synthetic minority oversampling technique (MSMOTE). The third approach, titled the cluster-based method (CBM), combines cluster-based synthetic oversampling (CBSO) and the proximity weighted synthetic oversampling technique (ProWSyn). The performance of the proposed hybrid methods is compared with existing methods using accelerometer data from three commonly used benchmark datasets. The results show that the DBM, NDBM and CBM can significantly reduce the impact of class imbalance and enhance F1 scores of the multilayer perceptron (MLP) by as much as 9 % to 20 % compared with their constituent sampling methods. Also, the Friedman statistical significance test was conducted to compare the effect of the different sampling methods. The test results confirm that the CBM is more effective than the other sampling approaches. This thesis also introduces a method based on the Wasserstein generative adversarial network (WGAN) for generating different types of data on human activity. The WGAN is more stable to train than a generative adversarial network (GAN) and this is due to the use of a stable metric, namely Wasserstein distance, to compare the similarity between the real data distribution with the generated data distribution. WGAN is a deep learning approach, and in contrast to the six existing sampling methods referred to previously, it can operate on raw sensor data as convolutional and recurrent layers can act as feature extractors. WGAN is used to generate raw sensor data to overcome the limitations of the traditional machine learning-based sampling methods that can only operate on extracted features. The synthetic data that is produced by WGAN is then used to oversample the imbalanced training data. This thesis demonstrates that this approach significantly enhances the learning ability of the convolutional neural network(CNN) by as much as 5 % to 6 % from imbalanced human activity datasets. This thesis concludes that the proposed sampling methods based on traditional machine learning are efficient when human activity training data is imbalanced and small. These methods are less complex to implement, require less human activity training data to produce synthetic data and fewer computational resources than the WGAN approach. The proposed WGAN method is effective at producing raw sensor data when a large quantity of human activity training data is available. Additionally, it is time-consuming to optimise the hyperparameters related to the WGAN architecture, which significantly impacts the performance of the method

Transfer Learning in Human Activity Recognition: A Survey

Sensor-based human activity recognition (HAR) has been an active research area, owing to its applications in smart environments, assisted living, fitness, healthcare, etc. Recently, deep learning based end-to-end training has resulted in state-of-the-art performance in domains such as computer vision and natural language, where large amounts of annotated data are available. However, large quantities of annotated data are not available for sensor-based HAR. Moreover, the real-world settings on which the HAR is performed differ in terms of sensor modalities, classification tasks, and target users. To address this problem, transfer learning has been employed extensively. In this survey, we focus on these transfer learning methods in the application domains of smart home and wearables-based HAR. In particular, we provide a problem-solution perspective by categorizing and presenting the works in terms of their contributions and the challenges they address. We also present an updated view of the state-of-the-art for both application domains. Based on our analysis of 205 papers, we highlight the gaps in the literature and provide a roadmap for addressing them. This survey provides a reference to the HAR community, by summarizing the existing works and providing a promising research agenda.Comment: 40 pages, 5 figures, 7 table

Movement recognition from wearable sensors data: power-aware evolutionary training for template matching and data annotation recovery methods

Human activities recognition finds numerous applications for example in sport training, patient rehabilitation, gait analysis and surgical skills evaluation. Wearable sensing and Template Matching Methods (TMMs) offer significant advantages compared to manual assessment methods as well as to more cumbersome camera-based setups and other machine learning (ML) algorithms. TMMs require less data for training than other ML methods, they are low-power and therefore suitable for integration on wearable sensor. They compute a sample-by-sample distance between two time series. When applied to gestures sensors data, this even enables a richer and more movement-specific assessment and feedback. However, TMMs lack of a standard training procedure. In this thesis, we introduce an innovative evolutionary training algorithm for TMMthat not only can maximize recognition performance, but it can also prefer power-minimisation by reducing the TMM’s computational cost with a configurable trade-off. We exhibit that a reduction is even possible without sacrificing recognition performance by exploiting the long-established concept of “time warping”. We demonstrate that our method is suitable for a wide variety of raw data as well as processed, fused and encoded sensor data. We present a new original multi-modal, multi-user dataset of beach volleyball movements that allowed to evaluate our training methods on a real-case of sport training actions. Moreover, the collection of this dataset helped to generate a set of guidelines for the collection of movement data in the wild, using wearable sensors. We introduce a 3D human model that can be animated through inertial wearable sensors data for troubleshooting, movement analysis and privacy-safe annotation of human activities. Finally, through a case study on a dataset of drinking actions, we demonstrate how TMM can improve the quality of a badly annotated but highly valuable dataset

Edistysaskeleita liikkeentunnistuksessa mobiililaitteilla

Motion sensing is one of the most important sensing capabilities of mobile devices, enabling monitoring physical movement of the device and associating the observed motion with predefined activities and physical phenomena. The present thesis is divided into three parts covering different facets of motion sensing techniques. In the first part of this thesis, we present techniques to identify the gravity component within three-dimensional accelerometer measurements. Our technique is particularly effective in the presence of sustained linear acceleration events. Using the estimated gravity component, we also demonstrate how the sensor measurements can be transformed into descriptive motion representations, able to convey information about sustained linear accelerations. To quantify sustained linear acceleration, we propose a set of novel peak features, designed to characterize movement during mechanized transportation. Using the gravity estimation technique and peak features, we proceed to present an accelerometer-based transportation mode detection system able to distinguish between fine-grained automotive modalities. In the second part of the thesis, we present a novel sensor-assisted method, crowd replication, for quantifying usage of a public space. As a key technical contribution within crowd replication, we describe construction and use of pedestrian motion models to accurately track detailed motion information. Fusing the pedestrian models with a positioning system and annotations about visual observations, we generate enriched trajectories able to accurately quantify usage of public spaces. Finally in the third part of the thesis, we present two exemplary mobile applications leveraging motion information. As the first application, we present a persuasive mobile application that uses transportation mode detection to promote sustainable transportation habits. The second application is a collaborative speech monitoring system, where motion information is used to monitor changes in physical configuration of the participating devices.Liikkeen havainnointi ja analysointi ovat keskeisimpiä kontekstitietoisten mobiililaitteiden ominaisuuksia. Tässä väitöskirjassa tarkastellaan kolmea eri liiketunnistuksen osa-aluetta. Väitöskirjan ensimmäinen osa käsittelee liiketunnistuksen menetelmiä erityisesti liikenteen ja ajoneuvojen saralla. Väitöskirja esittelee uusia menetelmiä gravitaatiokomponentin arviointiin tilanteissa, joissa laitteeseen kohdistuu pitkäkestoista lineaarista kiihtyvyyttä. Gravitaatiokomponentin tarkka arvio mahdollistaa ajoneuvon liikkeen erottelun muista laitteeseen kohdistuvista voimista. Menetelmän potentiaalin havainnollistamiseksi työssä esitellään kiihtyvyysanturipohjainen kulkumuototunnistusjärjestelmä, joka perustuu eri kulkumuotojen erotteluun näiden kiihtyvyysprofiilien perusteella. Väitöskirjan toinen osa keskittyy tapoihin mitata ja analysoida julkisten tilojen käyttöä liikkeentunnistuksen avulla. Työssä esitellään menetelmä, jolla kohdealueen käyttöä voidaan arvioida yhdistelemällä suoraa havainnointia ja mobiililaitteilla suoritettua havainnointia. Tämän esitellyn ihmisjoukkojen toisintamiseen (crowd replication) perustuvan menetelmän keskeisin tekninen kontribuutio on liikeantureihin perustuva liikkeenmallinnusmenetelmä, joka mahdollistaa käyttäjän tarkan askelten ja kävelyrytmin tunnistamisen. Yhdistämällä liikemallin tuottama tieto paikannusmenetelmään ja tutkijan omiin havaintoihin väitöskirjassa osoitetaan, kuinka käyttäjän osalta saadaan tallennettua tarkat tiedot hänen aktiviteeteistään ja liikeradoistaan sekä tilan että ajan suhteen. Väitöskirjan kolmannessa ja viimeisessä osassa esitellään kaksi esimerkkisovellusta liikkeentunnistuksen käytöstä mobiililaitteissa. Ensimmäinen näistä sovelluksista pyrkii edistämään ja tukemaan käyttäjää kohti kestäviä liikkumistapoja. Sovelluksen keskeisenä komponenttina toimii automaattinen kulkumuototunnistus, joka seuraa käyttäjän liikkumistottumuksia ja näistä koituvaa hiilidioksidijalanjälkeä. Toinen esiteltävä sovellus on mobiililaitepohjainen, yhteisöllinen puheentunnistus, jossa liikkeentunnistusta käytetään seuraamaan mobiililaiteryhmän fyysisen kokoonpanon pysyvyyttä

Applying Machine Learning Techniques to Improve Safety and Mobility of Urban Transportation Systems Using Infrastructure- and Vehicle-Based Sensors

The importance of sensing technologies in the field of transportation is ever increasing. Rapid improvements of cloud computing, Internet of Vehicles (IoV), and intelligent transport system (ITS) enables fast acquisition of sensor data with immediate processing. Machine learning algorithms provide a way to classify or predict outcomes in a selective and timely fashion. High accuracy and increased volatility are the main features of various learning algorithms. In this dissertation, we aim to use infrastructure- and vehicle-based sensors to improve safety and mobility of urban transportation systems. Smartphone sensors were used in the first study to estimate vehicle trajectory using lane change classification. It addresses the research gap in trajectory estimation since all previous studies focused on estimating trajectories at roadway segments only. Being a mobile application-based system, it can readily be used as on-board unit emulators in vehicles that have little or no connectivity. Secondly, smartphone sensors were also used to identify several transportation modes. While this has been studied extensively in the last decade, our method integrates a data augmentation method to overcome the class imbalance problem. Results show that using a balanced dataset improves the classification accuracy of transportation modes. Thirdly, infrastructure-based sensors like the loop detectors and video detectors were used to predict traffic signal states. This system can aid in resolving the complex signal retiming steps that is conventionally used to improve the performance of an intersection. The methodology was transferred to a different intersection where excellent results were achieved. Fourthly, magnetic vehicle detection system (MVDS) was used to generate traffic patterns in crash and non-crash events. Variational Autoencoder was used for the first time in this study as a data generation tool. The results related to sensitivity and specificity were improved by up to 8% as compared to other state-of-the-art data augmentation methods