Summary of the Sussex-Huawei Locomotion-Transportation Recognition Challenge

In this paper we summarize the contributions of participants to the Sussex-Huawei Transportation-Locomotion (SHL) Recognition Challenge organized at the HASCA Workshop of UbiComp 2018. The SHL challenge is a machine learning and data science competition, which aims to recognize eight transportation activities (Still, Walk, Run, Bike, Bus, Car, Train, Subway) from the inertial and pressure sensor data of a smartphone. We introduce the dataset used in the challenge and the protocol for the competition. We present a meta-analysis of the contributions from 19 submissions, their approaches, the software tools used, computational cost and the achieved results. Overall, two entries achieved F1 scores above 90%, eight with F1 scores between 80% and 90%, and nine between 50% and 80%

Situation inference and context recognition for intelligent mobile sensing applications

The usage of smart devices is an integral element in our daily life. With the richness of data streaming from sensors embedded in these smart devices, the applications of ubiquitous computing are limitless for future intelligent systems. Situation inference is a non-trivial issue in the domain of ubiquitous computing research due to the challenges of mobile sensing in unrestricted environments. There are various advantages to having robust and intelligent situation inference from data streamed by mobile sensors. For instance, we would be able to gain a deeper understanding of human behaviours in certain situations via a mobile sensing paradigm. It can then be used to recommend resources or actions for enhanced cognitive augmentation, such as improved productivity and better human decision making. Sensor data can be streamed continuously from heterogeneous sources with different frequencies in a pervasive sensing environment (e.g., smart home). It is difficult and time-consuming to build a model that is capable of recognising multiple activities. These activities can be performed simultaneously with different granularities. We investigate the separability aspect of multiple activities in time-series data and develop OPTWIN as a technique to determine the optimal time window size to be used in a segmentation process. As a result, this novel technique reduces need for sensitivity analysis, which is an inherently time consuming task. To achieve an effective outcome, OPTWIN leverages multi-objective optimisation by minimising the impurity (the number of overlapped windows of human activity labels on one label space over time series data) while maximising class separability. The next issue is to effectively model and recognise multiple activities based on the user's contexts. Hence, an intelligent system should address the problem of multi-activity and context recognition prior to the situation inference process in mobile sensing applications. The performance of simultaneous recognition of human activities and contexts can be easily affected by the choices of modelling approaches to build an intelligent model. We investigate the associations of these activities and contexts at multiple levels of mobile sensing perspectives to reveal the dependency property in multi-context recognition problem. We design a Mobile Context Recognition System, which incorporates a Context-based Activity Recognition (CBAR) modelling approach to produce effective outcome from both multi-stage and multi-target inference processes to recognise human activities and their contexts simultaneously. Upon our empirical evaluation on real-world datasets, the CBAR modelling approach has significantly improved the overall accuracy of simultaneous inference on transportation mode and human activity of mobile users. The accuracy of activity and context recognition can also be influenced progressively by how reliable user annotations are. Essentially, reliable user annotation is required for activity and context recognition. These annotations are usually acquired during data capture in the world. We research the needs of reducing user burden effectively during mobile sensor data collection, through experience sampling of these annotations in-the-wild. To this end, we design CoAct-nnotate --- a technique that aims to improve the sampling of human activities and contexts by providing accurate annotation prediction and facilitates interactive user feedback acquisition for ubiquitous sensing. CoAct-nnotate incorporates a novel multi-view multi-instance learning mechanism to perform more accurate annotation prediction. It also includes a progressive learning process (i.e., model retraining based on co-training and active learning) to improve its predictive performance over time. Moving beyond context recognition of mobile users, human activities can be related to essential tasks that the users perform in daily life. Conversely, the boundaries between the types of tasks are inherently difficult to establish, as they can be defined differently from the individuals' perspectives. Consequently, we investigate the implication of contextual signals for user tasks in mobile sensing applications. To define the boundary of tasks and hence recognise them, we incorporate such situation inference process (i.e., task recognition) into the proposed Intelligent Task Recognition (ITR) framework to learn users' Cyber-Physical-Social activities from their mobile sensing data. By recognising the engaged tasks accurately at a given time via mobile sensing, an intelligent system can then offer proactive supports to its user to progress and complete their tasks. Finally, for robust and effective learning of mobile sensing data from heterogeneous sources (e.g., Internet-of-Things in a mobile crowdsensing scenario), we investigate the utility of sensor data in provisioning their storage and design QDaS --- an application agnostic framework for quality-driven data summarisation. This allows an effective data summarisation by performing density-based clustering on multivariate time series data from a selected source (i.e., data provider). Thus, the source selection process is determined by the measure of data quality. Nevertheless, this framework allows intelligent systems to retain comparable predictive results by its effective learning on the compact representations of mobile sensing data, while having a higher space saving ratio. This thesis contains novel contributions in terms of the techniques that can be employed for mobile situation inference and context recognition, especially in the domain of ubiquitous computing and intelligent assistive technologies. This research implements and extends the capabilities of machine learning techniques to solve real-world problems on multi-context recognition, mobile data summarisation and situation inference from mobile sensing. We firmly believe that the contributions in this research will help the future study to move forward in building more intelligent systems and applications

Semantic Trajectories:Computing and Understanding Mobility Data

Thanks to the rapid development of mobile sensing technologies (like GPS, GSM, RFID, accelerometer, gyroscope, sound and other sensors in smartphones), the large-scale capture of evolving positioning data (called mobility data or trajectories) generated by moving objects with embedded sensors has become easily feasible, both technically and economically. We have already entered a world full of trajectories. The state-of-the-art on trajectory, either from the moving object database area or in the statistical analysis viewpoint, has built a bunch of sophisticated techniques for trajectory data ad-hoc storage, indexing, querying and mining etc. However, most of these existing methods mainly focus on a spatio-temporal viewpoint of mobility data, which means they analyze only the geometric movement of trajectories (e.g., the raw ‹x, y, t› sequential data) without enough consideration on the high-level semantics that can better understand the underlying meaningful movement behaviors. Addressing this challenging issue for better understanding movement behaviors from the raw mobility data, this doctoral work aims at providing a high-level modeling and computing methodology for semantically abstracting the rapidly increasing mobility data. Therefore, we bring top-down semantic modeling and bottom-up data computing together and establish a new concept called "semantic trajectories" for mobility data representation and understanding. As the main novelty contribution, this thesis provides a rich, holistic, heterogeneous and application-independent methodology for computing semantic trajectories to better understand mobility data at different levels. In details, this methodology is composed of five main parts with dedicated contributions. Semantic Trajectory Modeling. By investigating trajectory modeling requirements to better understand mobility data, this thesis first designs a hybrid spatio-semantic trajectory model that represents mobility with rich data abstraction at different levels, i.e., from the low-level spatio-temporal trajectory to the intermediate-level structured trajectory, and finally to the high-level semantic trajectory. In addition, a semantic based ontological framework has also been designed and applied for querying and reasoning on trajectories. Offline Trajectory Computing. To utilize the hybrid model, the thesis complementarily designs a holistic trajectory computing platform with dedicated algorithms for reconstructing trajectories at different levels. The platform can preprocess collected mobility data (i.e., raw movement tracks like GPS feeds) in terms of data cleaning/compression etc., identify individual trajectories, and segment them into structurally meaningful trajectory episodes. Therefore, this trajectory computing platform can construct spatio-temporal trajectories and structured trajectories from the raw mobility data. Such computing platform is initially designed as an offline solution which is supposed to analyze past trajectories via a batch procedure. Trajectory Semantic Annotation. To achieve the final semantic level for better understanding mobility data, this thesis additionally designs a semantic annotation platform that can enrich trajectories with third party sources that are composed of geographic background information and application domain knowledge, to further infer more meaningful semantic trajectories. Such annotation platform is application-independent that can annotate various trajectories (e.g., mobility data of people, vehicle and animals) with heterogeneous data sources of semantic knowledge (e.g., third party sources in any kind of geometric shapes like point, line and region) that can help trajectory enrichment. Online Trajectory Computing. In addition to the offline trajectory computing for analyzing past trajectories, this thesis also contributes to dealing with ongoing trajectories in terms of real-time trajectory computing from movement data streams. The online trajectory computing platform is capable of providing real-life trajectory data cleaning, compression, and segmentation over streaming movement data. In addition, the online platform explores the functionality of online tagging to achieve fully semantic-aware trajectories and further evaluate trajectory computing in a real-time setting. Mining Trajectories from Multi-Sensors. Previously, the focus is on computing semantic trajectories using single-sensory data (i.e., GPS feeds), where most datasets are from moving objects with wearable GPS-embedded sensors (e.g., mobility data of animal, vehicle and people tracking). In addition, we explore the problem of mining people trajectories using multi-sensory feeds from smartphones (GPS, gyroscope, accelerometer etc). The research results reveal that the combination of two sensors (GPS+accelerometer) can significantly infer a complete life-cycle semantic trajectories of people's daily behaviors, both outdoor movement via GPS and indoor activities via accelerometer

Locomotion Traces Data Mining for Supporting Frail People with Cognitive Impairment

The rapid increase in the senior population is posing serious challenges to national healthcare systems. Hence, innovative tools are needed to early detect health issues, including cognitive decline. Several clinical studies show that it is possible to identify cognitive impairment based on the locomotion patterns of older people. Thus, this thesis at first focused on providing a systematic literature review of locomotion data mining systems for supporting Neuro-Degenerative Diseases (NDD) diagnosis, identifying locomotion anomaly indicators and movement patterns for discovering low-level locomotion indicators, sensor data acquisition, and processing methods, as well as NDD detection algorithms considering their pros and cons. Then, we investigated the use of sensor data and Deep Learning (DL) to recognize abnormal movement patterns in instrumented smart-homes. In order to get rid of the noise introduced by indoor constraints and activity execution, we introduced novel visual feature extraction methods for locomotion data. Our solutions rely on locomotion traces segmentation, image-based extraction of salient features from locomotion segments, and vision-based DL. Furthermore, we proposed a data augmentation strategy to increase the volume of collected data and generalize the solution to different smart-homes with different layouts. We carried out extensive experiments with a large real-world dataset acquired in a smart-home test-bed from older people, including people with cognitive diseases. Experimental comparisons show that our system outperforms state-of-the-art methods

e-mission: an open source, extensible platform for human mobility systems

Transportation is the single largest source of carbon emissions in the US. Decarbonizing it is challenging because it depends on individual behaviors, which in turn, depend on local land use planning. The interdisciplinary field of Computational Mobility, focusing on collecting, analysing and influencing human travel behavior, can frame solutions to this challenge.Innovation flows in interdisciplinary fields are bi-directional. The flow to the domain is focused on building a strong foundation for methodological improvements. As the improvements are deployed, they result in use-inspired computational research. This temporal dependency results in our initial focus on the modularity, accuracy and reproducibility of e-mission, an extensible platform for instrumenting human mobility. This open source platform has a modular architecture that supports power efficient duty cycling using virtual sensors, a read-only data model and a pipeline with novel algorithm adaptations for smartphone sensing.We also perform the first empirical evaluations of smartphone-based platforms in this domain. The architectural evaluation is based on three real world deployments: a classic travel diary, a crowdsourcing initiative, and a behavioral study. The accuracy evaluation is based on an novel procedure that uses artificial trips and multiple parallel phones to mitigate concerns over privacy, context sensitive power consumption and inherent sensing error. Data collected from three artifical timelines was used to evaluate the trajectory, segmentation and classification accuracies vs. power for various configurations.On computational side, challenges derived from the deployments can contribute to ongoing CS research in privacy, trustworthiness, incentivization and decision making. On the mobility side, this enables methodological innovations such as Agile Urban Planning for prototyping infrastructure changes

Crowd simulation and visualization

Large-scale simulation and visualization are essential topics in areas as different as sociology, physics, urbanism, training, entertainment among others. This kind of systems requires a vast computational power and memory resources commonly available in High Performance Computing HPC platforms. Currently, the most potent clusters have heterogeneous architectures with hundreds of thousands and even millions of cores. The industry trends inferred that exascale clusters would have thousands of millions. The technical challenges for simulation and visualization process in the exascale era are intertwined with difficulties in other areas of research, including storage, communication, programming models and hardware. For this reason, it is necessary prototyping, testing, and deployment a variety of approaches to address the technical challenges identified and evaluate the advantages and disadvantages of each proposed solution. The focus of this research is interactive large-scale crowd simulation and visualization. To exploit to the maximum the capacity of the current HPC infrastructure and be prepared to take advantage of the next generation. The project develops a new approach to scale crowd simulation and visualization on heterogeneous computing cluster using a task-based technique. Its main characteristic is hardware agnostic. It abstracts the difficulties that imply the use of heterogeneous architectures like memory management, scheduling, communications, and synchronization — facilitating development, maintenance, and scalability. With the goal of flexibility and take advantage of computing resources as best as possible, the project explores different configurations to connect the simulation with the visualization engine. This kind of system has an essential use in emergencies. Therefore, urban scenes were implemented as realistic as possible; in this way, users will be ready to face real events. Path planning for large-scale crowds is a challenge to solve, due to the inherent dynamism in the scenes and vast search space. A new path-finding algorithm was developed. It has a hierarchical approach which offers different advantages: it divides the search space reducing the problem complexity, it can obtain a partial path instead of wait for the complete one, which allows a character to start moving and compute the rest asynchronously. It can reprocess only a part if necessary with different levels of abstraction. A case study is presented for a crowd simulation in urban scenarios. Geolocated data are used, they were produced by mobile devices to predict individual and crowd behavior and detect abnormal situations in the presence of specific events. It was also address the challenge of combining all these individual’s location with a 3D rendering of the urban environment. The data processing and simulation approach are computationally expensive and time-critical, it relies thus on a hybrid Cloud-HPC architecture to produce an efficient solution. Within the project, new models of behavior based on data analytics were developed. It was developed the infrastructure to be able to consult various data sources such as social networks, government agencies or transport companies such as Uber. Every time there is more geolocation data available and better computation resources which allow performing analysis of greater depth, this lays the foundations to improve the simulation models of current crowds. The use of simulations and their visualization allows to observe and organize the crowds in real time. The analysis before, during and after daily mass events can reduce the risks and associated logistics costs.La simulación y visualización a gran escala son temas esenciales en áreas tan diferentes como la sociología, la física, el urbanismo, la capacitación, el entretenimiento, entre otros. Este tipo de sistemas requiere una gran capacidad de cómputo y recursos de memoria comúnmente disponibles en las plataformas de computo de alto rendimiento. Actualmente, los equipos más potentes tienen arquitecturas heterogéneas con cientos de miles e incluso millones de núcleos. Las tendencias de la industria infieren que los equipos en la era exascale tendran miles de millones. Los desafíos técnicos en el proceso de simulación y visualización en la era exascale se entrelazan con dificultades en otras áreas de investigación, incluidos almacenamiento, comunicación, modelos de programación y hardware. Por esta razón, es necesario crear prototipos, probar y desplegar una variedad de enfoques para abordar los desafíos técnicos identificados y evaluar las ventajas y desventajas de cada solución propuesta. El foco de esta investigación es la visualización y simulación interactiva de multitudes a gran escala. Aprovechar al máximo la capacidad de la infraestructura actual y estar preparado para aprovechar la próxima generación. El proyecto desarrolla un nuevo enfoque para escalar la simulación y visualización de multitudes en un clúster de computo heterogéneo utilizando una técnica basada en tareas. Su principal característica es que es hardware agnóstico. Abstrae las dificultades que implican el uso de arquitecturas heterogéneas como la administración de memoria, las comunicaciones y la sincronización, lo que facilita el desarrollo, el mantenimiento y la escalabilidad. Con el objetivo de flexibilizar y aprovechar los recursos informáticos lo mejor posible, el proyecto explora diferentes configuraciones para conectar la simulación con el motor de visualización. Este tipo de sistemas tienen un uso esencial en emergencias. Por lo tanto, se implementaron escenas urbanas lo más realistas posible, de esta manera los usuarios estarán listos para enfrentar eventos reales. La planificación de caminos para multitudes a gran escala es un desafío a resolver, debido al dinamismo inherente en las escenas y el vasto espacio de búsqueda. Se desarrolló un nuevo algoritmo de búsqueda de caminos. Tiene un enfoque jerárquico que ofrece diferentes ventajas: divide el espacio de búsqueda reduciendo la complejidad del problema, puede obtener una ruta parcial en lugar de esperar a la completa, lo que permite que un personaje comience a moverse y calcule el resto de forma asíncrona, puede reprocesar solo una parte si es necesario con diferentes niveles de abstracción. Se presenta un caso de estudio para una simulación de multitud en escenarios urbanos. Se utilizan datos geolocalizados producidos por dispositivos móviles para predecir el comportamiento individual y público y detectar situaciones anormales en presencia de eventos específicos. También se aborda el desafío de combinar la ubicación de todos estos individuos con una representación 3D del entorno urbano. Dentro del proyecto, se desarrollaron nuevos modelos de comportamiento basados ¿¿en el análisis de datos. Se creo la infraestructura para poder consultar varias fuentes de datos como redes sociales, agencias gubernamentales o empresas de transporte como Uber. Cada vez hay más datos de geolocalización disponibles y mejores recursos de cómputo que permiten realizar un análisis de mayor profundidad, esto sienta las bases para mejorar los modelos de simulación de las multitudes actuales. El uso de simulaciones y su visualización permite observar y organizar las multitudes en tiempo real. El análisis antes, durante y después de eventos multitudinarios diarios puede reducir los riesgos y los costos logísticos asociadosPostprint (published version

Edistysaskeleita liikkeentunnistuksessa mobiililaitteilla

Motion sensing is one of the most important sensing capabilities of mobile devices, enabling monitoring physical movement of the device and associating the observed motion with predefined activities and physical phenomena. The present thesis is divided into three parts covering different facets of motion sensing techniques. In the first part of this thesis, we present techniques to identify the gravity component within three-dimensional accelerometer measurements. Our technique is particularly effective in the presence of sustained linear acceleration events. Using the estimated gravity component, we also demonstrate how the sensor measurements can be transformed into descriptive motion representations, able to convey information about sustained linear accelerations. To quantify sustained linear acceleration, we propose a set of novel peak features, designed to characterize movement during mechanized transportation. Using the gravity estimation technique and peak features, we proceed to present an accelerometer-based transportation mode detection system able to distinguish between fine-grained automotive modalities. In the second part of the thesis, we present a novel sensor-assisted method, crowd replication, for quantifying usage of a public space. As a key technical contribution within crowd replication, we describe construction and use of pedestrian motion models to accurately track detailed motion information. Fusing the pedestrian models with a positioning system and annotations about visual observations, we generate enriched trajectories able to accurately quantify usage of public spaces. Finally in the third part of the thesis, we present two exemplary mobile applications leveraging motion information. As the first application, we present a persuasive mobile application that uses transportation mode detection to promote sustainable transportation habits. The second application is a collaborative speech monitoring system, where motion information is used to monitor changes in physical configuration of the participating devices.Liikkeen havainnointi ja analysointi ovat keskeisimpiä kontekstitietoisten mobiililaitteiden ominaisuuksia. Tässä väitöskirjassa tarkastellaan kolmea eri liiketunnistuksen osa-aluetta. Väitöskirjan ensimmäinen osa käsittelee liiketunnistuksen menetelmiä erityisesti liikenteen ja ajoneuvojen saralla. Väitöskirja esittelee uusia menetelmiä gravitaatiokomponentin arviointiin tilanteissa, joissa laitteeseen kohdistuu pitkäkestoista lineaarista kiihtyvyyttä. Gravitaatiokomponentin tarkka arvio mahdollistaa ajoneuvon liikkeen erottelun muista laitteeseen kohdistuvista voimista. Menetelmän potentiaalin havainnollistamiseksi työssä esitellään kiihtyvyysanturipohjainen kulkumuototunnistusjärjestelmä, joka perustuu eri kulkumuotojen erotteluun näiden kiihtyvyysprofiilien perusteella. Väitöskirjan toinen osa keskittyy tapoihin mitata ja analysoida julkisten tilojen käyttöä liikkeentunnistuksen avulla. Työssä esitellään menetelmä, jolla kohdealueen käyttöä voidaan arvioida yhdistelemällä suoraa havainnointia ja mobiililaitteilla suoritettua havainnointia. Tämän esitellyn ihmisjoukkojen toisintamiseen (crowd replication) perustuvan menetelmän keskeisin tekninen kontribuutio on liikeantureihin perustuva liikkeenmallinnusmenetelmä, joka mahdollistaa käyttäjän tarkan askelten ja kävelyrytmin tunnistamisen. Yhdistämällä liikemallin tuottama tieto paikannusmenetelmään ja tutkijan omiin havaintoihin väitöskirjassa osoitetaan, kuinka käyttäjän osalta saadaan tallennettua tarkat tiedot hänen aktiviteeteistään ja liikeradoistaan sekä tilan että ajan suhteen. Väitöskirjan kolmannessa ja viimeisessä osassa esitellään kaksi esimerkkisovellusta liikkeentunnistuksen käytöstä mobiililaitteissa. Ensimmäinen näistä sovelluksista pyrkii edistämään ja tukemaan käyttäjää kohti kestäviä liikkumistapoja. Sovelluksen keskeisenä komponenttina toimii automaattinen kulkumuototunnistus, joka seuraa käyttäjän liikkumistottumuksia ja näistä koituvaa hiilidioksidijalanjälkeä. Toinen esiteltävä sovellus on mobiililaitepohjainen, yhteisöllinen puheentunnistus, jossa liikkeentunnistusta käytetään seuraamaan mobiililaiteryhmän fyysisen kokoonpanon pysyvyyttä