Ambulatory Monitoring Using Passive RFID Technology

Human activity recognition using wearable sensors is a growing field of study in pervasive computing that forms the basis for ubiquitous applications in areas like health care, manufacturing, human computer interaction and sports. A new generation of passive (batteryless) sensors such as sensor enabled RFID (Radio Frequency Identification) tags are creating new prospects for wearable sensor based applications. As passive sensors are lightweight and small, they can be used for unobtrusive monitoring. Furthermore, these sensors are maintenance free as they require no battery. However, recognising activities from passive sensor enabled RFID tags is challenging due to the sparse and noisy nature of the data streams from these sensors because they need to harvest adequate energy for successful operation. Therefore, within this thesis, we propose methods to recognise activities in real time using passive RFID technology by alleviating the adverse effects of sparsity and noise. We mainly consider ambulatory monitoring to facilitate mitigating falls in hospitals and older care settings as our application context. Specifically, three aspects are considered: i) data acquisition from sensor enabled RFID tags; ii) monitoring ambulatory movements using passive sensor enabled RFID tags to recognise activities leading to falls; and iii) detecting falls using a dense deployment of passive RFID tags. A generic middleware architecture and a generic tag ID format to embed sensor data and uniquely identify tag capabilities are proposed to acquire sensor data from passive sensor enabled RFID tags. The characteristics of this middleware are established using experiments with RFID readers and an example application scenario. In the context of ambulatory monitoring using passive sensor enabled RFID tags, first, an algorithm to facilitate the online interpolation of sparse accelerometer data from passive sensor enabled RFID tags is proposed followed by an investigation of features for activity recognition. Secondly, two data stream segmentation methods are proposed that can segment the data stream on possible activity boundaries to mitigate the adverse effects posed by data stream sparsity on segmentation. Thirdly, an algorithm to model the sequential nature considering previous sensor observations for a given time and their class labels to classify a sparse data stream in real time is proposed. Finally, a classification algorithm based on structured prediction is proposed to both segment and classify the sensor data stream simultaneously. The proposed methods are evaluated using four datasets that have been collected from a passive sensor enabled RFID tag with an accelerometer and successful monitoring of ambulatory movements is demonstrated to be possible by employing innovative data stream processing methods, based on machine learning. In order to detect falls, particularly long lie situation, using a dense deployment of passive RFID tags embedded in a carpet, an efficient and scalable machine learning based algorithm is proposed. This algorithm relies only on binary tag observation information. First, it identifies possible fall locations using heuristics and then the falls are identified using machine learning from features extracted considering possible fall locations alone. From an evaluation, it is demonstrated that the proposed algorithm could successfully identify falls in real time.Thesis (Ph.D.) -- University of Adelaide, School of Computer Science, 201

Deep Learning Methods for Human Activity Recognition using Wearables

Wearable sensors provide an infrastructure-less multi-modal sensing method. Current trends point to a pervasive integration of wearables into our lives with these devices providing the basis for wellness and healthcare applications across rehabilitation, caring for a growing older population, and improving human performance. Fundamental to these applications is our ability to automatically and accurately recognise human activities from often tiny sensors embedded in wearables. In this dissertation, we consider the problem of human activity recognition (HAR) using multi-channel time-series data captured by wearable sensors. Our collective know-how regarding the solution of HAR problems with wearables has progressed immensely through the use of deep learning paradigms. Nevertheless, this field still faces unique methodological challenges. As such, this dissertation focuses on developing end-to-end deep learning frameworks to promote HAR application opportunities using wearable sensor technologies and to mitigate specific associated challenges. In our efforts, the investigated problems cover a diverse range of HAR challenges and spans from fully supervised to unsupervised problem domains. In order to enhance automatic feature extraction from multi-channel time-series data for HAR, the problem of learning enriched and highly discriminative activity feature representations with deep neural networks is considered. Accordingly, novel end-to-end network elements are designed which: (a) exploit the latent relationships between multi-channel sensor modalities and specific activities, (b) employ effective regularisation through data-agnostic augmentation for multi-modal sensor data streams, and (c) incorporate optimization objectives to encourage minimal intra-class representation differences, while maximising inter-class differences to achieve more discriminative features. In order to promote new opportunities in HAR with emerging battery-less sensing platforms, the problem of learning from irregularly sampled and temporally sparse readings captured by passive sensing modalities is considered. For the first time, an efficient set-based deep learning framework is developed to address the problem. This framework is able to learn directly from the generated data, bypassing the need for the conventional interpolation pre-processing stage. In order to address the multi-class window problem and create potential solutions for the challenging task of concurrent human activity recognition, the problem of enabling simultaneous prediction of multiple activities for sensory segments is considered. As such, the flexibility provided by the emerging set learning concepts is further leveraged to introduce a novel formulation of HAR. This formulation treats HAR as a set prediction problem and elegantly caters for segments carrying sensor data from multiple activities. To address this set prediction problem, a unified deep HAR architecture is designed that: (a) incorporates a set objective to learn mappings from raw input sensory segments to target activity sets, and (b) precedes the supervised learning phase with unsupervised parameter pre-training to exploit unlabelled data for better generalisation performance. In order to leverage the easily accessible unlabelled activity data-streams to serve downstream classification tasks, the problem of unsupervised representation learning from multi-channel time-series data is considered. For the first time, a novel recurrent generative adversarial (GAN) framework is developed that explores the GAN’s latent feature space to extract highly discriminating activity features in an unsupervised fashion. The superiority of the learned representations is substantiated by their ability to outperform the de facto unsupervised approaches based on autoencoder frameworks. At the same time, they rival the recognition performance of fully supervised trained models on downstream classification benchmarks. In recognition of the scarcity of large-scale annotated sensor datasets and the tediousness of collecting additional labelled data in this domain, the hitherto unexplored problem of end-to-end clustering of human activities from unlabelled wearable data is considered. To address this problem, a first study is presented for the purpose of developing a stand-alone deep learning paradigm to discover semantically meaningful clusters of human actions. In particular, the paradigm is intended to: (a) leverage the inherently sequential nature of sensory data, (b) exploit self-supervision from reconstruction and future prediction tasks, and (c) incorporate clustering-oriented objectives to promote the formation of highly discriminative activity clusters. The systematic investigations in this study create new opportunities for HAR to learn human activities using unlabelled data that can be conveniently and cheaply collected from wearables.Thesis (Ph.D.) -- University of Adelaide, School of Computer Science, 202

Energy-efficient Continuous Context Sensing on Mobile Phones

With the ever increasing adoption of smartphones worldwide, researchers have found the perfect sensor platform to perform context-based research and to prepare for context-based services to be also deployed for the end-users. However, continuous context sensing imposes a considerable challenge in balancing the energy consumption of the sensors, the accuracy of the recognized context and its latency. After outlining the common characteristics of continuous sensing systems, we present a detailed overview of the state of the art, from sensors sub-systems to context inference algorithms. Then, we present the three main contribution of this thesis. The first approach we present is based on the use of local communications to exchange sensing information with neighboring devices. As proximity, location and environmental information can be obtained from nearby smartphones, we design a protocol for synchronizing the exchanges and fairly distribute the sensing tasks. We show both theoretically and experimentally the reduction in energy needed when the devices can collaborate. The second approach focuses on the way to schedule mobile sensors, optimizing for both the accuracy and energy needs. We formulate the optimal sensing problem as a decision problem and propose a two-tier framework for approximating its solution. The first tier is responsible for segmenting the sensor measurement time series, by fitting various models. The second tier takes care of estimating the optimal sampling, selecting the measurements that contributes the most to the model accuracy. We provide near-optimal heuristics for both tiers and evaluate their performances using environmental sensor data. In the third approach we propose an online algorithm that identifies repeated patterns in time series and produces a compressed symbolic stream. The first symbolic transformation is based on clustering with the raw sensor data. Whereas the next iterations encode repetitive sequences of symbols into new symbols. We define also a metric to evaluate the symbolization methods with regard to their capacity at preserving the systems' states. We also show that the output of symbols can be used directly for various data mining tasks, such as classification or forecasting, without impacting much the accuracy, but greatly reducing the complexity and running time. In addition, we also present an example of application, assessing the user's exposure to air pollutants, which demonstrates the many opportunities to enhance contextual information when fusing sensor data from different sources. On one side we gather fine grained air quality information from mobile sensor deployments and aggregate them with an interpolation model. And, on the other side, we continuously capture the user's context, including location, activity and surrounding air quality. We also present the various models used for fusing all these information in order to produce the exposure estimation

Data-Driven Simulation Modeling of Construction and Infrastructure Operations Using Process Knowledge Discovery

Within the architecture, engineering, and construction (AEC) domain, simulation modeling is mainly used to facilitate decision-making by enabling the assessment of different operational plans and resource arrangements, that are otherwise difficult (if not impossible), expensive, or time consuming to be evaluated in real world settings. The accuracy of such models directly affects their reliability to serve as a basis for important decisions such as project completion time estimation and resource allocation. Compared to other industries, this is particularly important in construction and infrastructure projects due to the high resource costs and the societal impacts of these projects. Discrete event simulation (DES) is a decision making tool that can benefit the process of design, control, and management of construction operations. Despite recent advancements, most DES models used in construction are created during the early planning and design stage when the lack of factual information from the project prohibits the use of realistic data in simulation modeling. The resulting models, therefore, are often built using rigid (subjective) assumptions and design parameters (e.g. precedence logic, activity durations). In all such cases and in the absence of an inclusive methodology to incorporate real field data as the project evolves, modelers rely on information from previous projects (a.k.a. secondary data), expert judgments, and subjective assumptions to generate simulations to predict future performance. These and similar shortcomings have to a large extent limited the use of traditional DES tools to preliminary studies and long-term planning of construction projects. In the realm of the business process management, process mining as a relatively new research domain seeks to automatically discover a process model by observing activity records and extracting information about processes. The research presented in this Ph.D. Dissertation was in part inspired by the prospect of construction process mining using sensory data collected from field agents. This enabled the extraction of operational knowledge necessary to generate and maintain the fidelity of simulation models. A preliminary study was conducted to demonstrate the feasibility and applicability of data-driven knowledge-based simulation modeling with focus on data collection using wireless sensor network (WSN) and rule-based taxonomy of activities. The resulting knowledge-based simulation models performed very well in properly predicting key performance measures of real construction systems. Next, a pervasive mobile data collection and mining technique was adopted and an activity recognition framework for construction equipment and worker tasks was developed. Data was collected using smartphone accelerometers and gyroscopes from construction entities to generate significant statistical time- and frequency-domain features. The extracted features served as the input of different types of machine learning algorithms that were applied to various construction activities. The trained predictive algorithms were then used to extract activity durations and calculate probability distributions to be fused into corresponding DES models. Results indicated that the generated data-driven knowledge-based simulation models outperform static models created based upon engineering assumptions and estimations with regard to compatibility of performance measure outputs to reality

Energy adaptive buildings:From sensor data to being aware of users

Energie besparen is fundamenteel voor het realiseren van een duurzame energievoorziening. Het besparen van energie draagt bij aan milieudoelstellingen, verbetert de zakelijke positie van landen, en levert werkgelegenheid. Er zijn tal van mogelijkheden voor het behalen van aanzienlijke energiebesparingen in gebouwen gezien individuen en bedrijven gebaat zijn bij energiebesparingen en daardoor zelf de verantwoordelijkheid nemen. Het is bewezen dat het gedrag van gebouwgebruikers een grote impact heeft op de verwarming en ventilatie van ruimtes, en op het energieverbruik van verlichting en huishoudelijke apparaten. Huidige gebouwautomatiseringssystemen kunnen niet overweg met veranderingen in het gedrag van gebruikers en zijn daardoor niet in staat om het energieverbruik terug te dringen met behoud van gebruikerscomfort. Mijn promotieonderzoek wordt gedreven door het doel om een dergelijk energy adaptive building te realiseren dat intelligent systemen aanstuurt en zich aanpast aan de gebruiker en gebruikersactiviteiten door deze te leren, terwijl energieverspilling wordt teruggedrongen. Mijn focus ligt op het ontwikkelen van een framework, beginnende bij de hardware infrastructuur voor sensoren en actuatoren, het verwerken en analyseren van de sensordata, en de nodige informatie over de omgeving en gebruikersactiviteiten verkrijgen zodat het gebouw aangestuurd kan worden. Onze oplossing kan 35% besparen op het totale energieverbruik van een gebouw. Als een succesverhaal, besparen de software systemen zelfs 80% op het energieverbruik van de verlichting in het restaurant van de Bernoulliborg. Wij commercialiseren de resultaten verkregen in ons onderzoek door het oprichten van de start-up SustainableBuildings, een spin-off bedrijf van onze universiteit, om onze oplossing aan te bieden aan kantoorgebouwen.Saving energy is the foundation for achieving a sustainable energy supply. Saving energy contributes to environmental objectives, improves the competitiveness of a country’s businesses, and boosts employment. There are numerous opportunities for achieving significant energy savings in buildings since individuals and businesses have an interest themselves in saving energy and will shoulder the responsibility for doing so.Occupant behaviour has shown to have large impact on space heating and cooling demand, energy consumption of lighting and appliances. Current building automation systems are unable to cope with changes caused by occupants’ behaviour and interaction with the environment, therefore they fail to reduce unnecessary energy consumption while preserving user comfort.My PhD research is driven by the aim of realising such energy adaptive buildings that facilitate intelligent control, that learn and adapt to the building users and their activities, while reducing energy waste. My particular focus is on a framework, going from the hardware infrastructure for sensing and actuating, to processing and analysing sensor data, providing necessary information about the environment and occupants’ activities for the system to produce adaptive control strategies, regulating the environment accordingly.Our solution can save 35% of energy for a single building. As a success story, the software system saves 80 percent on energy spent for lighting in the restaurant of the Bernoulliborg.We are commercialising the results of our research by creating the SustainableBuildings start-up, a spin-off from our university, to offer the solutions to non-residential buildings, first in the Netherlands, and later extending wider