Device-free indoor localisation with non-wireless sensing techniques : a thesis by publications presented in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Electronics and Computer Engineering, Massey University, Albany, New Zealand

Global Navigation Satellite Systems provide accurate and reliable outdoor positioning to support a large number of applications across many sectors. Unfortunately, such systems do not operate reliably inside buildings due to the signal degradation caused by the absence of a clear line of sight with the satellites. The past two decades have therefore seen intensive research into the development of Indoor Positioning System (IPS). While considerable progress has been made in the indoor localisation discipline, there is still no widely adopted solution. The proliferation of Internet of Things (IoT) devices within the modern built environment provides an opportunity to localise human subjects by utilising such ubiquitous networked devices. This thesis presents the development, implementation and evaluation of several passive indoor positioning systems using ambient Visible Light Positioning (VLP), capacitive-flooring, and thermopile sensors (low-resolution thermal cameras). These systems position the human subject in a device-free manner (i.e., the subject is not required to be instrumented). The developed systems improve upon the state-of-the-art solutions by offering superior position accuracy whilst also using more robust and generalised test setups. The developed passive VLP system is one of the first reported solutions making use of ambient light to position a moving human subject. The capacitive-floor based system improves upon the accuracy of existing flooring solutions as well as demonstrates the potential for automated fall detection. The system also requires very little calibration, i.e., variations of the environment or subject have very little impact upon it. The thermopile positioning system is also shown to be robust to changes in the environment and subjects. Improvements are made over the current literature by testing across multiple environments and subjects whilst using a robust ground truth system. Finally, advanced machine learning methods were implemented and benchmarked against a thermopile dataset which has been made available for other researchers to use

Privacy-preserving human mobility and activity modelling

The exponential proliferation of digital trends and worldwide responses to the COVID-19 pandemic thrust the world into digitalization and interconnectedness, pushing increasingly new technologies/devices/applications into the market. More and more intimate data of users are collected for positive analysis purposes of improving living well-being but shared with/without the user's consent, emphasizing the importance of making human mobility and activity models inclusive, private, and fair. In this thesis, I develop and implement advanced methods/algorithms to model human mobility and activity in terms of temporal-context dynamics, multi-occupancy impacts, privacy protection, and fair analysis. The following research questions have been thoroughly investigated: i) whether the temporal information integrated into the deep learning networks can improve the prediction accuracy in both predicting the next activity and its timing; ii) how is the trade-off between cost and performance when optimizing the sensor network for multiple-occupancy smart homes; iii) whether the malicious purposes such as user re-identification in human mobility modelling could be mitigated by adversarial learning; iv) whether the fairness implications of mobility models and whether privacy-preserving techniques perform equally for different groups of users. To answer these research questions, I develop different architectures to model human activity and mobility. I first clarify the temporal-context dynamics in human activity modelling and achieve better prediction accuracy by appropriately using the temporal information. I then design a framework MoSen to simulate the interaction dynamics among residents and intelligent environments and generate an effective sensor network strategy. To relieve users' privacy concerns, I design Mo-PAE and show that the privacy of mobility traces attains decent protection at the marginal utility cost. Last but not least, I investigate the relations between fairness and privacy and conclude that while the privacy-aware model guarantees group fairness, it violates the individual fairness criteria.Open Acces

VGAN-Based Image Representation Learning for Privacy-Preserving Facial Expression Recognition

Reliable facial expression recognition plays a critical role in human-machine interactions. However, most of the facial expression analysis methodologies proposed to date pay little or no attention to the protection of a user's privacy. In this paper, we propose a Privacy-Preserving Representation-Learning Variational Generative Adversarial Network (PPRL-VGAN) to learn an image representation that is explicitly disentangled from the identity information. At the same time, this representation is discriminative from the standpoint of facial expression recognition and generative as it allows expression-equivalent face image synthesis. We evaluate the proposed model on two public datasets under various threat scenarios. Quantitative and qualitative results demonstrate that our approach strikes a balance between the preservation of privacy and data utility. We further demonstrate that our model can be effectively applied to other tasks such as expression morphing and image completion

APPLICATIONS OF MACHINE LEARNING AND COMPUTER VISION FOR SMART INFRASTRUCTURE MANAGEMENT IN CIVIL ENGINEERING

Machine Learning and Computer Vision are the two technologies that have innovative applications in diverse fields, including engineering, medicines, agriculture, astronomy, sports, education etc. The idea of enabling machines to make human like decisions is not a recent one. It dates to the early 1900s when analogies were drawn out between neurons in a human brain and capability of a machine to function like humans. However, major advances in the specifics of this theory were not until 1950s when the first experiments were conducted to determine if machines can support artificial intelligence. As computation powers increased, in the form of parallel computing and GPU computing, the time required for training the algorithms decreased significantly. Machine Learning is now used in almost every day to day activities. This research demonstrates the use of machine learning and computer vision for smart infrastructure management. This research’s contribution includes two case studies – a) Occupancy detection using vibration sensors and machine learning and b) Traffic detection, tracking, classification and counting on Memorial Bridge in Portsmouth, NH using computer vision and machine learning. Each case study, includes controlled experiments with a verification data set. Both the studies yielded results that validated the approach of using machine learning and computer vision. Both case studies present a scenario where in machine learning is applied to a civil engineering challenge to create a more objective basis for decision-making. This work also includes a summary of the current state-of-the -practice of machine learning in Civil Engineering and the suggested steps to advance its application in civil engineering based on this research in order to use the technology more effectively

Data Processing for Device-Free Fine-Grained Occupancy Sensing Using Infrared Sensors

Fine-grained occupancy information plays an essential role for various emerging applications in smart homes, such as personalized thermal comfort control and human behavior analysis. Existing occupancy sensors, such as passive infrared (PIR) sensors generally provide limited coarse information such as motion. However, the detection of fine-grained occupancy information such as stationary presence, posture, identification, and activity tracking can be enabled with the advance of sensor technologies. Among these, infrared sensing is a low-cost, device-free, and privacy-preserving choice that detects the fluctuation (PIR sensors) or the thermal profiles (thermopile array sensors) from objects' infrared radiation. This work focuses on developing data processing models towards fine-grained occupancy sensing using the synchronized low-energy electronically chopped PIR (SLEEPIR) sensor or the thermopile array sensors. The main contributions of this dissertation include: (1) creating and validating the mathematical model of the SLEEPIR sensor output towards stationary occupancy detection; (2) developing the SLEEPIR detection algorithm using statistical features and long-short term memory (LSTM) deep learning; (3) building machine learning framework for posture detection and activity tracking using thermopile array sensors; and (4) creating convolutional neural network (CNN) models for facing direction detection and identification using thermopile array sensors

Privacy-Preserving Non-Wearable Occupancy Monitoring System Exploiting Wi-Fi Imaging for Next-Generation Body Centric Communication

Nano-scaled structures, wireless sensing, wearable devices, and wireless communications systems are anticipated to support the development of new next-generation technologies in the near future. Exponential rise in future Radio-Frequency (RF) sensing systems have demonstrated its applications in areas such as wearable consumer electronics, remote healthcare monitoring, wireless implants, and smart buildings. In this paper, we propose a novel, non-wearable, device-free, privacy-preserving Wi-Fi imaging-based occupancy detection system for future smart buildings. The proposed system is developed using off-the-shelf non-wearable devices such as Wi-Fi router, network interface card, and an omnidirectional antenna for future body centric communication. The core idea is to detect presence of person along its activities of daily living without deploying a device on person's body. The Wi-Fi signals received using non-wearable devices are converted into time-frequency scalograms. The occupancy is detected by classifying the scalogram images using an auto-encoder neural network. In addition to occupancy detection, the deep neural network also identifies the activity performed by the occupant. Moreover, a novel encryption algorithm using Chirikov and Intertwining map-based is also proposed to encrypt the scalogram images. This feature enables secure storage of scalogram images in a database for future analysis. The classification accuracy of the proposed scheme is 91.1%

VAE/WGAN-Based Image Representation Learning For Pose-Preserving Seamless Identity Replacement In Facial Images

We present a novel variational generative adversarial network (VGAN) based on Wasserstein loss to learn a latent representation from a face image that is invariant to identity but preserves head-pose information. This facilitates synthesis of a realistic face image with the same head pose as a given input image, but with a different identity. One application of this network is in privacy-sensitive scenarios; after identity replacement in an image, utility, such as head pose, can still be recovered. Extensive experimental validation on synthetic and real human-face image datasets performed under 3 threat scenarios confirms the ability of the proposed network to preserve head pose of the input image, mask the input identity, and synthesize a good-quality realistic face image of a desired identity. We also show that our network can be used to perform pose-preserving identity morphing and identity-preserving pose morphing. The proposed method improves over a recent state-of-the-art method in terms of quantitative metrics as well as synthesized image quality.Comment: 6 pages, 5 figures, 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP

Sensing within smart buildings: A survey

Increasingly, buildings are being fitted with sensors for the needs of different sectors, such as education, industry and business. Using Internet of Things (IoT) devices combined with analysis of data being generated by these devices, it is possible to infer a number of metrics, e.g. building occupancy and activities of occupants. The information thus gathered can be used to develop software applications to support energy management, occupant comfort, and space utilization. This survey explores the use of sensors in smart building environments, identifying different approaches to employ sensors in buildings. The most commonly used data-driven approaches for activity recognition in such buildings is also investigated, concluding by highlighting current research challenges and future research directions in this area