Data Analytics and Performance Enhancement in Edge-Cloud Collaborative Internet of Things Systems

Based on the evolving communications, computing and embedded systems technologies, Internet of Things (IoT) systems can interconnect not only physical users and devices but also virtual services and objects, which have already been applied to many different application scenarios, such as smart home, smart healthcare, and intelligent transportation. With the rapid development, the number of involving devices increases tremendously. The huge number of devices and correspondingly generated data bring critical challenges to the IoT systems. To enhance the overall performance, this thesis aims to address the related technical issues on IoT data processing and physical topology discovery of the subnets self-organized by IoT devices. First of all, the issues on outlier detection and data aggregation are addressed through the development of recursive principal component analysis (R-PCA) based data analysis framework. The framework is developed in a cluster-based structure to fully exploit the spatial correlation of IoT data. Specifically, the sensing devices are gathered into clusters based on spatial data correlation. Edge devices are assigned to the clusters for the R-PCA based outlier detection and data aggregation. The outlier-free and aggregated data are forwarded to the remote cloud server for data reconstruction and storage. Moreover, a data reduction scheme is further proposed to relieve the burden on the trunk link for data uploading by utilizing the temporal data correlation. Kalman filters (KFs) with identical parameters are maintained at the edge and cloud for data prediction. The amount of data uploading is reduced by using the data predicted by the KF in the cloud instead of uploading all the practically measured data. Furthermore, an unmanned aerial vehicle (UAV) assisted IoT system is particularly designed for large-scale monitoring. Wireless sensor nodes are flexibly deployed for environmental sensing and self-organized into wireless sensor networks (WSNs). A physical topology discovery scheme is proposed to construct the physical topology of WSNs in the cloud server to facilitate performance optimization, where the physical topology indicates both the logical connectivity statuses of WSNs and the physical locations of WSN nodes. The physical topology discovery scheme is implemented through the newly developed parallel Metropolis-Hastings random walk based information sampling and network-wide 3D localization algorithms, where UAVs are served as the mobile edge devices and anchor nodes. Based on the physical topology constructed in the cloud, a UAV-enabled spatial data sampling scheme is further proposed to efficiently sample data from the monitoring area by using denoising autoencoder (DAE). By deploying the encoder of DAE at the UAV and decoder in the cloud, the data can be partially sampled from the sensing field and accurately reconstructed in the cloud. In the final part of the thesis, a novel autoencoder (AE) neural network based data outlier detection algorithm is proposed, where both encoder and decoder of AE are deployed at the edge devices. Data outliers can be accurately detected by the large fluctuations in the squared error generated by the data passing through the encoder and decoder of the AE

Network anomaly detection research: a survey

Data analysis to identifying attacks/anomalies is a crucial task in anomaly detection and network anomaly detection itself is an important issue in network security. Researchers have developed methods and algorithms for the improvement of the anomaly detection system. At the same time, survey papers on anomaly detection researches are available. Nevertheless, this paper attempts to analyze futher and to provide alternative taxonomy on anomaly detection researches focusing on methods, types of anomalies, data repositories, outlier identity and the most used data type. In addition, this paper summarizes information on application network categories of the existing studies

Anomaly detection in smart city wireless sensor networks

Aquesta tesi proposa una plataforma de detecció d’intrusions per a revelar atacs a les xarxes de sensors sense fils (WSN, per les sigles en anglès) de les ciutats intel·ligents (smart cities). La plataforma està dissenyada tenint en compte les necessitats dels administradors de la ciutat intel·ligent, els quals necessiten accés a una arquitectura centralitzada que pugui gestionar alarmes de seguretat en un sistema altament heterogeni i distribuït. En aquesta tesi s’identifiquen els diversos passos necessaris des de la recollida de dades fins a l’execució de les tècniques de detecció d’intrusions i s’avalua que el procés sigui escalable i capaç de gestionar dades típiques de ciutats intel·ligents. A més, es comparen diversos algorismes de detecció d’anomalies i s’observa que els mètodes de vectors de suport d’una mateixa classe (one-class support vector machines) resulten la tècnica multivariant més adequada per a descobrir atacs tenint en compte les necessitats d’aquest context. Finalment, es proposa un esquema per a ajudar els administradors a identificar els tipus d’atacs rebuts a partir de les alarmes disparades.Esta tesis propone una plataforma de detección de intrusiones para revelar ataques en las redes de sensores inalámbricas (WSN, por las siglas en inglés) de las ciudades inteligentes (smart cities). La plataforma está diseñada teniendo en cuenta la necesidad de los administradores de la ciudad inteligente, los cuales necesitan acceso a una arquitectura centralizada que pueda gestionar alarmas de seguridad en un sistema altamente heterogéneo y distribuido. En esta tesis se identifican los varios pasos necesarios desde la recolección de datos hasta la ejecución de las técnicas de detección de intrusiones y se evalúa que el proceso sea escalable y capaz de gestionar datos típicos de ciudades inteligentes. Además, se comparan varios algoritmos de detección de anomalías y se observa que las máquinas de vectores de soporte de una misma clase (one-class support vector machines) resultan la técnica multivariante más adecuada para descubrir ataques teniendo en cuenta las necesidades de este contexto. Finalmente, se propone un esquema para ayudar a los administradores a identificar los tipos de ataques recibidos a partir de las alarmas disparadas.This thesis proposes an intrusion detection platform which reveals attacks in smart city wireless sensor networks (WSN). The platform is designed taking into account the needs of smart city administrators, who need access to a centralized architecture that can manage security alarms in a highly heterogeneous and distributed system. In this thesis, we identify the various necessary steps from gathering WSN data to running the detection techniques and we evaluate whether the procedure is scalable and capable of handling typical smart city data. Moreover, we compare several anomaly detection algorithms and we observe that one-class support vector machines constitute the most suitable multivariate technique to reveal attacks, taking into account the requirements in this context. Finally, we propose a classification schema to assist administrators in identifying the types of attacks compromising their networks

One-Class Classification: Taxonomy of Study and Review of Techniques

One-class classification (OCC) algorithms aim to build classification models when the negative class is either absent, poorly sampled or not well defined. This unique situation constrains the learning of efficient classifiers by defining class boundary just with the knowledge of positive class. The OCC problem has been considered and applied under many research themes, such as outlier/novelty detection and concept learning. In this paper we present a unified view of the general problem of OCC by presenting a taxonomy of study for OCC problems, which is based on the availability of training data, algorithms used and the application domains applied. We further delve into each of the categories of the proposed taxonomy and present a comprehensive literature review of the OCC algorithms, techniques and methodologies with a focus on their significance, limitations and applications. We conclude our paper by discussing some open research problems in the field of OCC and present our vision for future research.Comment: 24 pages + 11 pages of references, 8 figure

Exploitation of Data Correlation and Performance Enhancement in Wireless Sensor Networks

With the combination of wireless communications and embedded system, lots of progress has been made in the area of wireless sensor networks (WSNs). The networks have already been widely deployed, due to their self-organization capacity and low-cost advantage. However, there are still some technical challenges needed to be addressed. In the thesis, three algorithms are proposed in improving network energy efficiency, detecting data fault and reducing data redundancy. The basic principle behind the proposed algorithms is correlation in the data collected by WSNs. The first sensor scheduling algorithm is based on the spatial correlation between neighbor sensor readings. Given the spatial correlation, sensor nodes are clustered into groups. At each time instance, only one node within each group works as group representative, namely, sensing and transmitting sensor data. Sensor nodes take turns to be group representative. Therefore, the energy consumed by other sensor nodes within the same group can be saved. Due to the continuous nature of the data to be collected, temporal and spatial correlation of sensor data has been exploited to detect the faulty data. By exploitation of temporal correlation, the normal range of upcoming sensor data can be predicted by the historical observations. Based on spatial correlation, weighted neighbor voting can be used to diagnose whether the value of sensor data is reliable. The status of the sensor data, normal or faulty, is decided by the combination of these two proposed detection procedures. Similar to the sensor scheduling algorithm, the recursive principal component analysis (RPCA) based algorithm has been studied to detect faulty data and aggregate redundant data by exploitation of spatial correlation as well. The R-PCA model is used to process the sensor data, with the help of squared prediction error (SPE) score and cumulative percentage formula. When SPE score of a collected datum is distinctly larger than that of normal data, faults can be detected. The data dimension is reduced according to the calculation result of cumulative percentage formula. All the algorithms are simulated in OPNET or MATLAB based on practical and synthetic datasets. Performances of the proposed algorithms are evaluated in each chapter

Anomaly Detection in Streaming Sensor Data

In this chapter we consider a cell phone network as a set of automatically deployed sensors that records movement and interaction patterns of the population. We discuss methods for detecting anomalies in the streaming data produced by the cell phone network. We motivate this discussion by describing the Wireless Phone Based Emergency Response (WIPER) system, a proof-of-concept decision support system for emergency response managers. We also discuss some of the scientific work enabled by this type of sensor data and the related privacy issues. We describe scientific studies that use the cell phone data set and steps we have taken to ensure the security of the data. We describe the overall decision support system and discuss three methods of anomaly detection that we have applied to the data.Comment: 35 pages. Book chapter to appear in "Intelligent Techniques for Warehousing and Mining Sensor Network Data" (IGI Global), edited by A. Cuzzocre

From Intrusion Detection to Attacker Attribution: A Comprehensive Survey of Unsupervised Methods

Over the last five years there has been an increase in the frequency and diversity of network attacks. This holds true, as more and more organisations admit compromises on a daily basis. Many misuse and anomaly based Intrusion Detection Systems (IDSs) that rely on either signatures, supervised or statistical methods have been proposed in the literature, but their trustworthiness is debatable. Moreover, as this work uncovers, the current IDSs are based on obsolete attack classes that do not reflect the current attack trends. For these reasons, this paper provides a comprehensive overview of unsupervised and hybrid methods for intrusion detection, discussing their potential in the domain. We also present and highlight the importance of feature engineering techniques that have been proposed for intrusion detection. Furthermore, we discuss that current IDSs should evolve from simple detection to correlation and attribution. We descant how IDS data could be used to reconstruct and correlate attacks to identify attackers, with the use of advanced data analytics techniques. Finally, we argue how the present IDS attack classes can be extended to match the modern attacks and propose three new classes regarding the outgoing network communicatio

NeuDetect: A neural network data mining system for wireless network intrusion detection

This thesis proposes an Intrusion Detection System, NeuDetect, which applies Neural Network technique to wireless network packets captured through hardware sensors for purposes of real time detection of anomalous packets. To address the problem of high false alarm rate confronted by the current wireless intrusion detection systems, this thesis presents a method of applying the artificial neural networks technique to the wireless network intrusion detection system. The proposed system solution approach is to find normal and anomalous patterns on preprocessed wireless packet records by comparing them with training data using Back-propagation algorithm. An anomaly score is assigned to each packet by calculating the difference between the output error and threshold. If the anomaly score is positive then the wireless packet is flagged as anomalous and is negative then the packet is flagged as normal. If the anomaly score is zero or close to zero it will be flagged as an unknown attack and will be sent back to training process for re-evaluation