Spatiotemporal anomaly detection: streaming architecture and algorithms

Includes bibliographical references.2020 Summer.Anomaly detection is the science of identifying one or more rare or unexplainable samples or events in a dataset or data stream. The field of anomaly detection has been extensively studied by mathematicians, statisticians, economists, engineers, and computer scientists. One open research question remains the design of distributed cloud-based architectures and algorithms that can accurately identify anomalies in previously unseen, unlabeled streaming, multivariate spatiotemporal data. With streaming data, time is of the essence, and insights are perishable. Real-world streaming spatiotemporal data originate from many sources, including mobile phones, supervisory control and data acquisition enabled (SCADA) devices, the internet-of-things (IoT), distributed sensor networks, and social media. Baseline experiments are performed on four (4) non-streaming, static anomaly detection multivariate datasets using unsupervised offline traditional machine learning (TML), and unsupervised neural network techniques. Multiple architectures, including autoencoders, generative adversarial networks, convolutional networks, and recurrent networks, are adapted for experimentation. Extensive experimentation demonstrates that neural networks produce superior detection accuracy over TML techniques. These same neural network architectures can be extended to process unlabeled spatiotemporal streaming using online learning. Space and time relationships are further exploited to provide additional insights and increased anomaly detection accuracy. A novel domain-independent architecture and set of algorithms called the Spatiotemporal Anomaly Detection Environment (STADE) is formulated. STADE is based on federated learning architecture. STADE streaming algorithms are based on a geographically unique, persistently executing neural networks using online stochastic gradient descent (SGD). STADE is designed to be pluggable, meaning that alternative algorithms may be substituted or combined to form an ensemble. STADE incorporates a Stream Anomaly Detector (SAD) and a Federated Anomaly Detector (FAD). The SAD executes at multiple locations on streaming data, while the FAD executes at a single server and identifies global patterns and relationships among the site anomalies. Each STADE site streams anomaly scores to the centralized FAD server for further spatiotemporal dependency analysis and logging. The FAD is based on recent advances in DNN-based federated learning. A STADE testbed is implemented to facilitate globally distributed experimentation using low-cost, commercial cloud infrastructure provided by Microsoft™. STADE testbed sites are situated in the cloud within each continent: Africa, Asia, Australia, Europe, North America, and South America. Communication occurs over the commercial internet. Three STADE case studies are investigated. The first case study processes commercial air traffic flows, the second case study processes global earthquake measurements, and the third case study processes social media (i.e., Twitter™) feeds. These case studies confirm that STADE is a viable architecture for the near real-time identification of anomalies in streaming data originating from (possibly) computationally disadvantaged, geographically dispersed sites. Moreover, the addition of the FAD provides enhanced anomaly detection capability. Since STADE is domain-independent, these findings can be easily extended to additional application domains and use cases

DaaS: Dew Computing as a Service for Intelligent Intrusion Detection in Edge-of-Things Ecosystem

Edge of Things (EoT) enables the seamless transfer of services, storage, and data processing from the cloud layer to edge devices in a large-scale distributed Internet of Things (IoT) ecosystems (e.g., Industrial systems). This transition raises the privacy and security concerns in the EoT paradigm distributed at different layers. Intrusion detection systems (IDSs) are implemented in EoT ecosystems to protect the underlying resources from attackers. However, the current IDSs are not intelligent enough to control the false alarms, which significantly lower the reliability and add to the analysis burden on the IDSs. In this article, we present a Dew Computing as a Service (DaaS) for intelligent intrusion detection in EoT ecosystems. In DaaS, a deep learning-based classifier is used to design an intelligent alarm filtration mechanism. In this mechanism, the filtration accuracy is improved (or sustained) by using deep belief networks. In the past, the cloud-based techniques have been applied for offloading the EoT tasks, which increases the middle layer burden and raises the communication delay. Here, we introduce the dew computing features that are used to design the smart false alarm reduction system. DaaS, when experimented in a simulated environment, reflects lower response time to process the data in the EoT ecosystem. The revamped DBN model achieved the classification accuracy up to 95%. Moreover, it depicts a 60% improvement in the latency and 35% workload reduction of the cloud servers as compared to edge IDS

A deep learning approach for intrusion detection in Internet of Things using bi-directional long short-term memory recurrent neural network

Internet-of-Things connects every ‘thing’ with the Internet and allows these ‘things’ to communicate with each other. IoT comprises of innumerous interconnected devices of diverse complexities and trends. This fundamental nature of IoT structure intensifies the amount of attack targets which might affect the sustainable growth of IoT. Thus, security issues become a crucial factor to be addressed. A novel deep learning approach have been proposed in this thesis, for performing real-time detections of security threats in IoT systems using the Bi-directional Long Short-Term Memory Recurrent Neural Network (BLSTM RNN). The proposed approach have been implemented through Google TensorFlow implementation framework and Python programming language. To train and test the proposed approach, UNSW-NB15 dataset has been employed, which is the most up-to-date benchmark dataset with sequential samples and contemporary attack patterns. This thesis work employs binary classification of attack and normal patterns. The experimental result demonstrates the proficiency of the introduced model with respect to recall, precision, FAR and f-1 score. The model attains over 97% detection accuracy. The test result demonstrates that BLSTM RNN is profoundly effective for building highly efficient model for intrusion detection and offers a novel research methodology

Boosting precision crop protection towards agriculture 5.0 via machine learning and emerging technologies: A contextual review

Crop protection is a key activity for the sustainability and feasibility of agriculture in a current context of climate change, which is causing the destabilization of agricultural practices and an increase in the incidence of current or invasive pests, and a growing world population that requires guaranteeing the food supply chain and ensuring food security. In view of these events, this article provides a contextual review in six sections on the role of artificial intelligence (AI), machine learning (ML) and other emerging technologies to solve current and future challenges of crop protection. Over time, crop protection has progressed from a primitive agriculture 1.0 (Ag1.0) through various technological developments to reach a level of maturity closelyin line with Ag5.0 (section 1), which is characterized by successfully leveraging ML capacity and modern agricultural devices and machines that perceive, analyze and actuate following the main stages of precision crop protection (section 2). Section 3 presents a taxonomy of ML algorithms that support the development and implementation of precision crop protection, while section 4 analyses the scientific impact of ML on the basis of an extensive bibliometric study of >120 algorithms, outlining the most widely used ML and deep learning (DL) techniques currently applied in relevant case studies on the detection and control of crop diseases, weeds and plagues. Section 5 describes 39 emerging technologies in the fields of smart sensors and other advanced hardware devices, telecommunications, proximal and remote sensing, and AI-based robotics that will foreseeably lead the next generation of perception-based, decision-making and actuation systems for digitized, smart and real-time crop protection in a realistic Ag5.0. Finally, section 6 highlights the main conclusions and final remarks

Thirty Years of Machine Learning: The Road to Pareto-Optimal Wireless Networks

Future wireless networks have a substantial potential in terms of supporting a broad range of complex compelling applications both in military and civilian fields, where the users are able to enjoy high-rate, low-latency, low-cost and reliable information services. Achieving this ambitious goal requires new radio techniques for adaptive learning and intelligent decision making because of the complex heterogeneous nature of the network structures and wireless services. Machine learning (ML) algorithms have great success in supporting big data analytics, efficient parameter estimation and interactive decision making. Hence, in this article, we review the thirty-year history of ML by elaborating on supervised learning, unsupervised learning, reinforcement learning and deep learning. Furthermore, we investigate their employment in the compelling applications of wireless networks, including heterogeneous networks (HetNets), cognitive radios (CR), Internet of things (IoT), machine to machine networks (M2M), and so on. This article aims for assisting the readers in clarifying the motivation and methodology of the various ML algorithms, so as to invoke them for hitherto unexplored services as well as scenarios of future wireless networks.Comment: 46 pages, 22 fig

Trust Management for Internet of Things: A Systematic Literature Review

Internet of Things (IoT) is a network of devices that communicate with each other through the internet and provides intelligence to industry and people. These devices are running in potentially hostile environments, so the need for security is critical. Trust Management aims to ensure the reliability of the network by assigning a trust value in every node indicating its trust level. This paper presents an exhaustive survey of the current Trust Management techniques for IoT, a classification based on the methods used in every work and a discussion of the open challenges and future research directions.Comment: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessibl

A Deep Learning Approach to Network Intrusion Detection

Software Defined Networking (SDN) has recently emerged to become one of the promising solutions for the future Internet. With the logical centralization of controllers and a global network overview, SDN brings us a chance to strengthen our network security. However, SDN also brings us a dangerous increase in potential threats. In this paper, we apply a deep learning approach for flow-based anomaly detection in an SDN environment. We build a Deep Neural Network (DNN) model for an intrusion detection system and train the model with the NSL-KDD Dataset. In this work, we just use six basic features (that can be easily obtained in an SDN environment) taken from the forty-one features of NSL-KDD Dataset. Through experiments, we confirm that the deep learning approach shows strong potential to be used for flow-based anomaly detection in SDN environments

A Survey on Malware Detection with Graph Representation Learning

Malware detection has become a major concern due to the increasing number and complexity of malware. Traditional detection methods based on signatures and heuristics are used for malware detection, but unfortunately, they suffer from poor generalization to unknown attacks and can be easily circumvented using obfuscation techniques. In recent years, Machine Learning (ML) and notably Deep Learning (DL) achieved impressive results in malware detection by learning useful representations from data and have become a solution preferred over traditional methods. More recently, the application of such techniques on graph-structured data has achieved state-of-the-art performance in various domains and demonstrates promising results in learning more robust representations from malware. Yet, no literature review focusing on graph-based deep learning for malware detection exists. In this survey, we provide an in-depth literature review to summarize and unify existing works under the common approaches and architectures. We notably demonstrate that Graph Neural Networks (GNNs) reach competitive results in learning robust embeddings from malware represented as expressive graph structures, leading to an efficient detection by downstream classifiers. This paper also reviews adversarial attacks that are utilized to fool graph-based detection methods. Challenges and future research directions are discussed at the end of the paper.Comment: Preprint, submitted to ACM Computing Surveys on March 2023. For any suggestions or improvements, please contact me directly by e-mai

Consumer-facing technology fraud : economics, attack methods and potential solutions

The emerging use of modern technologies has not only benefited society but also attracted fraudsters and criminals to misuse the technology for financial benefits. Fraud over the Internet has increased dramatically, resulting in an annual loss of billions of dollars to customers and service providers worldwide. Much of such fraud directly impacts individuals, both in the case of browser-based and mobile-based Internet services, as well as when using traditional telephony services, either through landline phones or mobiles. It is important that users of the technology should be both informed of fraud, as well as protected from frauds through fraud detection and prevention systems. In this paper, we present the anatomy of frauds for different consumer-facing technologies from three broad perspectives - we discuss Internet, mobile and traditional telecommunication, from the perspectives of losses through frauds over the technology, fraud attack mechanisms and systems used for detecting and preventing frauds. The paper also provides recommendations for securing emerging technologies from fraud and attacks