A Distributed Stream Processing Middleware Framework for Real-Time Analysis of Heterogeneous Data on Big Data Platform: Case of Environmental Monitoring

ArticleIn recent years, the application and wide adoption of Internet of Things (IoT)-based technologies have increased the proliferation of monitoring systems, which has consequently exponentially increased the amounts of heterogeneous data generated. Processing and analysing the massive amount of data produced is cumbersome and gradually moving from classical ‘batch’ processing—extract, transform, load (ETL) technique to real-time processing. For instance, in environmental monitoring and management domain, time-series data and historical dataset are crucial for prediction models. However, the environmental monitoring domain still utilises legacy systems, which complicates the real-time analysis of the essential data, integration with big data platforms and reliance on batch processing. Herein, as a solution, a distributed stream processing middleware framework for real-time analysis of heterogeneous environmental monitoring and management data is presented and tested on a cluster using open source technologies in a big data environment. The system ingests datasets from legacy systems and sensor data from heterogeneous automated weather systems irrespective of the data types to Apache Kafka topics using Kafka Connect APIs for processing by the Kafka streaming processing engine. The stream processing engine executes the predictive numerical models and algorithms represented in event processing (EP) languages for real-time analysis of the data streams. To prove the feasibility of the proposed framework, we implemented the system using a case study scenario of drought prediction and forecasting based on the Effective Drought Index (EDI) model. Firstly, we transform the predictive model into a form that could be executed by the streaming engine for real-time computing. Secondly, the model is applied to the ingested data streams and datasets to predict drought through persistent querying of the infinite streams to detect anomalies. As a conclusion of this study, a performance evaluation of the distributed stream processing middleware infrastructure is calculated to determine the real-time effectiveness of the framework

Lotus: Serverless In-Transit Data Processing for Edge-based Pub/Sub

Publish-subscribe systems are a popular approach for edge-based IoT use cases: Heterogeneous, constrained edge devices can be integrated easily, with message routing logic offloaded to edge message brokers. Message processing, however, is still done on constrained edge devices. Complex content-based filtering, the transformation between data representations, or message extraction place a considerable load on these systems, and resulting superfluous message transfers strain the network. In this paper, we propose Lotus, adding in-transit data processing to an edge publish-subscribe middleware in order to offload basic message processing from edge devices to brokers. Specifically, we leverage the Function-as-a-Service paradigm, which offers support for efficient multi-tenancy, scale-to-zero, and real-time processing. With a proof-of-concept prototype of Lotus, we validate its feasibility and demonstrate how it can be used to offload sensor data transformation to the publish-subscribe messaging middleware

EVE: An Environment for On-board Processing

The Information Technology and Systems Center (ITSC) at The University of Alabama in Huntsville (UAH) is investigating and developing an innovative processing system capable of handling the unique constraints and characteristics of the on-board satellite data and information environment. The EnVironmEnt for On-Board Processing (EVE) system will serve as a proof-of-concept of advanced information systems technology for remote sensing platforms. EVE’s on-board, real-time processing will provide capabilities focused on the areas of autonomous data mining, classification and feature extraction. These will contribute to Earth Science research applications, including natural hazard detection and prediction, fusion of multi-sensor measurements, ntelligent sensor control, and the generation of customized data products for direct distribution to users. EVE is being engineered to provide high performance data processing in a real-time operational environment. A ground-based testbed is being created to provide testing of EVE and associated Earth Science applications in a heterogeneous embedded hardware and software environment

EVE: An Environment for On-board Processing

The Information Technology and Systems Center (ITSC) at The University of Alabama in Huntsville (UAH) is investigating and developing an innovative processing system capable of handling the unique constraints and characteristics of the on-board satellite data and information environment. The EnVironmEnt for On-Board Processing (EVE) system will serve as a proof-of-concept of advanced information systems technology for remote sensing platforms. EVE’s on-board, real-time processing will provide capabilities focused on the areas of autonomous data mining, classification and feature extraction. These will contribute to Earth Science research applications, including natural hazard detection and prediction, fusion of multi-sensor measurements, ntelligent sensor control, and the generation of customized data products for direct distribution to users. EVE is being engineered to provide high performance data processing in a real-time operational environment. A ground-based testbed is being created to provide testing of EVE and associated Earth Science applications in a heterogeneous embedded hardware and software environment

SMART: An Application Framework for Real Time Big Data Analysis on Heterogeneous Cloud Environments

International audienceThe amount of data that human activities generate poses a challenge to current computer systems. Big data processing techniques are evolving to address this challenge, with analysis increasingly being performed using cloud-based systems. Emerging services, however, require additional enhancements in order to ensure their applicability to highly dynamic and heterogeneous environments and facilitate their use by Small & Medium-sized Enterprises (SMEs). Observing this landscape in emerging computing system development, this work presents Small & Medium-sized Enterprise Data Analytic in Real Time (SMART) for addressing some of the issues in providing compute service solutions for SMEs. SMART offers a framework for efficient development of Big Data analysis services suitable to small and medium-sized organizations, considering very heterogeneous data sources, from wireless sensor networks to data warehouses, focusing on service composability for a number of domains. This paper presents the basis of this proposal and preliminary results on exploring application deployment on hybrid infrastructure

EVE: On-Board Process Planning and Execution

Abstract The Information Technology and Systems Center (ITSC) at The University of Alabama in Huntsville is developing an innovative processing framework aimed at assisting science users in the use of the unique constraints and characteristics of the on-board satellite data and information environment. The Environment for On-Board Processing (EVE) system serves as a proof-of-concept of advanced information systems technology for remote sensing platforms. With EVE, data is processed as it is collected, enabling the production of custom data products on-board and in real-time. The web-based drag-and-drop EVE editor allows science users to build processing plans, which are compatible with the constraints of on-board computing environments. The EVE onboard, real-time processing infrastructure, will upload, schedule, and control the execution of these plans. Operations within the plans provide capabilities focused on the areas of autonomous data mining, classification and feature extraction using both streaming and buffered data sources. These will contribute to science research applications, including natural hazard detection and prediction, fusion of multi-sensor measurements, intelligent sensor control, and the generation of customized data products for direct distribution to users. A ground-based testbed has been created to provide testing of EVE and associated science applications in a heterogeneous, embedded hardware and software environment. Testbed components include platforms that represent both space based and ground based sensor platforms, including wireless sensor mesh architectures

Multi-head CNN–RNN for multi-time series anomaly detection: An industrial case study

Detecting anomalies in time series data is becoming mainstream in a wide variety of industrial applications in which sensors monitor expensive machinery. The complexity of this task increases when multiple heterogeneous sensors provide information of di_erent nature, scales and frequencies from the same machine. Traditionally, machine learning techniques require a separate data preprocessing before training, which tends to be very time-consuming and often requires domain knowledge. Recent deep learning approaches have shown to perform well on raw time series data, eliminating the need for pre-processing. In this work, we propose a deep learning based approach for supervised multitime series anomaly detection that combines a Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN) in different ways. Unlike other approaches, we use independent CNNs, so-called convolutional heads, to deal with anomaly detection in multi-sensor systems. We address each sensor individually avoiding the need for data pre-processing and allowing for a more tailored architecture for each type of sensor. We refer to this architecture as Multi-head CNN-RNN. The proposed architecture is assessed against a real industrial case study, provided by an industrial partner, where a service elevator is monitored. Within this case study, three type of anomalies are considered: point, context-specific, and collective. The experimental results show that the proposed architecture is suitable for multi-time series anomaly detection as it obtained promising results on the real industrial scenario

EVE: On-Board Process Planning and Execution

The Information Technology and Systems Center (ITSC) at The University of Alabama in Huntsville(UAH) has designed and is now developing an innovative processing framework aimed at helping science users exploit the unique constraints and characteristics of the on-board satellite data and information environment. The Environment for On-Board Processing (EVE) system will serve as a proof-of-concept of advanced information systems technology for remote sensing platforms. Because data will be processed as it’s collected, such a system will produce custom data products on-board and in real-time. First, the EVE editor allows science users to build processing plans, which are compatible with the constraints of on-orbit computing environments. The EVE on-board, real-time processing infrastructure in turn, will upload, schedule, and control the execution of these plans. Operations within the plans provide capabilities focused on the areas of autonomous data mining, classification and feature extraction. These will contribute to Earth Science research applications, including natural hazard detection and prediction, fusion of multi-sensor measurements, intelligent sensor control, and the generation of customized data products for direct distribution to users. A ground-based testbed is being created to provide testing of EVE and associated Earth Science applications in a heterogeneous embedded hardware and software environment