A Distributed Stream Processing Middleware Framework for Real-Time Analysis of Heterogeneous Data on Big Data Platform: Case of Environmental Monitoring

ArticleIn recent years, the application and wide adoption of Internet of Things (IoT)-based technologies have increased the proliferation of monitoring systems, which has consequently exponentially increased the amounts of heterogeneous data generated. Processing and analysing the massive amount of data produced is cumbersome and gradually moving from classical ‘batch’ processing—extract, transform, load (ETL) technique to real-time processing. For instance, in environmental monitoring and management domain, time-series data and historical dataset are crucial for prediction models. However, the environmental monitoring domain still utilises legacy systems, which complicates the real-time analysis of the essential data, integration with big data platforms and reliance on batch processing. Herein, as a solution, a distributed stream processing middleware framework for real-time analysis of heterogeneous environmental monitoring and management data is presented and tested on a cluster using open source technologies in a big data environment. The system ingests datasets from legacy systems and sensor data from heterogeneous automated weather systems irrespective of the data types to Apache Kafka topics using Kafka Connect APIs for processing by the Kafka streaming processing engine. The stream processing engine executes the predictive numerical models and algorithms represented in event processing (EP) languages for real-time analysis of the data streams. To prove the feasibility of the proposed framework, we implemented the system using a case study scenario of drought prediction and forecasting based on the Effective Drought Index (EDI) model. Firstly, we transform the predictive model into a form that could be executed by the streaming engine for real-time computing. Secondly, the model is applied to the ingested data streams and datasets to predict drought through persistent querying of the infinite streams to detect anomalies. As a conclusion of this study, a performance evaluation of the distributed stream processing middleware infrastructure is calculated to determine the real-time effectiveness of the framework

ICT monitoring and mathematical modelling of dairy cows performances in hot climate conditions: a study case in Po valley (Italy)

Automatic Milking Systems (AMS) measure and record specific data about milk production and cow behaviour, providing farmers with useful real-time information for each animal. At the same time, indoor climatic conditions in terms of temperature and humidity within a dairy livestock barn represent a well-known crucial issue in farm building design and management, since these parameters can remarkably influence cows behaviour, milk yield and animal welfare.The goal of the study is to develop and test an innovative procedure for the comprehensive analysis of AMS-generated multi-variable time-series, with a focus on the analysis of the relationship between milk production and indoor climatic conditions. The specific purpose of the study is to develop and test a mathematical computer procedure using AMS-generated data and environmental parameters, designed to provide a forecasting model based on the integration of milking data and temperature and humidity levels surveyed from local sensor grids, designed to model milk production scenarios and, specifically, yield trends depending on the expected environmental conditions.For this purpose, a typical Italian farm with AMS has been adopted as a study case and internal climatic data of the barn have been analysed to understand the influence of high values of the Temperature Humidity Index (THI) on milk production in time. Then the correlation between yield variations and THI has been computed and characterized. Finally, external climatic data have been used to forecast the milk production in summertime. Once the model was validated, tests has led to predict milk yield with a relative error smaller than 2%.This study represents a step of a research aimed to define integrated systems for cow monitoring and to develop guidelines for the optimization of barn layouts

Latihan mengajar : Keberkesanannya terhadap pelajar Diploma Kejuruteraan serta Pendidikan di KUiTTHO (Kolej Universiti Teknologi Tun Hussein Onn) menurut persepsi pelajar

Kajian yang dijaiankan adaiah bertajuk "Latihan Mengajar : Kebersanannya Terhadap Pelajar Diploma Kejuruteraan berserta Pendidikan di KUiTTHO (Kolej Universiti Teknologi Tun Hussein Onn Menurut Persepsi Pelajar. Kajian ini bertujuan untuk meiihat sejauhmana keberkesanan program latihan mengajar terhadap pelajar yang telah melaluinya. Borang soalselidik diedarkan untuk mendapatkan maklumat dan seterusnya dianalisis untuk menghasilkan skor min dan peratusan. Hasil kajian menunjukkan kebanyakan responden memberikan reaksi positif terhadap keberkesanan program latihan mengajar. Hasil dari anaiisis kajian juga, pengkaji telah menghasilkan sebuah produk iaitu senarai semak yang boleh digunakan oleh pelajar yang akan menjalani program latihan mengajar supaya pelajar jelas dengan tindakan yang harus mereka ambil sebeium, semasa dan selepas menjalani latihan mengajar. Adaiah diharapkan agar produk ini dapat membantu untuk pelajar, pihak KUiTTHO dan seterusnya institusi tempat latihan mengajar supaya program ini dapat dilaksanakan dengan Iebih sempuma dan seterusnya mencapai objektif program latihan mengajar

Towards Semantic Integration of Heterogeneous Sensor Data with Indigenous Knowledge for Drought Forecasting

In the Internet of Things (IoT) domain, various heterogeneous ubiquitous devices would be able to connect and communicate with each other seamlessly, irrespective of the domain. Semantic representation of data through detailed standardized annotation has shown to improve the integration of the interconnected heterogeneous devices. However, the semantic representation of these heterogeneous data sources for environmental monitoring systems is not yet well supported. To achieve the maximum benefits of IoT for drought forecasting, a dedicated semantic middleware solution is required. This research proposes a middleware that semantically represents and integrates heterogeneous data sources with indigenous knowledge based on a unified ontology for an accurate IoT-based drought early warning system (DEWS).Comment: 5 pages, 3 figures, In Proceedings of the Doctoral Symposium of the 16th International Middleware Conference (Middleware Doct Symposium 2015), Ivan Beschastnikh and Wouter Joosen (Eds.). ACM, New York, NY, US

Air Quality Prediction in Smart Cities Using Machine Learning Technologies Based on Sensor Data: A Review

The influence of machine learning technologies is rapidly increasing and penetrating almost in every field, and air pollution prediction is not being excluded from those fields. This paper covers the revision of the studies related to air pollution prediction using machine learning algorithms based on sensor data in the context of smart cities. Using the most popular databases and executing the corresponding filtration, the most relevant papers were selected. After thorough reviewing those papers, the main features were extracted, which served as a base to link and compare them to each other. As a result, we can conclude that: (1) instead of using simple machine learning techniques, currently, the authors apply advanced and sophisticated techniques, (2) China was the leading country in terms of a case study, (3) Particulate matter with diameter equal to 2.5 micrometers was the main prediction target, (4) in 41% of the publications the authors carried out the prediction for the next day, (5) 66% of the studies used data had an hourly rate, (6) 49% of the papers used open data and since 2016 it had a tendency to increase, and (7) for efficient air quality prediction it is important to consider the external factors such as weather conditions, spatial characteristics, and temporal features