GR-45 Framework for Collecting Data from specialized IoT devices.

The Internet of Things (IoT) is the most significant and blooming technology in the 21st century. IoT has rapidly developed by covering hundreds of applications in the civil, health, military, and agriculture areas. IoT is based on the collection of sensor data through an embedded system, and this embedded system uploads the data on the internet. Devices and sensor technologies connected over a network can monitor and measure data in real-time. The main challenge is to collect data from IoT devices, transmit them to store in the Cloud, and later retrieve them at any time for visualization and data analysis. All these phases need to be secure by following security protocol to ensure data integrity. In this paper, we present the design of a lightweight and easy-to-use data collection framework for IoT devices. This framework consists of collecting data from sensors and sending them to Cloud storage securely and in real-time for further processing and visualization. Our main objective is to make a data-collecting platform that will be plug-and-play and secure so that any organization or research team can use it to collect data from any IoT device for further data analysis. This framework is expected to help with the data collection from a variety of different IoT devices.Advisors(s): Dr. Maria Valero, Dr. Hossain ShahriarTopic(s): IoT/Cloud/Networkin

Framework for collecting data from IoT Device

The Internet of Things (IoT) is the most significant and blooming technology in the 21st century. IoT has rapidly developed by covering hundreds of applications in the civil, health, military, and agriculture areas. IoT is based on the collection of sensor data through an embedded system, and this embedded system uploads the data on the internet. Devices and sensor technologies connected over a network can monitor and measure data in real-time. The main challenge is to collect data from IoT devices, transmit them to store in the Cloud, and later retrieve them at any time for visualization and data analysis. All these phases need to be secure by following security protocol to ensure data integrity. This work presents the design of a lightweight and easy-to-use data collection framework for IoT devices. This framework consists of collecting data from sensors and sending them to Cloud storage securely and in real-time for further processing and visualization. Our main objective is to make a data-collecting platform that will be plug-and-play and secure so that any organization or research team can use it to collect data from any IoT device for further data analysis. This framework is expected to help with the data collection from a variety of different IoT devices

Boosting big data streaming applications in clouds with burstFlow

The rapid growth of stream applications in financial markets, health care, education, social media, and sensor networks represents a remarkable milestone for data processing and analytic in recent years, leading to new challenges to handle Big Data in real-time. Traditionally, a single cloud infrastructure often holds the deployment of Stream Processing applications because it has extensive and adaptative virtual computing resources. Hence, data sources send data from distant and different locations of the cloud infrastructure, increasing the application latency. The cloud infrastructure may be geographically distributed and it requires to run a set of frameworks to handle communication. These frameworks often comprise a Message Queue System and a Stream Processing Framework. The frameworks explore Multi-Cloud deploying each service in a different cloud and communication via high latency network links. This creates challenges to meet real-time application requirements because the data streams have different and unpredictable latencies forcing cloud providers' communication systems to adjust to the environment changes continually. Previous works explore static micro-batch demonstrating its potential to overcome communication issues. This paper introduces BurstFlow, a tool for enhancing communication across data sources located at the edges of the Internet and Big Data Stream Processing applications located in cloud infrastructures. BurstFlow introduces a strategy for adjusting the micro-batch sizes dynamically according to the time required for communication and computation. BurstFlow also presents an adaptive data partition policy for distributing incoming streams across available machines by considering memory and CPU capacities. The experiments use a real-world multi-cloud deployment showing that BurstFlow can reduce the execution time up to 77% when compared to the state-of-the-art solutions, improving CPU efficiency by up to 49%

A Novel Real-Time Edge-Cloud Big Data Management and Analytics Framework for Smart Cities

Exposing city information to dynamic, distributed, powerful, scalable, and user-friendly big data systems is expected to enable the implementation of a wide range of new opportunities; however, the size, heterogeneity and geographical dispersion of data often makes it difficult to combine, analyze and consume them in a single system. In the context of the H2020 CLASS project, we describe an innovative framework aiming to facilitate the design of advanced big-data analytics workflows. The proposal covers the whole compute continuum, from edge to cloud, and relies on a well-organized distributed infrastructure exploiting: a) edge solutions with advanced computer vision technologies enabling the real-time generation of “rich” data from a vast array of sensor types; b) cloud data management techniques offering efficient storage, real-time querying and updating of the high-frequency incoming data at different granularity levels. We specifically focus on obstacle detection and tracking for edge processing, and consider a traffic density monitoring application, with hierarchical data aggregation features for cloud processing; the discussed techniques will constitute the groundwork enabling many further services. The tests are performed on the real use-case of the Modena Automotive Smart Area (MASA)

Newsletter of the Digital Earth Project Contributions of the Alfred Wegener Institute to Digital Earth

As an important technical pillar of Digital Earth AWI computing centre provides data management and cloud processing services to the project partners. We develop project specific extensions to the AWI data flow framework O2A (Observation to Archive). Sensor registration in O2A will support a flexible handling of sensors and their metadata, e.g. for the Digital Earth showcases, methane and soil moisture measurements are in focus for smart monitoring designs and for the access to data in near real time (NRT). Furthermore, data exploration is supported by a rasterdata manager service that can be easily coupled in user ́s data workflows with other data sources, like NRT sensor data. In the following we give more details on O2A, its components and concept

MOSAiC goes O2A - Arctic Expedition Data Flow from Observations to Archives

During the largest polar expedition in history starting in September 2019, the German research icebreaker Polarstern spends a whole year drifting with the ice through the Arctic Ocean. The MOSAiC expedition takes the closest look ever at the Arctic even throughout the polar winter to gain fundamental insights and most unique on-site data for a better understanding of global climate change. Hundreds of researchers from 20 countries are involved. Scientists will use the in situ gathered data instantaneously in near-real time modus as well as long afterwards all around the globe taking climate research to a completely new level. Hence, proper data management, sampling strategies beforehand, and monitoring actual data flow as well as processing, analysis and sharing of data during and long after the MOSAiC expedition are the most essential tools for scientific gain and progress. To prepare for that challenge we adapted and integrated the research data management framework O2A “Data flow from Observations to Archives” to the needs of the MOSAiC expedition on board Polarstern as well as on land for data storage and access at the Alfred Wegener Institute Computing and Data Center in Bremerhaven, Germany. Our O2A-framework assembles a modular research infrastructure comprising a collection of tools and services. These components allow researchers to register all necessary sensor metadata beforehand linked to automatized data ingestion and to ensure and monitor data flow as well as to process, analyze, and publish data to turn the most valuable and uniquely gained arctic data into scientific outcomes. The framework further allows for the integration of data obtained with discrete sampling devices into the data flow. These requirements have led us to adapt the generic and cost-effective framework O2A to enable, control, and access the flow of sensor observations to archives in a cloud-like infrastructure on board Polarstern and later on to land based repositories for international availability. Major roadblocks of the MOSAiC-O2A data flow framework are (i) the increasing number and complexity of research platforms, devices, and sensors, (ii) the heterogeneous interdisciplinary driven requirements towards, e. g., satellite data, sensor monitoring, in situ sample collection, quality assessment and control, processing, analysis and visualization, and (iii) the demand for near real time analyses on board as well as on land with limited satellite bandwidth. The key modules of O2A's digital research infrastructure established by AWI are implementing the FAIR principles: SENSORWeb, to register sensor applications and sampling devices and capture controlled meta data before and alongside any measurements in the field Data ingest, allowing researchers to feed data into storage systems and processing pipelines in a prepared and documented way, at best in controlled near real-time data streams Dashboards allowing researchers to find and access data and share and collaborate among partners Workspace enabling researchers to access and use data with research software utilizing a cloud-based virtualized infrastructure that allows researchers to analyze massive amounts of data on the spot Archiving and publishing data via repositories and Digital Object Identifiers (DOI

Designing the next generation intelligent transportation sensor system using big data driven machine learning techniques

Accurate traffic data collection is essential for supporting advanced traffic management system operations. This study investigated a large-scale data-driven sequential traffic sensor health monitoring (TSHM) module that can be used to monitor sensor health conditions over large traffic networks. Our proposed module consists of three sequential steps for detecting different types of abnormal sensor issues. The first step detects sensors with abnormally high missing data rates, while the second step uses clustering anomaly detection to detect sensors reporting abnormal records. The final step introduces a novel Bayesian changepoint modeling technique to detect sensors reporting abnormal traffic data fluctuations by assuming a constant vehicle length distribution based on average effective vehicle length (AEVL). Our proposed method is then compared with two benchmark algorithms to show its efficacy. Results obtained by applying our method to the statewide traffic sensor data of Iowa show it can successfully detect different classes of sensor issues. This demonstrates that sequential TSHM modules can help transportation agencies determine traffic sensors’ exact problems, thereby enabling them to take the required corrective steps. The second research objective will focus on the traffic data imputation after we discard the anomaly/missing data collected from failure traffic sensors. Sufficient high-quality traffic data are a crucial component of various Intelligent Transportation System (ITS) applications and research related to congestion prediction, speed prediction, incident detection, and other traffic operation tasks. Nonetheless, missing traffic data are a common issue in sensor data which is inevitable due to several reasons, such as malfunctioning, poor maintenance or calibration, and intermittent communications. Such missing data issues often make data analysis and decision-making complicated and challenging. In this study, we have developed a generative adversarial network (GAN) based traffic sensor data imputation framework (TSDIGAN) to efficiently reconstruct the missing data by generating realistic synthetic data. In recent years, GANs have shown impressive success in image data generation. However, generating traffic data by taking advantage of GAN based modeling is a challenging task, since traffic data have strong time dependency. To address this problem, we propose a novel time-dependent encoding method called the Gramian Angular Summation Field (GASF) that converts the problem of traffic time-series data generation into that of image generation. We have evaluated and tested our proposed model using the benchmark dataset provided by Caltrans Performance Management Systems (PeMS). This study shows that the proposed model can significantly improve the traffic data imputation accuracy in terms of Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) compared to state-of-the-art models on the benchmark dataset. Further, the model achieves reasonably high accuracy in imputation tasks even under a very high missing data rate (\u3e50%), which shows the robustness and efficiency of the proposed model. Besides the loop and radar sensors, traffic cameras have shown great ability to provide insightful traffic information using the image and video processing techniques. Therefore, the third and final part of this work aimed to introduce an end to end real-time cloud-enabled traffic video analysis (IVA) framework to support the development of the future smart city. As Artificial intelligence (AI) growing rapidly, Computer vision (CV) techniques are expected to significantly improve the development of intelligent transportation systems (ITS), which are anticipated to be a key component of future Smart City (SC) frameworks. Powered by computer vision techniques, the converting of existing traffic cameras into connected ``smart sensors called intelligent video analysis (IVA) systems has shown the great capability of producing insightful data to support ITS applications. However, developing such IVA systems for large-scale, real-time application deserves further study, as the current research efforts are focused more on model effectiveness instead of model efficiency. Therefore, we have introduced a real-time, large-scale, cloud-enabled traffic video analysis framework using NVIDIA DeepStream, which is a streaming analysis toolkit for AI-based video and image analysis. In this study, we have evaluated the technical and economic feasibility of our proposed framework to help traffic agency to build IVA systems more efficiently. Our study shows that the daily operating cost for our proposed framework on Google Cloud Platform (GCP) is less than $0.14 per camera, and that, compared with manual inspections, our framework achieves an average vehicle-counting accuracy of 83.7% on sunny days

SMART: An Application Framework for Real Time Big Data Analysis on Heterogeneous Cloud Environments

International audienceThe amount of data that human activities generate poses a challenge to current computer systems. Big data processing techniques are evolving to address this challenge, with analysis increasingly being performed using cloud-based systems. Emerging services, however, require additional enhancements in order to ensure their applicability to highly dynamic and heterogeneous environments and facilitate their use by Small & Medium-sized Enterprises (SMEs). Observing this landscape in emerging computing system development, this work presents Small & Medium-sized Enterprise Data Analytic in Real Time (SMART) for addressing some of the issues in providing compute service solutions for SMEs. SMART offers a framework for efficient development of Big Data analysis services suitable to small and medium-sized organizations, considering very heterogeneous data sources, from wireless sensor networks to data warehouses, focusing on service composability for a number of domains. This paper presents the basis of this proposal and preliminary results on exploring application deployment on hybrid infrastructure

Aprendizaje automático basado en mezcla Gaussiana mejorada Modelo para IoT en tiempo real: Análisis de los datos

Introduction: The article is the product of the research “Due to the increase in popularity of Internet of Things (IoT), a huge amount of sensor data is being generated from various smart city applications”, developed at Pondicherry University in the year 2019. Problem:To acquire and analyze the huge amount of sensor-generated data effectively is a significant problem when processing the data. Objective:  To propose a novel framework for IoT sensor data analysis using machine learning based improved Gaussian Mixture Model (GMM) by acquired real-time data.  Methodology:In this paper, the clustering based GMM models are used to find the density patterns on a daily or weekly basis for user requirements. The ThingSpeak cloud platform used for performing analysis and visualizations. Results:An analysis has been performed on the proposed mechanism implemented on real-time traffic data with Accuracy, Precision, Recall, and F-Score as measures. Conclusions:The results indicate that the proposed mechanism is efficient when compared with the state-of-the-art schemes. Originality:Applying GMM and ThingSpeak Cloud platform to perform analysis on IoT real-time data is the first approach to find traffic density patterns on busy roads. Restrictions:There is a need to develop the application for mobile users to find the optimal traffic routes based on density patterns. The authors could not concentrate on the security aspect for finding density patterns.Introducción: el artículo es producto de la investigación "Debido al aumento en la popularidad de Internet de las cosas (IoT), se está generando una gran cantidad de datos de sensores a partir de varias aplicaciones de ciudades inteligentes", desarrollado en la Universidad de Pondicherry en el año 2019. Problema: adquirir y analizar datos generados por sensores de manera efectiva pues es un problema importante al procesar los datos. Objetivo: proponer un marco novedoso para el análisis de datos del sensor IoT utilizando el aprendizaje automático basado en mejoras desde el Modelo de mezcla gaussiana (GMM) por datos adquiridos en tiempo real. Metodología: en este documento, los modelos GMM basados en agrupamiento se utilizan para encontrar los patrones de densidad en un día o semanalmente para los requisitos del usuario. La plataforma en la nube ThingSpeak utilizada para realizar análisis y visualizaciones. Resultados: se realizó un análisis sobre el mecanismo propuesto implementado en datos de tráfico en tiempo real con precisión, recuperación y F-Score como medidas. Conclusiones: los resultados indican que el mecanismo propuesto es eficiente en comparación con el estado de esquemas de arte. Originalidad: la aplicación de la plataforma GMM y ThingSpeak Cloud para realizar análisis de datos en tiempo real de IoT es el primer enfoque para encontrar patrones de densidad de tráfico en carreteras transitadas. Limitación: existe la necesidad de desarrollar la aplicación para que los usuarios móviles encuentren las rutas de tráfico óptimas basadas en patrones de densidad. Los autores no pudieron desarrollar el aspecto de seguridad para encontrar patrones de densidad