A Novel Real-Time Edge-Cloud Big Data Management and Analytics Framework for Smart Cities

Exposing city information to dynamic, distributed, powerful, scalable, and user-friendly big data systems is expected to enable the implementation of a wide range of new opportunities; however, the size, heterogeneity and geographical dispersion of data often makes it difficult to combine, analyze and consume them in a single system. In the context of the H2020 CLASS project, we describe an innovative framework aiming to facilitate the design of advanced big-data analytics workflows. The proposal covers the whole compute continuum, from edge to cloud, and relies on a well-organized distributed infrastructure exploiting: a) edge solutions with advanced computer vision technologies enabling the real-time generation of “rich” data from a vast array of sensor types; b) cloud data management techniques offering efficient storage, real-time querying and updating of the high-frequency incoming data at different granularity levels. We specifically focus on obstacle detection and tracking for edge processing, and consider a traffic density monitoring application, with hierarchical data aggregation features for cloud processing; the discussed techniques will constitute the groundwork enabling many further services. The tests are performed on the real use-case of the Modena Automotive Smart Area (MASA)

A study of multivariate behavior and anomaly patterns : tensor decomposition for multiway big data

University of Technology Sydney. Faculty of Engineering and Information Technology.A vast majority of the today’s information haul is through Cyber-Physical Systems (CPS). They represent the confluence of extensive data sets, tight time-constraints, latency issues and heterogeneous components. CPS architectures demand newer Big Data processing approaches. Typical applications span from the Internet-of-Things, across the World Wide Web to Smart Cities and Intelligent machines. A standard heterogeneous CPS installation, the Smart Energy Grid, is observed and the logistics are analyzed. The Smart Grid domain is weighed down by lack of unifying framework and systemic intelligence for autonomic management. Preliminary studies of the field under investigation shows how processing of Real-Time data, communication and control signaling is vital. Purely autonomic system governance is proven to be different from the contemporary definition. It takes the form of Interoperability (achieved through automation) instead of elemental Integration. That means autonomic (smart) management requires all elements to have fully controllable behavior. This dissertation teste the hypothesis of applying Tensor decompositions and Factorizations - a momentum-gaining arithmetic tool - to this problem. The aim is to validate the prospects of higher order Anomaly Pattern Processing to capture intelligence along multiple modes of data flow. Tensorial Data representation captures information flows in Big Data, while Multivariate Anomaly Detection performs tracking of the time-series behavioral changes. Together, they implement Autonomic management in CPS super-systems. Uniqueness of this approach is highlighted by the novel multi-modal data flow imaging and models. Requirements of traditional anomalous event definition and cataloging in Data streams are removed. Tensor algebra is then studied for the scope of implementation concerning features, significance, and interpretation in terms of multi-modal data. Standard Decomposition rules and their derivatives, literature analysis on contemporary applications of Tensor algebra, and its scope on prominent real-world data processing problems are studied. Finally, the decomposition tool for Multi-way analysis is inferred, and proposed methodology is derived. The Smart Grid Smart City Project commissioned by the Australian government is chosen as the data source investigated. The need for exhaustive examination of such repositories in the CPS Anomaly Detection context is also highlighted. Experimentation is done by applying Tensor Decomposition on the data set after normalization and pre-processing. Details of those phases, as well as the choice of coding platforms, the design of experimental frameworks, timelines estimated, and testing operations, are included in this work. The outcomes are the defined patterns extracted and their analysis-interpretation defended by proofs from actual events of the Project Trial phase

Big Data and the Internet of Things

Advances in sensing and computing capabilities are making it possible to embed increasing computing power in small devices. This has enabled the sensing devices not just to passively capture data at very high resolution but also to take sophisticated actions in response. Combined with advances in communication, this is resulting in an ecosystem of highly interconnected devices referred to as the Internet of Things - IoT. In conjunction, the advances in machine learning have allowed building models on this ever increasing amounts of data. Consequently, devices all the way from heavy assets such as aircraft engines to wearables such as health monitors can all now not only generate massive amounts of data but can draw back on aggregate analytics to "improve" their performance over time. Big data analytics has been identified as a key enabler for the IoT. In this chapter, we discuss various avenues of the IoT where big data analytics either is already making a significant impact or is on the cusp of doing so. We also discuss social implications and areas of concern.Comment: 33 pages. draft of upcoming book chapter in Japkowicz and Stefanowski (eds.) Big Data Analysis: New algorithms for a new society, Springer Series on Studies in Big Data, to appea

Intelligent Management and Efficient Operation of Big Data

This chapter details how Big Data can be used and implemented in networking and computing infrastructures. Specifically, it addresses three main aspects: the timely extraction of relevant knowledge from heterogeneous, and very often unstructured large data sources, the enhancement on the performance of processing and networking (cloud) infrastructures that are the most important foundational pillars of Big Data applications or services, and novel ways to efficiently manage network infrastructures with high-level composed policies for supporting the transmission of large amounts of data with distinct requisites (video vs. non-video). A case study involving an intelligent management solution to route data traffic with diverse requirements in a wide area Internet Exchange Point is presented, discussed in the context of Big Data, and evaluated.Comment: In book Handbook of Research on Trends and Future Directions in Big Data and Web Intelligence, IGI Global, 201

Middleware Technologies for Cloud of Things - a survey

The next wave of communication and applications rely on the new services provided by Internet of Things which is becoming an important aspect in human and machines future. The IoT services are a key solution for providing smart environments in homes, buildings and cities. In the era of a massive number of connected things and objects with a high grow rate, several challenges have been raised such as management, aggregation and storage for big produced data. In order to tackle some of these issues, cloud computing emerged to IoT as Cloud of Things (CoT) which provides virtually unlimited cloud services to enhance the large scale IoT platforms. There are several factors to be considered in design and implementation of a CoT platform. One of the most important and challenging problems is the heterogeneity of different objects. This problem can be addressed by deploying suitable "Middleware". Middleware sits between things and applications that make a reliable platform for communication among things with different interfaces, operating systems, and architectures. The main aim of this paper is to study the middleware technologies for CoT. Toward this end, we first present the main features and characteristics of middlewares. Next we study different architecture styles and service domains. Then we presents several middlewares that are suitable for CoT based platforms and lastly a list of current challenges and issues in design of CoT based middlewares is discussed.Comment: http://www.sciencedirect.com/science/article/pii/S2352864817301268, Digital Communications and Networks, Elsevier (2017