Short Term Load Forecasting for Smart Grids Using Apache Spark and a Modified Transformer Model

Smart grid is an advanced electrical grid that enables more efficient distribution of electricity. It counters many of the problems presented by renewable energy sources such as variability in production through techniques like load forecasting and dynamic pricing. Smart grid generates massive amounts of data through smart meters, this data is used to forecast future load to adjust distribution. To process all this data, big data analysis is necessary. Most existing schemes use Apache Hadoop for big data processing and various techniques for load forecasting that include methods based on statistical theory, machine learning and deep learning. This paper proposes using Apache Spark for big data analysis and a modified version of the transformer model for forecasting load profiles of households. The modified transformer model has been tested against several state-of-the-art machine learning models. The proposed scheme was tested against several baseline and state-of-the-art machine learning models and evaluated in terms of the RMSE, MAE, MedAE and R2 scores. The obtained results show that the proposed model has better performance in terms of RMSE and R2 which are the preferred metrics when evaluating a regression model on data with a large number of outliers

Smart Grids Data Management: A Case for Cassandra

The objective of this paper is to present a SMACK based platform for microgrids data storage and management. The platform is being used in a real microgrid, with an infrastructure that monitors and controls 3 buildings within the GECAD - ISEP/IPP campus, while, at the same time, receives and manages data sources coming from different types of buildings from associated partners, to whom intelligent services are being provided. Microgrid data comes in different formats, different rates and with an increasing volume, as the microgrid itself covers more customers and areas. Based on the atual available computational resources, a Big Data platform based on the SMACK stack was implemented and is presented. The Cassandra component of the stack has evolved. AC version 2 is still supported until the version 4 release, and is often still used in production environments. However, a new stable version, version 3, introduces major optimizations in the storage that bring disk space savings. The main focus of this work is on the Data Storage and the formalization of the data mapping in Cassandra version 3, which is contextualized by means of a short example with data coming from the monitoring infrastructure of the microgrid.This work has received funding from EU Horizon 2020 R&D programme under Marie Sklodowska-Curie grant agreement No 641794 (project DREAM-GO), EUREKA - ITEA2 Project FUSE-IT (nr.13023), Project GREEDI (ANI|P2020 17822), FEDER Funds through COMPETE program and National Funds FCT under the UID/EEA/00760/2013 and SFRH/BD/103089/2014.info:eu-repo/semantics/publishedVersio

Benchmarking Big Data Technologies for Energy Procurement Efficiency

The electrical power industry is undergoing radical change due to the push for renewable energy that makes energy supply less predictable. Smart meters along with analytics software can grant insights into customer-specific consumption and thereby enable a better match between the demand and supply side for an electric utility. However, the vast amount of allocatable smart metering data and complexity of analytics pose challenges to database system. We address the implementation of an analytics ap-proach to optimize customer portfolios, eventually preventing excess energy procurement. Using real-world and simulated data, we test the suitability of big data approaches as well as traditional relational database technology. Furthermore, we present solutions based on big data platforms and demonstrate their cost effectiveness and performance. Our findings suggest economic feasibility of big data solutions for large utilities. Small and medium-sized utilities are advised to invest in more cost-effective solutions such as cluster-based systems

Novel information and data exchange within power systems using enhanced blockchain technologies

This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonCurrent energy systems are primarily designed for centralized power generation and supplying bulk electricity to users with stable and predictable usage patterns. However, with the increasing penetration of renewable energy sources (RES), future energy systems will require greater flexibility and wider distribution of both demand and supply. Integrating RES on a large scale poses challenges to the hosting capacity of distribution systems. To address these challenges, the digitalization of energy systems through novel Information and Communication Technologies (ICT) infrastructure is essential. The shift from centralized to highly distributed systems necessitates increased coordination and communication efforts. This is because a distributed system is composed of multiple independent entities that need to communicate and collaborate effectively to accomplish a shared objective. Coordination and communication are necessary to ensure that the system is operating efficiently and effectively. Traditional centralized cloud-based data exchange schemes depend on a single trusted third party, this may lead to single-point failure and lack of data privacy and access control. To overcome these issues, a novel approach is proposed for exchanging data within power systems using blockchain technology. This approach enables users to securely exchange data while maintaining ownership. The experiments conducted demonstrate that the proposed approach can handle more users and enables information and data exchange within power systems. Secondly, this thesis proposes an Artificial Neural Network (ANN) based prediction model to optimize the performance of the blockchain-enabled data exchange approach. A use case for exchanging data within the power system is implemented on the proposed platform using various performance metrics. The results of the proposed approach are compared to two other schemes: the baseline scheme and an optimized scheme. The evaluation results indicate that the proposed approach can enhance network performance when compared to the baseline and optimized schemes. In summary, the proposed novel approach to ICT infrastructure for successfully exchanging information and data within power systems entities. The performance of the novel approach is evaluated based on the ability to handle multiple users, scalability, reliability, and security

A New Framework for the Analysis of Large Scale Multi-Rate Power Data

A new framework for the analysis of large scale, multi-rate power data is introduced. The system comprises high rate power grid data acquisition devices, software modules for big data management and large scale time series analysis. The power grid modeling and simulation modules enable to run power flow simulations. Visualization methods support data exploration for captured, simulated and analyzed energy data. A remote software control module for the proposed tools is provided

Applying Big Data analytics for energy efficiency.

Global energy requirements are continuously increasing. Conventional methods of producing more energy to meet this growth pose a great threat to the environment. CO2 emissions and other bi-products of energy production and distribution processes have dire consequences for the environment. Efficient use of energy is one of the main tools to restrain energy consumption growth without compromising on the customers requirements. Improving energy efficiency requires understanding of the usage patterns and practices. Smart energy grids, pervasive computing, and communication technologies have enabled the stakeholders in the energy industry to collect large amounts of useful and highly granular energy usage data. This data is generated in large volumes and in a variety of different formats depending on its purpose and systems used to collect it. The volume and diversity of data also increase with time.\ All these data characteristics refer to the application of Big Data. This thesis focuses on harnessing the power of Big Data tools and techniques such as MapReduce and Apache Hadoop ecosystem tools to collect, process and analyse energy data and generate insights that can be used to improve energy efficiency. Furthermore, it also includes studying energy efficiency to formulate the use cases, studying Big Data technologies to present a conceptual model for an end-to-end Big Data analytics platform, implementation of a part of the conceptual model with the capacity to handle energy efficiency use cases and performing data analysis to generate useful insights. The analysis was performed on two data sets. The first data set contained hourly consumption of electricity consumed by a set of different buildings. The data was analysed to discover the seasonal and daily usage trends. The analysis also includes the classification of buildings on the basis of energy efficiency while observing the seasonal impacts on this classification. The analysis was used to build a model for segregating the energy inefficient buildings from energy efficient buildings. The second data set contained device level electricity consumption of various home appliances used in an apartment. This data was used to evaluate different prediction models to forecast future consumption on the basis of previous usage. The main purpose of this research is to provide the basis for enabling data driven decision making in organizations working to improve energy efficiency

Big Data and the Internet of Things

Advances in sensing and computing capabilities are making it possible to embed increasing computing power in small devices. This has enabled the sensing devices not just to passively capture data at very high resolution but also to take sophisticated actions in response. Combined with advances in communication, this is resulting in an ecosystem of highly interconnected devices referred to as the Internet of Things - IoT. In conjunction, the advances in machine learning have allowed building models on this ever increasing amounts of data. Consequently, devices all the way from heavy assets such as aircraft engines to wearables such as health monitors can all now not only generate massive amounts of data but can draw back on aggregate analytics to "improve" their performance over time. Big data analytics has been identified as a key enabler for the IoT. In this chapter, we discuss various avenues of the IoT where big data analytics either is already making a significant impact or is on the cusp of doing so. We also discuss social implications and areas of concern.Comment: 33 pages. draft of upcoming book chapter in Japkowicz and Stefanowski (eds.) Big Data Analysis: New algorithms for a new society, Springer Series on Studies in Big Data, to appea

EMPOWERING, a smart Big Data framework for sustainable electricity suppliers

This paper presents the EMPOWERING project, a Big Data environment aimed at helping domestic customers to save electricity by managing their consumption positively. This is achieved by improving the information received about energy bills and offering online tools. The main contributions of EMPOWERING are the creation of a novel workflow in the electricity utility sector regarding the implementation of data analytics for their customers and the fast implementation of data-mining techniques in massive datasets within a Big Data platform to achieve scalability. The results obtained show that EMPOWERING can be of use for customers of electrical suppliers by changing their energy habits to decrease consumption and so increase environmental sustainability