Parallel detrended fluctuation analysis for fast event detection on massive PMU data

("(c) 2015 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works.")Phasor measurement units (PMUs) are being rapidly deployed in power grids due to their high sampling rates and synchronized measurements. The devices high data reporting rates present major computational challenges in the requirement to process potentially massive volumes of data, in addition to new issues surrounding data storage. Fast algorithms capable of processing massive volumes of data are now required in the field of power systems. This paper presents a novel parallel detrended fluctuation analysis (PDFA) approach for fast event detection on massive volumes of PMU data, taking advantage of a cluster computing platform. The PDFA algorithm is evaluated using data from installed PMUs on the transmission system of Great Britain from the aspects of speedup, scalability, and accuracy. The speedup of the PDFA in computation is initially analyzed through Amdahl's Law. A revision to the law is then proposed, suggesting enhancements to its capability to analyze the performance gain in computation when parallelizing data intensive applications in a cluster computing environment

Parallel detrended fluctuation analysis for fast event detection on massive PMU data

("(c) 2015 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works.")Phasor measurement units (PMUs) are being rapidly deployed in power grids due to their high sampling rates and synchronized measurements. The devices high data reporting rates present major computational challenges in the requirement to process potentially massive volumes of data, in addition to new issues surrounding data storage. Fast algorithms capable of processing massive volumes of data are now required in the field of power systems. This paper presents a novel parallel detrended fluctuation analysis (PDFA) approach for fast event detection on massive volumes of PMU data, taking advantage of a cluster computing platform. The PDFA algorithm is evaluated using data from installed PMUs on the transmission system of Great Britain from the aspects of speedup, scalability, and accuracy. The speedup of the PDFA in computation is initially analyzed through Amdahl's Law. A revision to the law is then proposed, suggesting enhancements to its capability to analyze the performance gain in computation when parallelizing data intensive applications in a cluster computing environment

Wide-Area Synchrophasor Data Server System and Data Analytics Platform

As synchrophasor data start to play a significant role in power system operation and dynamic study, data processing and data analysis capability are critical to Wide-area measurement systems (WAMS). The Frequency Monitoring Network (FNET/GridEye) is a WAMS network that collects data from hundreds of Frequency Disturbance Recorders (FDRs) at the distribution level. The previous FNET/GridEye data center is limited by its data storage capability and computation power. Targeting scalability, extensibility, concurrency and robustness, a distributed data analytics platform is proposed to process large volume, high velocity dataset. A variety of real-time and non-real-time synchrophasor data analytics applications are hosted by this platform. The computation load is shared with balance by multiple nodes of the analytics cluster, and big data analytics tools such as Apache Spark are adopted to manage large volume data and to boost the data processing speed. Multiple power system disturbance detection and analysis applications are redesigned to take advantage of this platform. Data quality and data security are monitored in real-time. Future data analytics applications can be easily developed and plugged into the system with simple configuration

Exploiting phasor measurement units for enhanced transmission network operation and control

This thesis was submitted for the degree of Doctor of Engineering and awarded by Brunel UniversityIn order to achieve binding Government targets towards the decarbonisation of the electricity network, the GB power system is undergoing an unprecedented amount of change. A series of new technologies designed to integrate massive volumes of renewable generation, predominantly in the form of offshore wind, asynchronously connecting to the periphery of the transmission system, are transforming the requirements of the network. This displacement of traditional thermal generation is leading to a significant reduction in system inertia, thus making the task of system operation more challenging. It is therefore deemed necessary to develop tools and technologies that provide far greater insight into the state of the power system in real-time and give rise to methods for improving offline modelling practices through an enhanced understanding of the systems performance. To that extent PMUs are seen as one of the key enablers of the Smart Grid, providing accurate time-synchronised measurements on the state of the power system, allowing the true dynamics of the power system to be captured and analysed. This thesis provides an analysis of the existing PMU deployment on the GB transmission system with a view to the future system monitoring requirements. A critical evaluation and comparison is also provided on the suitability of a University based Low Voltage PMU network to further enhance the visibility of the GB system. In addition a novel event detection algorithm based on Detrended Fluctuation Analysis is developed and demonstrated, designed to determine the exact start time of a transmission event, as well as the suitability of such an event for additional transmission system analysis, namely inertia estimation. Finally, a reliable method for the estimation of total system inertia is proposed that includes an estimate of the contribution from residual sources, of which there is currently no visibility. The proposed method identifies the importance of regional inertia and its impact to the operation of the GB transmission system.Engineering and Physical Sciences Research Council (EPSRC) and National Grid

Hadoop performance modeling and job optimization for big data analytics

This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonBig data has received a momentum from both academia and industry. The MapReduce model has emerged into a major computing model in support of big data analytics. Hadoop, which is an open source implementation of the MapReduce model, has been widely taken up by the community. Cloud service providers such as Amazon EC2 cloud have now supported Hadoop user applications. However, a key challenge is that the cloud service providers do not a have resource provisioning mechanism to satisfy user jobs with deadline requirements. Currently, it is solely the user responsibility to estimate the require amount of resources for their job running in a public cloud. This thesis presents a Hadoop performance model that accurately estimates the execution duration of a job and further provisions the required amount of resources for a job to be completed within a deadline. The proposed model employs Locally Weighted Linear Regression (LWLR) model to estimate execution time of a job and Lagrange Multiplier technique for resource provisioning to satisfy user job with a given deadline. The performance of the propose model is extensively evaluated in both in-house Hadoop cluster and Amazon EC2 Cloud. Experimental results show that the proposed model is highly accurate in job execution estimation and jobs are completed within the required deadlines following on the resource provisioning scheme of the proposed model. In addition, the Hadoop framework has over 190 configuration parameters and some of them have significant effects on the performance of a Hadoop job. Manually setting the optimum values for these parameters is a challenging task and also a time consuming process. This thesis presents optimization works that enhances the performance of Hadoop by automatically tuning its parameter values. It employs Gene Expression Programming (GEP) technique to build an objective function that represents the performance of a job and the correlation among the configuration parameters. For the purpose of optimization, Particle Swarm Optimization (PSO) is employed to find automatically an optimal or a near optimal configuration settings. The performance of the proposed work is intensively evaluated on a Hadoop cluster and the experimental results show that the proposed work enhances the performance of Hadoop significantly compared with the default settings.Abdul Wali Khan University Marda

Smart Grids Data Management: A Case for Cassandra

The objective of this paper is to present a SMACK based platform for microgrids data storage and management. The platform is being used in a real microgrid, with an infrastructure that monitors and controls 3 buildings within the GECAD - ISEP/IPP campus, while, at the same time, receives and manages data sources coming from different types of buildings from associated partners, to whom intelligent services are being provided. Microgrid data comes in different formats, different rates and with an increasing volume, as the microgrid itself covers more customers and areas. Based on the atual available computational resources, a Big Data platform based on the SMACK stack was implemented and is presented. The Cassandra component of the stack has evolved. AC version 2 is still supported until the version 4 release, and is often still used in production environments. However, a new stable version, version 3, introduces major optimizations in the storage that bring disk space savings. The main focus of this work is on the Data Storage and the formalization of the data mapping in Cassandra version 3, which is contextualized by means of a short example with data coming from the monitoring infrastructure of the microgrid.This work has received funding from EU Horizon 2020 R&D programme under Marie Sklodowska-Curie grant agreement No 641794 (project DREAM-GO), EUREKA - ITEA2 Project FUSE-IT (nr.13023), Project GREEDI (ANI|P2020 17822), FEDER Funds through COMPETE program and National Funds FCT under the UID/EEA/00760/2013 and SFRH/BD/103089/2014.info:eu-repo/semantics/publishedVersio

Systems For Delivering Electric Vehicle Data Analytics

n the recent times, advances in scientific research related to electric vehicles led to generation of large amounts of data. This data is majorly logger data collected from various sensors in the vehicle. It is predominantly unstructured and non-relational in nature, also called Big Data. Analysis of such data needs a high performance information technology infrastructure that provides superior computational efficiency and storage capacity. It should be scalable to accommodate the growing data and ensure its security over a network. This research proposes an architecture built over Hadoop to effectively support distributed data management over a network for real-time data collection and storage, parallel processing, and faster random read access for information retrieval for decision-making. Once imported into a database, the system can support efficient analysis and visualization of data as per user needs. These analytics can help understand correlations between data parameters under various circumstances. This system provides scalability to support data accumulation in the future and still perform analytics with less overhead. Overall, these open problems in EV data analytics are taken into consideration and a low-cost architecture for data management is researched

Advanced Wide-Area Monitoring System Design, Implementation, and Application

Wide-area monitoring systems (WAMSs) provide an unprecedented way to collect, store and analyze ultra-high-resolution synchrophasor measurements to improve the dynamic observability in power grids. This dissertation focuses on designing and implementing a wide-area monitoring system and a series of applications to assist grid operators with various functionalities. The contributions of this dissertation are below: First, a synchrophasor data collection system is developed to collect, store, and forward GPS-synchronized, high-resolution, rich-type, and massive-volume synchrophasor data. a distributed data storage system is developed to store the synchrophasor data. A memory-based cache system is discussed to improve the efficiency of real-time situation awareness. In addition, a synchronization system is developed to synchronize the configurations among the cloud nodes. Reliability and Fault-Tolerance of the developed system are discussed. Second, a novel lossy synchrophasor data compression approach is proposed. This section first introduces the synchrophasor data compression problem, then proposes a methodology for lossy data compression, and finally presents the evaluation results. The feasibility of the proposed approach is discussed. Third, a novel intelligent system, SynchroService, is developed to provide critical functionalities for a synchrophasor system. Functionalities including data query, event query, device management, and system authentication are discussed. Finally, the resiliency and the security of the developed system are evaluated. Fourth, a series of synchrophasor-based applications are developed to utilize the high-resolution synchrophasor data to assist power system engineers to monitor the performance of the grid as well as investigate the root cause of large power system disturbances. Lastly, a deep learning-based event detection and verification system is developed to provide accurate event detection functionality. This section introduces the data preprocessing, model design, and performance evaluation. Lastly, the implementation of the developed system is discussed