3D Power-map for Smart Grids---An Integration of High-dimensional Analysis and Visualization

Data with features of volume, velocity, variety, and veracity are challenging traditional tools to extract useful analysis for decision-making. By integrating high-dimensional analysis with visualization, this paper develops a 3D power-map animation as an effective solution to the challenge. An architecture design, with detailed data processing procedure, is proposed to realize the integration. Two of the most important components in the architecture are presented: the Single-Ring Law for random matrices as solid mathematic foundation, and the proposed statistical index MSR as high-dimensional data for visualization. The whole procedure is easy in logic, fast in speed, objective and even robust against bad data. Moreover, it is an unsupervised machine learning mechanism directly oriented to the raw data rather than logics or models based on simplifications and assumptions. A case study validates the effectiveness and performance of the developed 3D power-map in analysis extraction.Comment: 5 pages, 7 figures, submitted to PESGM 2015. arXiv admin note: substantial text overlap with arXiv:1502.0006

A Random Matrix Theoretical Approach to Early Event Detection in Smart Grid

Power systems are developing very fast nowadays, both in size and in complexity; this situation is a challenge for Early Event Detection (EED). This paper proposes a data- driven unsupervised learning method to handle this challenge. Specifically, the random matrix theories (RMTs) are introduced as the statistical foundations for random matrix models (RMMs); based on the RMMs, linear eigenvalue statistics (LESs) are defined via the test functions as the system indicators. By comparing the values of the LES between the experimental and the theoretical ones, the anomaly detection is conducted. Furthermore, we develop 3D power-map to visualize the LES; it provides a robust auxiliary decision-making mechanism to the operators. In this sense, the proposed method conducts EED with a pure statistical procedure, requiring no knowledge of system topologies, unit operation/control models, etc. The LES, as a key ingredient during this procedure, is a high dimensional indictor derived directly from raw data. As an unsupervised learning indicator, the LES is much more sensitive than the low dimensional indictors obtained from supervised learning. With the statistical procedure, the proposed method is universal and fast; moreover, it is robust against traditional EED challenges (such as error accumulations, spurious correlations, and even bad data in core area). Case studies, with both simulated data and real ones, validate the proposed method. To manage large-scale distributed systems, data fusion is mentioned as another data processing ingredient.Comment: 12 pages, 11 figures, submitted to IEEE Transactions on Smart Gri

Spatio-Temporal Big Data Analysis for Smart Grids Based on Random Matrix Theory: A Comprehensive Study

A cornerstone of the smart grid is the advanced monitorability on its assets and operations. Increasingly pervasive installation of the phasor measurement units (PMUs) allows the so-called synchrophasor measurements to be taken roughly 100 times faster than the legacy supervisory control and data acquisition (SCADA) measurements, time-stamped using the global positioning system (GPS) signals to capture the grid dynamics. On the other hand, the availability of low-latency two-way communication networks will pave the way to high-precision real-time grid state estimation and detection, remedial actions upon network instability, and accurate risk analysis and post-event assessment for failure prevention. In this chapter, we firstly modelling spatio-temporal PMU data in large scale grids as random matrix sequences. Secondly, some basic principles of random matrix theory (RMT), such as asymptotic spectrum laws, transforms, convergence rate and free probability, are introduced briefly in order to the better understanding and application of RMT technologies. Lastly, the case studies based on synthetic data and real data are developed to evaluate the performance of the RMT-based schemes in different application scenarios (i.e., state evaluation and situation awareness).Comment: Book chapter#23 for the book "Transportation and Power Grid in Smart Cities: Communication Networks and Services". arXiv admin note: text overlap with arXiv:1302.0885 by other author

Designing for Situation Awareness of Future Power Grids: An Indicator System Based on Linear Eigenvalue Statistics of Large Random Matrices

Future power grids are fundamentally different from current ones, both in size and in complexity; this trend imposes challenges for situation awareness (SA) based on classical indicators, which are usually model-based and deterministic. As an alternative, this paper proposes a statistical indicator system based on linear eigenvalue statistics (LESs) of large random matrices: 1) from a data modeling viewpoint, we build, starting from power flows equations, the random matrix models (RMMs) only using the real-time data flow in a statistical manner; 2) for a data analysis that is fully driven from RMMs, we put forward the high-dimensional indicators, called LESs that have some unique statistical features such as Gaussian properties; and 3) we develop a three-dimensional (3D) power-map to visualize the system, respectively, from a high-dimensional viewpoint and a low-dimensional one. Therefore, a statistical methodology of SA is employed; it conducts SA with a model-free and data-driven procedure, requiring no knowledge of system topologies, units operation/control models, causal relationship, etc. This methodology has numerous advantages, such as sensitivity, universality, speed, and flexibility. In particular, its robustness against bad data is highlighted, with potential advantages in cyber security. The theory of big data based stability for on-line operations may prove feasible along with this line of work, although this critical development will be reported elsewhere.Comment: 8 pages, 8 figures, 3 table

A Random Matrix Theoretical Approach to Early Event Detection Using Experimental Data

In this paper, High-dimensional data analysis methods are proposed to deal with random matrix which is composed by the real data from power network before and after the fault. The mean spectral radius (MSR) of non-Hermitian random matrices is defined as a statistic analytic for the fault detection. By analyzing the characteristics of random matrices and observing the changes of the spectral radius of random matrices, grid failure detection will be achieved. This paper describes the basic mathematical theory of this big data method, and the real-world data of a certain China power grid is used to verify the methods.Comment: 4 pages, 6 figure

Internet of Things for Residential Areas: Toward Personalized Energy Management Using Big Data

Intelligent management of machines, particularly in a residence area, has been of interest for many years. However, such system design has always been limited to simple control of machines from a local area or remotely from the Internet. In this report, for the first time, an intelligent system is proposed, where not only provides intelligent control ability of machines to user, but also utilizes big data and optimization techniques to provide promotional offers to the user to optimize energy consumption of machines. Since a high traffic communication is involved among the machines and the optimization-big data core of system, the communication core of the proposed system is designed based on cloud, where many challenging issues such as spectrum assignment and resource management are involved. To deal with that, the communication network in the home area network (HAN) is designed based on the cognitive radio system, where a new spectrum assignment method based on the ant colony optimization (ACO) algorithm is proposed to perform spectrum assignment to the machines in the HAN. Performance evaluation of the proposed spectrum assignment method shows its performance in fair spectrum assignment among machines.Comment: Draft of technical report. Limited version under preparation for submissio

A Data-driven Approach to Multi-event Analytics in Large-scale Power Systems Using Factor Model

Multi-event detection and recognition in real time is of challenge for a modern grid as its feature is usually non-identifiable. Based on factor model, this paper porposes a data-driven method as an alternative solution under the framework of random matrix theory. This method maps the raw data into a high-dimensional space with two parts: 1) the principal components (factors, mapping event signals); and 2) time series residuals (bulk, mapping white/non-Gaussian noises). The spatial information is extracted form factors, and the termporal infromation from residuals. Taking both spatial-tempral correlation into account, this method is able to reveal the multi-event: its components and their respective details, e.g., occurring time. Case studies based on the standard IEEE 118-bus system validate the proposed method.Comment: 7 pages, 2 figure

A Correlation Analysis Method for Power Systems Based on Random Matrix Theory

The operating status of power systems is influenced by growing varieties of factors, resulting from the developing sizes and complexity of power systems; in this situation, the modelbased methods need be revisited. A data-driven method, as the novel alternative, on the other hand, is proposed in this paper: it reveals the correlations between the factors and the system status through statistical properties of data. An augmented matrix, as the data source, is the key trick for this method; it is formulated by two parts: 1) status data as the basic part, and 2) factor data as the augmented part. The random matrix theory (RMT) is applied as the mathematical framework. The linear eigenvalue statistics (LESs), such as the mean spectral radius (MSR), are defined to study data correlations through large random matrices. Compared with model-based methods, the proposed method is inspired by a pure statistical approach, without a prior knowledge of operation and interaction mechanism models for power systems and factors. In general, this method is direct in analysis, robust against bad data, universal to various factors, and applicable for real-time analysis. A case study, based on the standard IEEE 118-bus system, validates the proposed method.Comment: 9 pages, 9 figures, Accepted by IEEE Trans on Smart Gri

Invisible Units Detection and Estimation Based on Random Matrix Theory

Invisible units mainly refer to small-scale units that are not monitored by, and thus are not visible to utilities. Integration of these invisible units into power systems does significantly affect the way in which a distribution grid is planned and operated. This paper, based on random matrix theory (RMT), proposes a statistical, data-driven framework to handle the massive grid data, in contrast to its deterministic, model-based counterpart. Combining the RMT-based data-mining framework with conventional techniques, some heuristics are derived as the solution to the invisible units detection and estimation task: linear eigenvalue statistic indicators (LESs) are suggested as the main ingredients of the solution; according to the statistical properties of LESs, the hypothesis testing is formulated to conduct change point detection in the high-dimensional space. The proposed method is promising for anomaly detection and pertinent to current distribution networks---it is capable of detecting invisible power usage and fraudulent behavior while even being able to locate the suspect's location. Case studies, using both simulated data and actual data, validate the proposed method.Comment: 10 pages,Accepted by IEEE Transaction on Power System

Review of Smart Meter Data Analytics: Applications, Methodologies, and Challenges

The widespread popularity of smart meters enables an immense amount of fine-grained electricity consumption data to be collected. Meanwhile, the deregulation of the power industry, particularly on the delivery side, has continuously been moving forward worldwide. How to employ massive smart meter data to promote and enhance the efficiency and sustainability of the power grid is a pressing issue. To date, substantial works have been conducted on smart meter data analytics. To provide a comprehensive overview of the current research and to identify challenges for future research, this paper conducts an application-oriented review of smart meter data analytics. Following the three stages of analytics, namely, descriptive, predictive and prescriptive analytics, we identify the key application areas as load analysis, load forecasting, and load management. We also review the techniques and methodologies adopted or developed to address each application. In addition, we also discuss some research trends, such as big data issues, novel machine learning technologies, new business models, the transition of energy systems, and data privacy and security.Comment: IEEE Transactions on Smart Grid, 201