Load curve data cleansing and imputation via sparsity and low rank

The smart grid vision is to build an intelligent power network with an unprecedented level of situational awareness and controllability over its services and infrastructure. This paper advocates statistical inference methods to robustify power monitoring tasks against the outlier effects owing to faulty readings and malicious attacks, as well as against missing data due to privacy concerns and communication errors. In this context, a novel load cleansing and imputation scheme is developed leveraging the low intrinsic-dimensionality of spatiotemporal load profiles and the sparse nature of "bad data.'' A robust estimator based on principal components pursuit (PCP) is adopted, which effects a twofold sparsity-promoting regularization through an $\ell_1$-norm of the outliers, and the nuclear norm of the nominal load profiles. Upon recasting the non-separable nuclear norm into a form amenable to decentralized optimization, a distributed (D-) PCP algorithm is developed to carry out the imputation and cleansing tasks using networked devices comprising the so-termed advanced metering infrastructure. If D-PCP converges and a qualification inequality is satisfied, the novel distributed estimator provably attains the performance of its centralized PCP counterpart, which has access to all networkwide data. Computer simulations and tests with real load curve data corroborate the convergence and effectiveness of the novel D-PCP algorithm.Comment: 8 figures, submitted to IEEE Transactions on Smart Grid - Special issue on "Optimization methods and algorithms applied to smart grid

Recent Techniques for Regularization in Partial Differential Equations and Imaging

abstract: Inverse problems model real world phenomena from data, where the data are often noisy and models contain errors. This leads to instabilities, multiple solution vectors and thus ill-posedness. To solve ill-posed inverse problems, regularization is typically used as a penalty function to induce stability and allow for the incorporation of a priori information about the desired solution. In this thesis, high order regularization techniques are developed for image and function reconstruction from noisy or misleading data. Specifically the incorporation of the Polynomial Annihilation operator allows for the accurate exploitation of the sparse representation of each function in the edge domain. This dissertation tackles three main problems through the development of novel reconstruction techniques: (i) reconstructing one and two dimensional functions from multiple measurement vectors using variance based joint sparsity when a subset of the measurements contain false and/or misleading information, (ii) approximating discontinuous solutions to hyperbolic partial differential equations by enhancing typical solvers with l1 regularization, and (iii) reducing model assumptions in synthetic aperture radar image formation, specifically for the purpose of speckle reduction and phase error correction. While the common thread tying these problems together is the use of high order regularization, the defining characteristics of each of these problems create unique challenges. Fast and robust numerical algorithms are also developed so that these problems can be solved efficiently without requiring fine tuning of parameters. Indeed, the numerical experiments presented in this dissertation strongly suggest that the new methodology provides more accurate and robust solutions to a variety of ill-posed inverse problems.Dissertation/ThesisDoctoral Dissertation Mathematics 201

Power System State Estimation and Renewable Energy Optimization in Smart Grids

The future smart grid will benefit from real-time monitoring, automated outage management, increased renewable energy penetration, and enhanced consumer involvement. Among the many research areas related to smart grids, this dissertation will focus on two important topics: power system state estimation using phasor measurement units (PMUs), and optimization for renewable energy integration. In the first topic, we consider power system state estimation using PMUs, when phase angle mismatch exists in the measurements. In particular, we build a measurement model that takes into account the measurement phase angle mismatch. We then propose algorithms to increase state estimation accuracy by taking into account the phase angle mismatch. Based on the proposed measurement model, we derive the posterior Cramér-Rao bound on the estimation error, and propose a method for PMU placement in the grid. Using numerical examples, we show that by considering the phase angle mismatch in the measurements, the estimation accuracy can be significantly improved compared with the traditional weighted least-squares estimator or Kalman filtering. We also show that using the proposed PMU placement strategy can increase the estimation accuracy by placing a limited number of PMUs in proper locations. In the second topic, we consider optimization for renewable energy integration in smart grids. We first consider a scenario where individual energy users own on-site renewable generators, and can both purchase and sell electricity to the main grid. Under this setup, we develop a method for parallel load scheduling of different energy users, with the goal of reducing the overall cost to energy users as well as to energy providers. The goal is achieved by finding the optimal load schedule of each individual energy user in a parallel distributed manner, to flatten the overall load of all the energy users. We then consider the case of a micro-grid, or an isolated grid, with a large penetration of renewable energy. In this case, we jointly optimize the energy storage and renewable generator capacity, in order to ensure an uninterrupted power supply with minimum costs. To handle the large dimensionality of the problem due to large historical datasets used, we reformulate the original optimization problem as a consensus problem, and use the alternating direction method of multipliers to solve for the optimal solution in a distributed manner

In-situ Data Analytics In Cyber-Physical Systems

Cyber-Physical System (CPS) is an engineered system in which sensing, networking, and computing are tightly coupled with the control of the physical entities. To enable security, scalability and resiliency, new data analytics methodologies are required for computing, monitoring and optimization in CPS. This work investigates the data analytics related challenges in CPS through two study cases: Smart Grid and Seismic Imaging System. For smart grid, this work provides a complete solution for system management based on novel in-situ data analytics designs. We first propose methodologies for two important tasks of power system monitoring: grid topology change and power-line outage detection. To address the issue of low measurement redundancy in topology identification, particularly in the low-level distribution network, we develop a maximum a posterior based mechanism, which is capable of embedding prior information on the breakers status to enhance the identification accuracy. In power-line outage detection, existing approaches suer from high computational complexity and security issues raised from centralized implementation. Instead, this work presents a distributed data analytics framework, which carries out in-network processing and invokes low computational complexity, requiring only simple matrix-vector multiplications. To complete the system functionality, we also propose a new power grid restoration strategy involving data analytics for topology reconfiguration and resource planning after faults or changes. In seismic imaging system, we develop several innovative in-situ seismic imaging schemes in which each sensor node computes the tomography based on its partial information and through gossip with local neighbors. The seismic data are generated in a distributed fashion originally. Dierent from the conventional approach involving data collection and then processing in order, our proposed in-situ data computing methodology is much more ecient. The underlying mechanisms avoid the bottleneck problem on bandwidth since all the data are processed distributed in nature and only limited decisional information is communicated. Furthermore, the proposed algorithms can deliver quicker insights than the state-of-arts in seismic imaging. Hence they are more promising solutions for real-time in-situ data analytics, which is highly demanded in disaster monitoring related applications. Through extensive experiments, we demonstrate that the proposed data computing methods are able to achieve near-optimal high quality seismic tomography, retain low communication cost, and provide real-time seismic data analytics

The design and demonstration of an advanced data collection/position locating system, addendum

The theoretical background for a coherent demodulator for minimum shift keying signals generated by the advanced data collection/position locating system breadboard is presented along with a discussion of the design concept. Various tests and test results, obtained with the breadboard system described, include evaluation of bit-error rate performance, acquisition time, clock recovery, recycle time, frequency measurement accuracy, and mutual interference

Techniques for low jitter clock multiplication

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2008.Includes bibliographical references (p. 115-121).Phase realigning clock multipliers, such as Multiplying Delay-Locked Loops (MDLL), offer significantly reduced random jitter compared to typical Phase-Locked Loops (PLL). This is achieved by introducing the reference signal directly into their voltage controlled oscillators (VCO) to realign the phase to the clean reference. However, the typical cost of this benefit is a significant increase in deterministic jitter due to path mismatch in the detector as well as analog nonidealities in the tuning circuits. This thesis proposes a mostly-digital tuning technique that drastically reduces deterministic jitter in phase realigning clock multipliers. The proposed technique eliminates path mismatch by using a single-path digital detection method that leverages a scrambling time-to-digital converter (TDC) and correlated double sampling to infer the tuning error from the difference in cycle periods of the output. By using a digital loop filter that consists of a digital accumulator, the tuning technique avoids the analog nonidealities of typical tuning paths. The scrambling TDC is not a contribution of this thesis. A highly-digital MDLL prototype that uses the proposed tuning technique consists of two custom 0.13 [mu]m ICs, an FPGA board, a discrete digital-to-analog converter (DAC) with effective 8 bits, and a simple RC filter. The measured performance (for a 1.6 GHz output and 50 MHz reference) demonstrated an overall jitter of 0.93 ps rms, and estimated random and deterministic jitter of 0.68 ps rms and 0.76 ps peak-to-peak, respectively. The proposed MDLL architecture is especially suitable for digital ICs, since its highly-digital architecture is mostly compatible with digital design flows, which eases its porting between technologies.by Belal Moheedin Helal.Ph.D

Performance Improvement of Wide-Area-Monitoring-System (WAMS) and Applications Development

Wide area monitoring system (WAMS), as an application of situation awareness, provides essential information for power system monitoring, planning, operation, and control. To fully utilize WAMS in smart grid, it is important to investigate and improve its performance, and develop advanced applications based on the data from WAMS. In this dissertation, the work on improving the WAMS performance and developing advanced applications are introduced.To improve the performance of WAMS, the work includes investigation of the impacts of measurement error and the requirements of system based on WAMS, and the solutions. PMU is one of the main sensors for WAMS. The phasor and frequency estimation algorithms implemented highly influence the performance of PMUs, and therefore the WAMS. The algorithms of PMUs are reviewed in Chapter 2. To understand how the errors impact WAMS application, different applications are investigated in Chapter 3, and their requirements of accuracy are given. In chapter 4, the error model of PMUs are developed, regarding different parameters of input signals and PMU operation conditions. The factors influence of accuracy of PMUs are analyzed in Chapter 5, including both internal and external error sources. Specifically, the impacts of increase renewables are analyzed. Based on the analysis above, a novel PMU is developed in Chapter 6, including algorithm and realization. This PMU is able to provide high accurate and fast responding measurements during both steady and dynamic state. It is potential to improve the performance of WAMS. To improve the interoperability, the C37.118.2 based data communication protocol is curtailed and realized for single-phase distribution-level PMUs, which are presented in Chapter 7.WAMS-based applications are developed and introduced in Chapter 8-10. The first application is to use the spatial and temporal characterization of power system frequency for data authentication, location estimation and the detection of cyber-attack. The second application is to detect the GPS attack on the synchronized time interval. The third application is to detect the geomagnetically induced currents (GIC) resulted from GMD and EMP-E3. These applications, benefited from the novel PMU proposed in Chapter 6, can be used to enhance the security and robust of power system

Efficient implementations of high-resolution wideband FFT-spectrometers and their application to an APEX Galactic Center line survey

Spectroscopy has been a major technique in radioastronomy for decades and spectrometers are used in a wide range of radioastronomical applications. With more stable receivers that are wider in bandwidth, spectrometers are required that possess both wide bandwidth and high spectral resolution. The availability of analog-to-digital converters (ADCs) that sample a signal at rates of multiple GHz allowed the development of a novel type of spectrometer. The fast Fourier transform spectrometer (FFTS) digitizes a radio signal and calculates its power spectrum at high speed. The increased complexity of field-programmable gate arrays (FPGAs) provides the processing power necessary for such high-speed operation at a low price and with high flexibility. However, to fully utilize the speed and flexibility offered by FPGAs and to achieve a bandwidth of 2.5 GHz with up to 65536 channels, it is necessary to develop efficient algorithms that are optimized for FPGA-based implementation. This thesis first explains the basic principles behind an FFTS. Then it describes the requirements of astronomical applications that utilize FFTSs and evaluates their requirements. Besides the main application of wideband spectroscopy, the demands of high resolution spectroscopy, of an incoherent pulsar search, and of a readout for microwave kinetic inductance detectors (MKIDs) are specified. The thesis then presents efficient algorithms, that satisfy these requirements. After defining the components of an FFTS and their purpose, the technical requirements of each component are described, and algorithms or implementations are discussed with respect to their processing speed, hardware utilization, memory occupation, flexibility, or just simplicity. Concepts are developed to partition algorithms between the FPGA and the personal computer (PC) to create simple, hardware-efficient components inside the FPGA. To achieve both, high bandwidth and high spectral resolution, parallel and pipelined algorithms are combined. The hardware utilization and the flexibility of different such fast Fourier transform (FFT) architectures are compared, dependent on the significance of either bandwidth or resolution. Control mechanisms are developed and implemented to function in different time frames, dependent on the application. Two fully functional high-resolution wideband spectrometers, in which such algorithms are implemented, benefit from the optimization of the processing pipeline: the Array Fast Fourier Transform Spectrometer (AFFTS) and the eXtended-bandwidth Fast Fourier Transform Spectrometer (XFFTS). Finally, an astronomical application of the aforementioned spectrometers is presented: two unbiased line surveys of molecular cloud positions near the center of our Galaxy with the First Light APEX Submillimeter Heterodyne receiver (FLASH) in the Atacama Pathfinder EXperiment (APEX) telescope. Containing hundreds of spectral lines, those surveys provide a large amount of information on the physical and chemical conditions of the observed objects and thus work for several years of analysis. We present the basic results that can be extracted from a first iteration with the data: line identification, selection of the best molecular tracers, and analysis of those tracers to obtain the physical properties in the studied regions. Unidentified lines and so far unaccessed information, and the possibility to add this data to unbiased surveys taken with other telescopes are a legacy to future astronomical research and thus demonstrate the benefits of the presented concepts

Distributed Coordination and Optimisation of Network-Aware Electricity Prosumers

Electricity networks are undergoing a transformation brought on by new technologies, market pressures and environmental concerns. This includes a shift from large centralised generators to small-scale distributed generators. The dramatic cost reductions in rooftop solar PV and battery storage means that prosumers (houses and other entities that can both produce and consume electricity) will have a large role to play in future networks. How can networks be managed going forward so that they run as efficiently as possible in this new prosumer paradigm? Our vision is to treat prosumers as active participants by developing a mechanism that incentivises them to help balance power and support the network. The whole process is automated to produce a near-optimal outcome and to reduce the need for human involvement. The first step is to design an autonomous energy management system (EMS) that can optimise the local costs of each prosumer in response to network electricity prices. In particular, we investigate different optimisation strategies for an EMS in an uncertain household environment. We find that the uncertainty associated with weather, network pricing and occupant behaviour can be effectively handled using online optimisation techniques using a forward receding horizon. The next step is to coordinate the actions of many EMSs spread out across the network, in order to minimise the overall cost of supplying electricity. We propose a distributed algorithm that can efficiently coordinate a network with thousands of prosumers without violating their privacy. We experiment with a range of power flow models of varying degrees of accuracy in order to test their convergence rate, computational burden and solution quality on a suburb-sized microgrid. We find that the higher accuracy model, although non-convex, converges in a timely manner and produces near-optimal solutions. We also develop simple but effective techniques for dealing with residential shiftable loads which require discrete decisions. The final part of the problem we explore is prosumer manipulation of the coordination mechanism. The receding horizon nature of our algorithm is great for managing uncertainty, but it opens up unique opportunities for prosumers to manipulate the actions of others. We formalise this form of receding horizon manipulation and investigate the benefits manipulative agents can obtain. We find that indeed strategic agents can harm the system, but only if they are large enough and have information about the behaviour of other agents. For the rare cases where this is possible, we develop simple privacy-preserving identifiers that monitor agents and distinguish manipulation from uncertainty. Together, these components create a complete solution for the distributed coordination and optimisation of network-aware electricity prosumers