Drayton, G

Edmunds, T

Epperly, T W

Lamont, A

Meyers, C

Smith, S

Yao, Y

UNT Digital Library

LLNL-PROC-528131High-Performance Computing forElectric Grid Planning andOperationsT. G. W. Epperly, T. Edmunds, A. Lamont, C.Meyers, S. Smith, Y. Yao, G. DraytonFebruary 9, 2012IEEE Power & Energy Society 2012 General MeetingSan Diego, CA, United StatesJuly 22, 2012 through July 26, 2012Disclaimer  This document was prepared as an account of work sponsored by an agency of the United States government. Neither the United States government nor Lawrence Livermore National Security, LLC, nor any of their employees makes any warranty, expressed or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States government or Lawrence Livermore National Security, LLC. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States government or Lawrence Livermore National Security, LLC, and shall not be used for advertising or product endorsement purposes.  1High-Performance Computing for Electric GridPlanning and OperationsThomas Epperly, Thomas Edmunds, Alan Lamont, Carol Meyers, Steven Smith, Yiming Yao, Glenn DraytonAbstract—High-performance computing is having a profoundimpact on scientific discovery and engineering in a number of ar-eas, and researchers are beginning to study how HPC can impactenergy grid planning and operations problems. Contemporarysupercomputers can perform over 1015 floating point operationsper second and have more than 300 terabytes of memory —roughly 5 orders of magnitude more than a commodity PCworkstation. This level of computer power changes what ispossible.Resarchers are LLNL have already used HPC systems toaccelerate a multi-scenario planning study by solving a thousandunit commitment models in parallel which enabled a moredetailed study than would have been achievable otherwise inthe time given. LLNL is developing and testing new parallelalgorithms for the unit commitment problems, including thestochastic unit commitment. These algorithms will enable highernetwork and time resolution and provide better solutions becauseof the increase model fidelity. solution of unit commitment modelIndex Terms—Supercomputers, parallel machines, power gen-eration planning, photovoltaic systems, wind power generation.I. INTRODUCTIONSUPERCOMPUTING is having a profound impact inscience discovery and engineering. Supercomputers arebeing used to perform climate modeling, weather prediction,astrophysical simulation, drug discovery, fusion reactor mod-eling, and many other important simulations. For example, re-searchers at Lawrence Livermore National Laboratory (LLNL)have used computational fluid dynamics (CFD) models run-ning on supercomputers to study the aerodynamics of trucks.These studies identified ways to retrofit trucks to improve atruck’s contours to result in a 17 percent efficiency gain. Thesechanges could save the shipping industry 6 billion gallons offuel and $24 billion a year [1]. Similarly, Boeing has used CFDto dramatically reduce the number of wing prototypes theyneed to construct for testing in the development of new planes.Figure 1 summarizes decades of research and development atBoeing, and on the bottom it shows the number of prototypewings tested starting with 77 for the 767 going to 5 for thelatest aircraft [2].T. Epperly, T. Edmunds, A. Lamont, C. Meyer, S. Smith and Y. Yao arefrom Lawrence Livermore National Laboratory, P.O. Box 808, Livermore, CA94551 email: epperly2@llnl.govThis work performed under the auspices of the U.S. Department ofEnergy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344. LLNL-JRNL-xxxxxxxG. Drayton is with Energy Exemplar Pty Ltd, 3/154-160 Prospect Road,Prospect, Adelaide, SA 5082 Australia.Manuscript received February 9, 2012.Fig. 1. Boeing code advances showing the reduction in the number of wingprototypes that required testing. [2]Contemporary leading supercomputers are able to performover a quadrillion floating people operations per second(petaFLOPS) and have over 1.4 million gigabytes of memory[3]. To put this in perspective, a top-of-the-line desktop work-station achieves roughly 66.9 gigaFLOPS on the LINPACKbenchmark. Thus today’s fastest supercomputer has roughly 5orders of magnitude more computing power and memory thana top-of-the-line desktop system1.Orders of magnitude increases in computing power changeswhat’s possible. For example, using LLNL’s BG/L super-computer, which has 212,992 CPUs, LLNL scientists wereable to perform the first micron-scale simulation of a Kelvin-Helmholtz instability using molecular dynamics. This simu-lation required 2.8 CPU-millennia of run time with over 1.5CPU-millennia of continuous runtime. [5] A simulation of thismagnitude is only conceivable on the largest supercomputers.The power of supercomputers is advancing at an exponentialrate. Figure 2 shows performance on the high-performanceLINPACK benchmark for the fastest supercomputers for thepast 19 years. The dashed line shows the trend for the topranked machine and extrapolates the results through 2019.Current U.S. Department of Energy plans call for an exaFLOP(1018 FLOPS) computer to be fielded before 2020. Areas ofscience and engineering that are able to utilize supercomputerswill be able to solve larger problems with higher fidelitymodels.Research in supercomputing is also valuable because su-percomputers and desktop systems are looking more andmore alike. Due to the difficulty of increasing clock speeds,hardware manufacturers are expanding performance by addingmore processing cores to commodity desktop systems. Su-percomputer developers already have decades of experience1Based on a Dell Dimension T5500 with dual quad-core Intel Xeonprocessors running at 2.64GHz Using Intel’s optimized high performanceLINPACK download [4]21993199519971999200120032005200720092011201320152017201910810101012101410161018FLOPSPerformance of Top500 Supercomputing Sites Since 1993Rank 1 Average Rank 500Fig. 2. Performance of top ranked, bottom rank, and average of the top 500 supercomputers based on the LINPACK benchmark. [3]writing algorithms that can take advantage of multiple concur-rent processors. Supercomputer developers are investigatingtechnologies like OpenMP, CUDA, and OpenCL to addressmulti-core and GPU programming, and these tools are alsoavailable on desktop workstation systems [6]–[8].Researchers at LLNL are researching application of super-computers to problems in the energy grid. Section II covers acase study in partnership with Energy Exemplar where LLNLsupercomputers were used to accelerate a study of the 33%renewable portfolio standard for the California Public UtilitiesCommission (CPUC). Section III covers our conclusions anddiscusses future work.II. SUPERCOMPUTERS FOR GRID PLANNINGIn late 2008, the CPUC initiated the CAISO/PLEXOS 33%Renewable Portfolio Standard (RPS) study to investigate theimpact of California’s 33% renewables target. Initially, thestudy was a collaboration between the California IndependentSystems Operator (CAISO), PLEXOS Solutions who built themodel, and Southern California Edison (SCE) who was taskedwith running the models. Mid-project, the CPUC contactedLLNL to investigate using supercomputers to accelerate theexection of the ensemble of simulations required for the study.LLNL partnered with Energy Exemplar, makers of PLEXOSfor Power Systems R©, to accelerate the solution by runningthe large computational simulations on LLNL’s Hyperionsupercomputer and by modifying the problem formulation [9].The CAISO/PLEXOS 33% RPS model is based on datamaintained and provided by CAISO. It incorporates resourcedata, hourly load data for wind, import, hydro, solar, geother-mal, and qualifying facility generation. The model covers 12regions total across the WECC grid (shown in Figure 3), andit includes 42 utilities, 2157 generators, and 104 transmissionlines [9].Fig. 3. Tranmission grid use in the CAISO/PLEXOS 33% RPS study.The basic form of the model is an ensemble of deterministicunit commitment models. PLEXOS reads the model definitionand data, and it generates a large collection of individual unitcommitment models that must be solved independently. Eachunit commitment model instance is formulated as a mixed-integer program (MIP), and PLEXOS invokes a MIP solversuch as FICO’s Xpress-MP or IBM’s CPLEX. MIP solutionrequires approximately 99% of the overall execution time.LLNL focused on running the ensemble of unit commitmentmodels in parallel. To accomplish this, parts of the PLEXOSsystem, written to run on Windows using the .Net framework,were ported to Linux using mono, a Linux/Unix implemen-tation of the .Net framework [10]. The system was extendedto utilitize the a parallel computer to solve a number of unitcommitment models in parallel — allocating one core to thesolution of each unit commitment model. Figure 4 showsLLNL progress in improving the execution speed measuredin the number of scenarios per day that could be run. The firstimprovement (Sept to Oct) came from porting the system to3Fig. 4. Improvements in the number of simultaneous scenarios able to beexecuted, September–November 2010 [9].Fig. 6. LLNL and Energy Exemplar researchers were also able to improvesolution time by improving the formulations used in the PLEXOS model.Linux, and the improvement from Oct. to Nov. was by utilizingthe Hyperion supercomputer at LLNL.The ability to run thousands of scenarios in parallel trans-lates into the abililty to perform large-scale sensitivity analyseson the data and results, a capability the study previouslylacked. LLNL performed a massively parallel suite of runsvarying the random seed used to generate the scenarios thuscreating numerous patterns of generator outages. The resultsof this sensitivity study were enlightening: for some outputparameters of interest, the impact of varying the generatoroutages caused a normally distributed pattern of ensuing re-sults; for others, this distribution was exponential in shape; andfor yet others, the distribution appeared bimodal (see Figure 5a–c). This implies that constraining events such as generatoroutages to a fixed pattern can have a potentially large impacton the results. For instance, if the fixed pattern was chosenfrom one of those on the tail of an exponential distribution,this would produce markedly different results than a patternfrom the mean. Altogether, this suggests that such sensitivityanalyses may be very important in building confidence in theresults obtained overall.In addition to parallelizing the scenario modeling, EnergyExemplar and LLNL staff worked together to improve the MIPformulation underlying the PLEXOS model. Figure 6 showsthe improvement in the solution times for each month of aone year simulation. The overall average is 4x speedup inexecution time.III. CONCLUSIONS AND ONGOING WORKOverall, LLNL’s collaboration with Energy Exemplaryielded over a thousand fold increase in execution speed forthe CAISO/PLEXOS 33% RPS study. It is likely that evengreater increases are possible by utilizing a bigger supercom-puter. By bringing more computational power to bear on theproblem, we were able to provide a more thorough analysis insignificantly less time than would have been required to runthe simulations on standard desktop computers.The collaboration between Energy Exemplar and LLNL isongoing. LLNL is developing a system to enable modelersrunning PLEXOS on their Windows desktop to automati-cally launch simulations in parallel on supercomputers. Thenetwork-based system manages transfering the model detailsfrom the desktop to the supercomputer and transfering indi-vidual simulation results back to PLEXOS on the desktop.LLNL researchers have also investigated the impact on MIPsolver tolerances on solution time and results. Lowering thetolerance can improve solution times with only small impactson the simulation results.LLNL is investigating new parallel algorithms to solve asingle unit commitment model rather than an ensemble of runs.Our initial approach uses Lagrangian relaxation to decouplethe MIP into a collection of independent MIP that can berun in parallel. LLNL is also beginning work on semi-definiteprogramming to solve the optimal power flow problem. Workin both these areas is in the early phase.REFERENCES[1] J. Tozzi, “Kambiz Salari and the aerodynamics of trucks,” BloombergBusinessweek, July 2011.[2] F. T. Johnson, E. N. Tinoco, and N. J. Yu, “Thirty years of developmentand appliation of CFD at Boeing Commercial Airplanes, Seattle,”Computers and Fluids, vol. 34, pp. 1115–1151, 2005.[3] “Top500 supercomputing sites,” http://top500.org/.[4] Intel Math Kernel Library, Intel Corporation, 2011,http://software.intel.com/en-us/articles/intel-math-kernel-library-documentation/.[5] J. N. Glosli, K. J. Caspersen, J. A. Gunnels, D. F. Richards, R. E.Rudd, and F. H. Streitz, “Extending stability beyond CPU millenium,”in Proceeding of Supercomputing 2007 (SC07), Reno, NV, November2007.[6] “The OpenMP API specification for parallel programming,”http://openmp.org/.[7] “CUDA: Parallel programming made easy,”http://www.nvidia.com/object/cuda home new.html.[8] “OpenCL — the open standard for parallel programming of heteroge-nous systems,” http://www.khronos.org/opencl/.[9] C. Meyers, F. Streitz, Y. Yao, S. Smith, and A. Lamont, “Using super-computers to speed execution of the caiso/plexos 33National Laboratory,Tech. Rep. LLNL-TR-500257, 2011.[10] “mono,” http://www.mono-project.com/.4(a) (b) (c)Fig. 5. Probability distribution for different output parameters in the sensitivity analysis: (a) an approximately normal distribution for the load followingdown price, (b) an exponential distribution for the MUNI regulation down price, and (c) a bimodal distribution for the Southern CA Edison’s net export.Thomas Epperly is the Computer Science GroupLeader in the Center for Applied Scientific Com-puting (CASC) at the Lawrence Livermore NationalLaboratory (LLNL). He has worked as co-architectof Babel, the R&D 100 award winning language in-teroperability technology. Dr. Epperly, a Departmentof Energy Computational Science Graduate Fellow,earned his Ph. D. in Chemical Engineering from theUniversity of Wisconsin and his B. S. in ChemicalEngineering from Carnegie-Mellon University. HisPh. D. research focused on parallel global optimiza-tion algorithms for nonconvex nonlinear programs. As a post-doc in theCentre for Process Systems Engineering at Imperial College in London, heworked on global optimization algorithms for reactor design under uncertainty.His undergraduate research project contributed to ASCEND, an object-oriented language for building large mathematical models. After his post-doc, he worked for Aspen Technology, Inc. on an equation-oriented modelingframework for real time optimization of chemical plant operations.Dr. Thomas Edmunds has over thirty years ofexperience in designing and analyzing power sys-tems. He has power plant design experience withWestinghouse and Bechtel, and power system op-erations analysis experience with the Texas PublicUtilities Commission, independent power produc-ers, and Lawrence Livermore National Laboratory,where he is currently Chief Scientist of the Systemsand Decision Sciences Section. He is President ofthe Northern California Chapter of the Institute forOperations Research and the Management Sciences.Dr. Edmunds earned B.S. and M.S. degrees in nuclear engineering at theUniversity of Virginia, and an M.B.A. and Ph.D. in operations research atUniversity of Texas.Alan Lamont graduated from Stanford University in1970 with Bachelors and Masters degrees in CivilEngineering. After a term with the Peace Corpsin Venezuela designing small dams, he worked forWoodward-Clyde Consultants in earthquake analysisfor nuclear power plants and pipeline constructionin Alaska. He returned to Stanford, and completeda PhD in Engineering Economic Systems in 1983.From 1983 to 1987 he was part of the DecisionAnalysis Group at Woodward-Clyde. Since 1987he has been with Lawrence Livermore NationalLaboratory working in analysis of security systems, decision analysis, riskanalysis, and the economics of energy systems.Carol Meyers has worked as a mathematician atLawrence Livermore National Laboratory since re-ceiving her PhD in operations research from MITin 2006. Her expertise is in the areas of integerand linear programming optimization, agent-basedmodeling, discrete event simulation, and decisiontheory. She has been involved in energy modelingsince 2009, leading an effort to port the PLEXOSpower market modeling software to run on high-performance computers, in conjunction with devel-opers at Energy Exemplar (owners of the PLEXOScode) and optimization assistance from IBM. When she is not working inthe energy arena, she spends her time trying to help optimize planning ofoperations in the nuclear arena.Steven Smith received the B.S. degree in Computer Science from theUniversity of Illinois Urbana/Champaign in 1991 with highest honors andwas a recipient of the Bronze Table award. Since then he has been withLawrence Livermore National Laboratory where he has worked on a verity ofhigh performance computing applications and frameworks. He has served assoftware project lead for several projects, including Parflow, the StructuredAdaptive Mesh Refinement Application Infrastructure (SAMRAI) and theMulti-Scale Epidemiologic Simulation and Analysis Decision Support System(MESA).Yiming Yao (BSEE and PhD, Operations Research) is experienced inmodeling, simulation and optimization of complex systems including com-munication networks and power grids. Recently he has been working onpower grid planning and operations. He’s interested in developing paralleloptimization algorithms for solving unit commit commitment and economicdispatch problems that explicitly model stochastic demands and supplies. Hehas developed optimization software on various platforms, from desktops tosupercomputers.Glenn Drayton has experience in power system modelling and economics thatspans 15 years. He was involved in the design of the New Zealand, Australianand Singapore power markets, in particular the energy-ancillary services co-optimisation. Glenn started Drayton Analytics (now called Energy Exemplar)in 1999 to produce a next-generation power market model based entirelyon optimisation, the result being PLEXOS for Power Systems R©. Glennobtained his B.Sc.(Hons) in Operations Research, and Ph.D. in ManagementScience from the University of Canterbury, New Zealand. Glenn managesthe development team for PLEXOS in the Adelaide, Australia office. Hespecialises in mathematical optimisation applied to power system planning.

High-Performance Computing for Electric Grid Planning and Operations

T. Epperly

T. Edmunds

A. Lamont

C. Meyers

S. Smith

null Yiming Yao

G. Drayton

Crossref

High-performance computing for electric grid planning and operations

https://digital.library.unt.edu/ark:/67531/metadc829177/m2/1/high_res_d/1091372.pdf

High-performance computing for electric grid planning and operations

Abstract

Similar works

Full text

Available Versions

UNT Digital Library

Crossref