Reinforcement learning for enhancing the stability and management of power systems with new resources

Modern power systems face numerous challenges due to uncertainties arising from factors such as renewable energy source intermittency, stochastic load demand, and evolving grid dynamics. These uncertainties can lead to imbalances in power supply and demand, resulting in frequency and voltage deviations and, in extreme cases, blackouts. To address these challenges, advanced control and optimization techniques, particularly reinforcement learning (RL), have gained significant interest in ensuring efficient and reliable power system operations. RL offers a promising approach for decision-making under uncertainty, enabling agents to learn optimal policies without explicit uncertainty modeling. This thesis explores the application of RL to two classes of operational problems within power systems. The first class focuses on power system resource management, including optimal battery control (OBC) and electric vehicle charging station (EVCS) operation. Challenges arise when formulating these problems as Markov Decision Process (MDP) to adopt RL. For example, incorporating cycle-based degradation costs into the MDP for OBC is not straightforward due to its dependence on past state of charge (SoC) trajectories. Similarly, the state and action spaces in EVCS problem scale with the number of EVs, leading to high-dimensional MDP formulations. This thesis proposes RL-based solutions for these resource management problems, while addressing the challenges by incorporating precise battery degradation model and efficient aggregation schemes to MDP. The second class of problems deals with wide-area dynamics control for power system stability enhancement. Here, it is crucial for RL approaches to account for risk measures in offline-trained RL policies, considering uncertainties and perturbations in practice. The thesis focuses on load frequency control (LFC), which is vulnerable to variability due to high load perturbations, especially in small-scale systems like networked microgrids. Additionally, wide-area damping control (WADC) relies on communication networks, and communication delays can negatively impact its performance, given its fast time-scale. Moreover, the increasing integration of grid-forming inverters (GFMs) poses challenges in accurately modeling the overall system dynamics, which results in high variability in the system. To address these uncertainties and perturbations, this thesis integrates a mean-variance risk constraint into classic linear quadratic regulator (LQR) problems with linearized dynamics, limiting deviations of state costs from their expected values and reducing system variability in worst-case scenarios. In addition, structured feedback controllers need to be considered to match specific information-exchange graphs, which complicates the geometry of feasible region. To design risk-aware controllers for constrained LQR problems, a stochastic gradient-descent with max-oracle (SGDmax) algorithm is developed. This algorithm ensures convergence to a stationary point with a high probability, making it computationally efficient as it solves the inner loop problem of a dual problem easily and utilizes zero-order policy gradients (ZOPG) to estimate unbiased gradients, eliminating the need to compute first-order values. The policy gradient nature of SGDmax also allows the incorporation of structure by considering only non-zero entries in the ZOPG. In summary, this thesis presents RL applications for effectively managing emerging energy resources and enhancing the stability of interconnected power systems. The analytical and numerical results offer efficient and reliable solutions to address uncertainty, supporting the transition towards a sustainable and resilient electricity infrastructure.Electrical and Computer Engineerin

Performance Assessment of TSO–DSO using Volt-Var Control at Smart-Inverters

The massive penetration of distributed energy resources (DERs) in distribution networks provides a strategic opportunity for the distribution system operator (DSO) to coordinate the assets appropriately and offer services to the transmission systems. The IEEE std. 1547-2018 introduced a control mechanism to enable the power electronic converters (PECs) to offer several services, including voltage regulation by controlling the reactive power injection/absorption; this type of PECs is also known as "smart inverter". The participation of the smart-inverters in the voltage regulation with a novel customer-centred piece of legislation and markets provide the DSO with powerful tools to enforce very positive TSO/DSO interactions. This research paper presents a comprehensive assessment of the steady-state performance provided by voltage control at the smart-inverters to the TSO – DSO system. The assessment includes analysing main indicators using time series considering short term (24-hours, 1-minute resolution) and long-term (one-year) horizon. In this paper, the three leading indicators are used as criteria for the assessment: total energy losses voltage profile in the TSO-DSO system and the power flow interaction at the interface between the systems. The assessment is based on numerical results using the DIgSILENT PowerFactory simulation tool, where the voltage controllers have been implemented, and regional electrical system in south-eastern Norway, the area of Vestfold and Telemark as been used for illustrative purpose

Flexitranstore

This open access book comprises 10 high-level papers on research and innovation within the Flexitranstore Project that were presented at the FLEXITRANSTORE special session organized as part of the 21st International Symposium on High Voltage Engineering. FLEXITRANSTORE (An Integrated Platform for Increased FLEXIbility in smart TRANSmission grids with STORage Entities and large penetration of Renewable Energy Sources) aims to contribute to the development of a pan-European transmission network with high flexibility and high interconnection levels. This will facilitate the transformation of the current energy production mix by hosting an increasing share of renewable energy sources. Novel smart grid technologies, control and storage methods, and new market approaches will be developed, installed, demonstrated, and tested introducing flexibility to the European power system. FLEXITRANSTORE is developing a next-generation Flexible Energy Grid (FEG) that will be integrated into the European Internal Energy Market (IEM) through the valorization of flexibility services. This FEG addresses the capabilities of a power system to maintain continuous service in the face of rapid and large swings in supply or demand. As such, a wholesale market infrastructure and new business models within this integrated FEG must be upgraded for network players, and offer incentives for new ones to join, while at the same time demonstrating new business perspectives for cross-border resource management and energy trading

The 1982 NASA/ASEE Summer Faculty Fellowship Program

A NASA/ASEE Summer Faculty Fellowship Research Program was conducted to further the professional knowledge of qualified engineering and science faculty members, to stimulate an exchange of ideas between participants and NASA, to enrich and refresh the research and teaching activities of participants' institutions, and to contribute to the research objectives of the NASA Centers

Impact analysis and optimized control in renewable energy Integrated power network

This thesis quantifies the power quality impacts in hybrid renewable energy integrated power network and explores the voltage regulation method under various network conditions. This thesis also provides an optimized controller for DFIG to significantly ride through the symmetric and asymmetric faults meeting Australian grid code requirements. Thesis has extensive implications in terms of voltage improvement and LVRT enhancement in a grid tied renewable energy integrated power network

Proceedings of the NASA Conference on Space Telerobotics, volume 1

The theme of the Conference was man-machine collaboration in space. Topics addressed include: redundant manipulators; man-machine systems; telerobot architecture; remote sensing and planning; navigation; neural networks; fundamental AI research; and reasoning under uncertainty

Muscle activation patterns in shoulder impingement patients

Introduction: Shoulder impingement is one of the most common presentations of shoulder joint problems 1. It appears to be caused by a reduction in the sub-acromial space as the humerus abducts between 60o -120o – the 'painful arc'. Structures between the humeral head and the acromion are thus pinched causing pain and further pathology 2. Shoulder muscle activity can influence this joint space but it is unclear whether this is a cause or effect in impingement patients. This study aimed to observe muscle activation patterns in normal and impingement shoulder patients and determine if there were any significant differences. Method: 19 adult subjects were asked to perform shoulder abduction in their symptomatic arm and non-symptomatic. 10 of these subjects (age 47.9 ± 11.2) were screened for shoulder impingement, and 9 subjects (age 38.9 ± 14.3) had no history of shoulder pathology. Surface EMG was used to collect data for 6 shoulder muscles (Upper, middle and lower trapezius, serratus anterior, infraspinatus, middle deltoids) which was then filtered and fully rectified. Subjects performed 3 smooth unilateral abduction movements at a cadence of 16 beats of a metronome set at 60bpm, and the mean of their results was recorded. T-tests were used to indicate any statistical significance in the data sets. Significance was set at P<0.05. Results: There was a significant difference in muscle activation with serratus anterior in particular showing a very low level of activation throughout the range when compared to normal shoulder activation patterns (<30%). Middle deltoid recruitment was significantly reduced between 60-90o in the impingement group (30:58%).Trends were noted in other muscles with upper trapezius and infraspinatus activating more rapidly and erratically (63:25%; 60:27% respectively), and lower trapezius with less recruitment (13:30%) in the patient group, although these did not quite reach significance. Conclusion: There appears to be some interesting alterations in muscle recruitment patterns in impingement shoulder patients when compared against their own unaffected shoulders and the control group. In particular changes in scapula control (serratus anterior and trapezius) and lateral rotation (infraspinatus), which have direct influence on the sub-acromial space, should be noted. It is still not clear whether these alterations are causative or reactionary, but this finding gives a clear indication to the importance of addressing muscle reeducation as part of a rehabilitation programme in shoulder impingement patients