Oppositional Reinforcement Learning with Applications

Machine intelligence techniques contribute to solving real-world problems. Reinforcement learning (RL) is one of the machine intelligence techniques with several characteristics that make it suitable for the applications, for which the model of the environment is not available to the agent. In real-world applications, intelligent agents generally face a very large state space which limits the usability of reinforcement learning. The condition for convergence of reinforcement learning implies that each state-action pair must be visited infinite times, a condition which can be considered impossible to be satisfied in many practical situations. The goal of this work is to propose a class of new techniques to overcome this problem for off-policy, step-by-step (incremental) and model-free reinforcement learning with discrete state and action space. The focus of this research is using the design characteristics of RL agent to improve its performance regarding the running time while maintaining an acceptable level of accuracy. One way of improving the performance of the intelligent agents is using the model of environment. In this work, a special type of knowledge about the agent actions is employed to improve its performance because in many applications the model of environment may only be known partially or not at all. The concept of opposition is employed in the framework of reinforcement learning to achieve this goal. One of the components of RL agent is the action. For each action we define its associate opposite action. The actions and opposite actions are implemented in the framework of reinforcement learning to update the value function resulting in a faster convergence. At the beginning of this research the concept of opposition is incorporated in the components of reinforcement learning, states, actions, and reinforcement signal which results in introduction of the oppositional target domain estimation algorithm, OTE. OTE reduces the search and navigation area and accelerates the speed of search for a target. The OTE algorithm is limited to the applications, in which the model of the environment is provided for the agent. Hence, further investigation is conducted to extend the concept of opposition to the model-free reinforcement learning algorithms. This extension contributes to the generating of several algorithms based on using the concept of opposition for Q(lambda) technique. The design of reinforcement learning agent depends on the application. The emphasize of this research is on the characteristics of the actions. Hence, the primary challenge of this work is design and incorporation of the opposite actions in the framework of RL agents. In this research, three different applications, namely grid navigation, elevator control problem, and image thresholding are implemented to address this challenge in context of different applications. The design challenges and some solutions to overcome the problems and improve the algorithms are also investigated. The opposition-based Q(lambda) algorithms are tested for the applications mentioned earlier. The general idea behind the opposition-based Q(lambda) algorithms is that in Q-value updating, the agent updates the value of an action in a given state. Hence, if the agent knows the value of opposite action then instead of one value, the agent can update two Q-values at the same time without taking its corresponding opposite action causing an explicit transition to opposite state. If the agent knows both values of action and its opposite action for a given state, then it can update two Q-values. This accelerates the learning process in general and the exploration phase in particular. Several algorithms are outlined in this work. The OQ(lambda) will be introduced to accelerate Q(lambda) algorithm in discrete state spaces. The NOQ(lambda) method is an extension of OQ(lambda) to operate in a broader range of non-deterministic environments. The update of the opposition trace in OQ(lambda) depends on the next state of the opposite action (which generally is not taken by the agent). This limits the usability of this technique to the deterministic environments because the next state should be known to the agent. NOQ(lambda) will be presented to update the opposition trace independent of knowing the next state for the opposite action. The results show the improvement of the performance in terms of running time for the proposed algorithms comparing to the standard Q(lambda) technique

Linearized biogeography-based optimization with re-initialization and local search

Biogeography-based optimization (BBO) is an evolutionary optimization algorithm that uses migration to share information among candidate solutions. One limitation of BBO is that it changes only one independent variable at a time in each candidate solution. In this paper, a linearized version of BBO, called LBBO, is proposed to reduce rotational variance. The proposed method is combined with periodic re-initialization and local search operators to obtain an algorithm for global optimization in a continuous search space. Experiments have been conducted on 45 benchmarks from the 2005 and 2011 Congress on Evolutionary Computation, and LBBO performance is compared with the results published in those conferences. The results show that LBBO provides competitive performance with state-of-the-art evolutionary algorithms. In particular, LBBO performs particularly well for certain types of multimodal problems, including high-dimensional real-world problems. Also, LBBO is insensitive to whether or not the solution lies on the search domain boundary, in a wide or narrow basin, and within or outside the initialization domain

Microgrids/Nanogrids Implementation, Planning, and Operation

Today’s power system is facing the challenges of increasing global demand for electricity, high-reliability requirements, the need for clean energy and environmental protection, and planning restrictions. To move towards a green and smart electric power system, centralized generation facilities are being transformed into smaller and more distributed ones. As a result, the microgrid concept is emerging, where a microgrid can operate as a single controllable system and can be viewed as a group of distributed energy loads and resources, which can include many renewable energy sources and energy storage systems. The energy management of a large number of distributed energy resources is required for the reliable operation of the microgrid. Microgrids and nanogrids can allow for better integration of distributed energy storage capacity and renewable energy sources into the power grid, therefore increasing its efficiency and resilience to natural and technical disruptive events. Microgrid networking with optimal energy management will lead to a sort of smart grid with numerous benefits such as reduced cost and enhanced reliability and resiliency. They include small-scale renewable energy harvesters and fixed energy storage units typically installed in commercial and residential buildings. In this challenging context, the objective of this book is to address and disseminate state-of-the-art research and development results on the implementation, planning, and operation of microgrids/nanogrids, where energy management is one of the core issues

Load Frequency Control (LFC) Strategies in Renewable Energy‐Based Hybrid Power Systems:A Review

The hybrid power system is a combination of renewable energy power plants and conventional energy power plants. This integration causes power quality issues including poor settling times and higher transient contents. The main issue of such interconnection is the frequency variations caused in the hybrid power system. Load Frequency Controller (LFC) design ensures the reliable and efficient operation of the power system. The main function of LFC is to maintain the system frequency within safe limits, hence keeping power at a specific range. An LFC should be supported with modern and intelligent control structures for providing the adequate power to the system. This paper presents a comprehensive review of several LFC structures in a diverse configuration of a power system. First of all, an overview of a renewable energy-based power system is provided with a need for the development of LFC. The basic operation was studied in single-area, multi-area and multi-stage power system configurations. Types of controllers developed on different techniques studied with an overview of different control techniques were utilized. The comparative analysis of various controllers and strategies was performed graphically. The future scope of work provided lists the potential areas for conducting further research. Finally, the paper concludes by emphasizing the need for better LFC design in complex power system environments

Advanced Signal Processing Techniques Applied to Power Systems Control and Analysis

The work published in this book is related to the application of advanced signal processing in smart grids, including power quality, data management, stability and economic management in presence of renewable energy sources, energy storage systems, and electric vehicles. The distinct architecture of smart grids has prompted investigations into the use of advanced algorithms combined with signal processing methods to provide optimal results. The presented applications are focused on data management with cloud computing, power quality assessment, photovoltaic power plant control, and electrical vehicle charge stations, all supported by modern AI-based optimization methods

Optimal integration and management of solar generation and battery storage system in distribution systems under uncertain environment

The simultaneous placement of solar photovoltaics (SPVs) and battery energy storage systems (BESSs) in distribution systems is a highly complex combinatorial optimization problem. It not only involves siting and sizing but is also embedded with charging and discharging dispatches of BESSs under dynamically varying system states with intermittency of SPVs and operational constraints. This makes the simultaneous allocation a nested problem, where the operational part acts as a constraint for the planning part and adds complexity to the problem. This paper presents a bi-layer optimization strategy to optimally place SPVs and BESSs in the distribution system. A simple and effective operating BESS strategy model is developed to mitigate reverse power flow, enhance load deviation index and absorb variability of load and power generation which are essential features for the faithful exploitation of available renewable energy sources (RESs). In the proposed optimization strategy, the inner layer optimizes the energy management of BESSs for the sizing and siting as suggested by the outer layer. Since the inner layer optimizes each system state separately, the problem search space of GA is significantly reduced. The application results on a benchmark 33-bus test distribution system highlight the importance of the proposed method

Optimization of Aggregators Energy Resources considering Local Markets and Electric Vehicle Penetration

O sector elétrico tem vindo a evoluir ao longo do tempo. Esta situação deve-se ao facto de surgirem novas metodologias para lidarem com a elevada penetração dos recursos energéticos distribuídos (RED), principalmente veículos elétricos (VEs). Neste caso, a gestão dos recursos energéticos tornou-se mais proeminente devido aos avanços tecnológicos que estão a ocorrer, principalmente no contexto das redes inteligentes. Este facto torna-se importante, devido à incerteza decorrente deste tipo de recursos. Para resolver problemas que envolvem variabilidade, os métodos baseados na inteligência computacional estão a se tornar os mais adequados devido à sua fácil implementação e baixo esforço computacional, mais precisamente para o caso tratado na tese, algoritmos de computação evolucionária (CE). Este tipo de algoritmo tenta imitar o comportamento observado na natureza. Ao contrário dos métodos determinísticos, a CEé tolerante à incerteza; ou seja, é adequado para resolver problemas relacionados com os sistemas energéticos. Estes sistemas são geralmente de grandes dimensões, com um número crescente de variáveis e restrições. Aqui a IC permite obter uma solução quase ótima em tempo computacional aceitável com baixos requisitos de memória. O principal objetivo deste trabalho foi propor um modelo para a programação dos recursos energéticos dos recursos dedicados para o contexto intradiário, para a hora seguinte, partindo inicialmente da programação feita para o dia seguinte, ou seja, 24 horas para o dia seguinte. Esta programação é feita por cada agregador (no total cinco) através de meta-heurísticas, com o objetivo de minimizar os custos ou maximizar os lucros. Estes agregadores estão inseridos numa cidade inteligente com uma rede de distribuição de 13 barramentos com elevada penetração de RED, principalmente energia renovável e VEs (2000 VEs são considerados nas simulações). Para modelar a incerteza associada ao RED e aos preços de mercado, vários cenários são gerados através da simulação de Monte Carlo usando as funções de distribuição de probabilidade de erros de previsão, neste caso a função de distribuição normal para o dia seguinte. No que toca à incerteza no modelo para a hora seguinte, múltiplos cenários são gerados a partir do cenário com maior probabilidade do dia seguinte. Neste trabalho, os mercados locais de eletricidade são também utilizados como estratégia para satisfazer a equação do balanço energético onde os agregadores vão para vender o excesso de energia ou comprar para satisfazer o consumo. Múltiplas metaheurísticas de última geração são usadas para fazer este escalonamento, nomeadamente Differential Evolution (DE), Hybrid-Adaptive DE with Decay function (HyDE-DF), DE with Estimation of Distribution Algorithm (DEEDA), Cellular Univariate Marginal Distribution Algorithm with Normal-Cauchy Distribution (CUMDANCauchy++), Hill Climbing to Ring Cellular Encode-Decode UMDA (HC2RCEDUMDA). Os resultados mostram que o modelo proposto é eficaz para os múltiplos agregadores com variações de custo na sua maioria abaixo dos 5% em relação ao dia seguinte, exceto para o agregador e de VEs. É também aplicado um teste Wilcoxon para comparar o desempenho do algoritmo CUMDANCauchy++ com as restantes meta-heurísticas. O CUMDANCauchy++ mostra resultados competitivos tendo melhor performance que todos os algoritmos para todos os agregadores exceto o DEEDA que apresenta resultados semelhantes. Uma estratégia de aversão ao risco é implementada para um agregador no contexto do dia seguinte para se obter uma solução mais segura e robusta. Os resultados mostram um aumento de quase 4% no investimento, mas uma redução de até 14% para o custo dos piores cenários.The electrical sector has been evolving. This situation is because new methodologies emerge to deal with the high penetration of distributed energy resources (DER), mainly electric vehicles (EVs). In this case, energy resource management has become increasingly prominent due to the technological advances that are taking place, mainly in the context of smart grids. This factor becomes essential due to the uncertainty of this type of resource. To solve problems involving variability, methods based on computational intelligence (CI) are becoming the most suitable because of their easy implementation and low computational effort, more precisely for the case treated in this thesis, evolutionary computation (EC) algorithms. This type of algorithm tries to mimic behavior observed in nature. Unlike deterministic methods, the EC is tolerant of uncertainty, and thus it is suitable for solving problems related to energy systems. These systems are usually of high dimensions, with an increased number of variables and restrictions. Here the CI allows obtaining a near-optimal solution in good computational time with low memory requirements. This work's main objective is to propose a model for the energy resource scheduling of the dedicated resources for the intraday context, for the our-ahead, starting initially from the scheduling done for the day ahead, that is, 24 hours for the next day. This scheduling is done by each aggregator (in total five) through metaheuristics to minimize the costs or maximize the profits. These aggregators are inserted in a smart city with a distribution network of 13 buses with a high penetration of DER, mainly renewable energy and EVs (2000 EVs are considered in the simulations). Several scenarios are generated through Monte Carlo Simulation using the forecast errors' probability distribution functions, the normal distribution function for the day-ahead to model the uncertainty associated with DER and market prices. Multiple scenarios are developed through the highest probability scenario from the day-ahead when it comes to intraday uncertainty. In this work, local electricity markets are used as a mechanism to satisfy the energy balance equation where each aggregator can sell the excess of energy or buy more to meet the demand. Several recent and modern metaheuristics are used to solve the proposed problems in the thesis, namely Differential Evolution (DE), Hybrid-Adaptive DE with Decay function (HyDE-DF), DE with Estimation of Distribution Algorithm (DEEDA), Cellular Univariate Marginal Distribution Algorithm with NormalCauchy Distribution (CUMDANCauchy++), Hill Climbing to Ring Cellular Encode-Decode UMDA (HC2RCEDUMDA). Results show that the proposed model is effective for the multiple aggregators. The metaheuristics present satisfactory results and mostly less than 5% variation in costs from the day-ahead except for the EV aggregator. A Wilcoxon test is also applied to compare the performance of the CUMDANCauchy++ algorithm with the remaining metaheuristics. CUMDANCauchy++ shows competitive results beating all algorithms in all aggregators except for DEEDA, which presents similar results. A risk aversion strategy is implemented for an aggregator in the day-ahead context to get a safer and more robust solution. Results show an increase of nearly 4% in day-ahead cost but a reduction of up to 14% of worst scenario cost

Data-Intensive Computing in Smart Microgrids

Microgrids have recently emerged as the building block of a smart grid, combining distributed renewable energy sources, energy storage devices, and load management in order to improve power system reliability, enhance sustainable development, and reduce carbon emissions. At the same time, rapid advancements in sensor and metering technologies, wireless and network communication, as well as cloud and fog computing are leading to the collection and accumulation of large amounts of data (e.g., device status data, energy generation data, consumption data). The application of big data analysis techniques (e.g., forecasting, classification, clustering) on such data can optimize the power generation and operation in real time by accurately predicting electricity demands, discovering electricity consumption patterns, and developing dynamic pricing mechanisms. An efficient and intelligent analysis of the data will enable smart microgrids to detect and recover from failures quickly, respond to electricity demand swiftly, supply more reliable and economical energy, and enable customers to have more control over their energy use. Overall, data-intensive analytics can provide effective and efficient decision support for all of the producers, operators, customers, and regulators in smart microgrids, in order to achieve holistic smart energy management, including energy generation, transmission, distribution, and demand-side management. This book contains an assortment of relevant novel research contributions that provide real-world applications of data-intensive analytics in smart grids and contribute to the dissemination of new ideas in this area