Progress and summary of reinforcement learning on energy management of MPS-EV

The high emission and low energy efficiency caused by internal combustion engines (ICE) have become unacceptable under environmental regulations and the energy crisis. As a promising alternative solution, multi-power source electric vehicles (MPS-EVs) introduce different clean energy systems to improve powertrain efficiency. The energy management strategy (EMS) is a critical technology for MPS-EVs to maximize efficiency, fuel economy, and range. Reinforcement learning (RL) has become an effective methodology for the development of EMS. RL has received continuous attention and research, but there is still a lack of systematic analysis of the design elements of RL-based EMS. To this end, this paper presents an in-depth analysis of the current research on RL-based EMS (RL-EMS) and summarizes the design elements of RL-based EMS. This paper first summarizes the previous applications of RL in EMS from five aspects: algorithm, perception scheme, decision scheme, reward function, and innovative training method. The contribution of advanced algorithms to the training effect is shown, the perception and control schemes in the literature are analyzed in detail, different reward function settings are classified, and innovative training methods with their roles are elaborated. Finally, by comparing the development routes of RL and RL-EMS, this paper identifies the gap between advanced RL solutions and existing RL-EMS. Finally, this paper suggests potential development directions for implementing advanced artificial intelligence (AI) solutions in EMS

Modelling and control of hybrid electric vehicles (a comprehensive review)

The gradual decline in global oil reserves and presence of ever so stringent emissions rules around the world, have created an urgent need for the production of automobiles with improved fuel economy. HEVs (hybrid electric vehicles) have proved a viable option to guarantying improved fuel economy and reduced emissions.The fuel consumption benefits which can be realised when utilising HEV architecture are dependent on how much braking energy is regenerated, and how well the regenerated energy is utilized. The challenge in developing an HEV control strategy lies in the satisfaction of often conflicting control constraints involving fuel consumption, emissions and driveability without over-depleting the battery state of charge at the end of the defined driving cycle.To this effect, a number of power management strategies have been proposed in literature. This paper presents a comprehensive review of these literatures, focusing primarily on contributions in the aspect of parallel hybrid electric vehicle modelling and control. As part of this treatise, exploitable research gaps are also identified. This paper prides itself as a comprehensive reference for researchers in the field of hybrid electric vehicle development, control and optimization

Improving Computational Efficiency for Energy Management Systems in Plug-in Hybrid Electric Vehicles Using Dynamic Programming Based Controllers

Reducing computational time has become a critical issue in recent years, particularly in the transportation field, where the complexity of scenarios demands lightweight controllers to run large simulations and gather results to study different behaviors. This study proposes two novel formulations of the Optimal Control Problem (OCP) for the Energy Management System of a Plug-in Hybrid Electric Vehicle (PHEV) and compares their performance with a benchmark found in the literature. Dynamic Programming was chosen as the optimization algorithm to solve the OCP in a Matlab environment, using the DynaProg toolbox. The objective is to address the optimality of the fuel economy solution and computational time. In order to improve the computational efficiency of the algorithm, an existing formulation from the literature was modified, which originally utilized three control inputs. The approach involves leveraging the unique equations that describe the Input-Split Hybrid powertrain, resulting in a reduction of control inputs firstly to two and finally to one in the proposed solutions. The aforementioned formulations are referred to as 2-Controls and a 1-Control. Virtual tests were conducted to evaluate the performance of the two formulations. The simulations were carried out in various scenarios, including urban and highway driving, to ensure the versatility of the controllers. The results demonstrate that both proposed formulations achieve a reduction in computational time compared to the benchmark. The 2-Controls formulation achieved a reduction in computational time of approximately 40 times, while the 1-Control formulation achieved a remarkable reduction of approximately 850 times. These reductions in computational time were achieved while obtaining a maximum difference in fuel economy of approximately 1.5% for the 1-Control formulation with respect to the benchmark solution. Overall, this study provides valuable insights into the development of efficient and optimal controllers for PHEVs, which can be applied to various transportation scenarios. The proposed formulations reduce computational time without sacrificing the optimality of the fuel economy solution, making them a promising approach for future research in this area

Development of Real-time Optimal Control Strategy of Hybrid Transit Bus Based on Predicted Driving Pattern

The control strategy of a hybrid electric vehicle (HEV) has been an active research area in the past decades. The main goal of the optimal control strategy is to maximize the fuel economy and minimize exhaust emissions while also satisfying the expected vehicle performance. Dynamic programming (DP) is an algorithm capable of finding the global optimal solution of HEV operation. However, DP cannot be used as a real-time control approach as it requires pre-known driving information. The equivalent consumption minimization strategy (ECMS) is a real-time control approach, but it always results in local optima due to the non-convex cost function. In my research, a ECMS with DP combined model (ECMSwDP) was proposed, which was a compromise between optimality and real-time capability. In this approach, the optimal equivalent factor (lambda) for a real-time ECMS controller can be derived using ECMSwDP for a given driving condition. The optimal lambda obtained using ECMSwDP was further processed to derive the lambda map, which was a function of vehicle operation and driving information. Six lambda maps were generated corresponding to the developed representative driving patterns. At each distance segment of a drive cycle, the suitable lambda value is available from one of the six lambda maps based on the identified driving pattern and current vehicle operation.;An adaptive ECMS (A-ECMS) model with a driving pattern identification model is developed to achieve the real-time optimal control for a HEV. A-ECMS was capable of controlling the ratio of power provided by the ICE and battery of a hybrid vehicle by selecting the lambda based on the identified lambda map. The effect on fuel consumption of the control strategies developed using the rule-based controller, ECMSwDP, A-ECMS, and DP was simulated using the parallel hybrid bus model developed in this research. The control strategies developed using A-ECMS are able to significantly improve the fuel economy while maintaining the battery charge sustainability. The corrected fuel economy observed with A-ECMS with a penalty function and the average lambda of RDPs was 5.55%, 13.67%, and 19.19% gap to that observed with DP when operated over the Beijing cycle, WVU-CSI cycle, and the actual transit bus route, respectively. The corrected fuel economy observed with A-ECMS with lambda maps of the RDPs was 4.83%, 10.61%, and 14.33% gap to that observed with DP when operated on the Beijing cycle, WVU-CSI cycle, and actual transit bus route, respectively. The simulation results demonstrated that the proposed A-ECMS approaches have the capability to achieve real time suboptimal control of a HEV while maintaining the charge sustainability of the battery

Implementation Of Fuzzy Logic Control Into An Equivalent Minimization Strategy For Adaptive Energy Management Of A Parallel Hybrid Electric Vehicle

As government agencies continue to tighten emissions regulations due to the continued increase in greenhouse gas production, automotive industries are seeking to produce increasingly efficient vehicle technology. Electric vehicles have been introduced by the industry, showing promising signs of reducing emissions production in the automotive sector. However, many consumers may be hesitant to purchase fully electric vehicles due to several uncertainty variables including available charging stations. Hybrid electric vehicles (HEVs) have been introduced to reduce problems while improving fuel economy. HEVs have led to the demand of creating more advanced controls software to consider multiple components for propulsive power in a vehicle. A large section in the software development process is the implementation of an optimal energy management strategy meant to improve the overall fuel efficiency of the vehicle. Optimal strategies can be implemented when driving conditions are known a prior. The Equivalent Consumption Minimization Strategy (ECMS) is an optimal control strategy that uses an equivalence factor to equate electrical to mechanical power when performing torque split determination between the internal combustion engine and electric motor for propulsive and regenerative torque. This equivalence factor is determined from offline vehicle simulations using a sensitivity analysis to provide optimal fuel economy results while maintaining predetermined high voltage battery state of charge (SOC) constraints. When the control hierarchy is modified or different driving styles are applied, the analysis must be redone to update the equivalence factor. The goal of this work is to implement a fuzzy logic controller that dynamically updates the equivalence factor to improve fuel economy, maintain a strict charge sustaining window of operation for the high voltage battery, and reduce computational time required during algorithm development. The adaptive algorithm is validated against global optimum fuel economy and charge sustaining results from a sensitivity analysis performed for multiple drive cycles. Results show a maximum fuel economy improvement of 9.82% when using a mild driving style and a 95% success rate when maintaining an ending SOC within 5% regardless of starting SOC. Recommendations for modification of the fuzzy logic controller are made to produce additional fuel economy and charge sustaining benefits from the parallel hybrid vehicle model

Energy management of hybrid and battery electric vehicles

This work focuses on improving the fuel economy of parallel Hybrid Electric Vehicles (HEVs) and dual-motor Electric Vehicles (EVs) through energy management strategies. Both vehicle models have two propulsion branches, each powering a separate axle: An engine and an electric motor in the HEV and two electric motors in the EV. This similarity in the vehicle models emphasises the need for similar energy management solutions. In Part Energy Management of HEVs of this thesis, a high-fidelity parallel Through-The-Road (TTR) HEV model is developed to study and test conventional control strategies. The traditional control strategies serve as a guide for developing novel heuristic control strategies. The Equivalent Consumption Minimisation Strategy (ECMS) is an optimisation-based control strategy used as the benchmark in this part of the work. A family of rule-based energy management strategies is proposed for parallel HEVs, including the Torque-levelling Threshold-changing Strategy (TTS) and its simplified version, the Simplified Torque-levelling Threshold-changing Strategy (STTS). The TTS applies a concept of torque-levelling, which ensures the engine works efficiently by operating with a constant torque as the load demand crosses a certain threshold, unlike the load-following approach commonly used. However, the TTS requires finely tuned constant torque and threshold parameters, making it unsuitable for real-time applications. To address this, two feedback-like updating laws are incorporated into the TTS to determine the constant torque and threshold online for real-time applications. Real-time versions of these strategies, Real-time Torque-levelling Threshold-changing Strategy (RTTS) and Real-time Simplified Torque-levelling Threshold-changing Strategy (RSTTS) are developed using a novel Driving Pattern Recognition (DPR) algorithm. The effectiveness of the RTTS is demonstrated by implementing it on a high-fidelity parallel hybrid passenger car and benchmarking it against ECMS. In Part Energy Management of EVs of the thesis, a low-fidelity model of a novel EV powertrain with two electric propulsion systems, one at each axle, has been developed to study and test its energy management with one of the main conventional optimal control methods, Dynamic Programming (DP). The EV model uses two differently sized traction motors at the front and rear axles. The thermal dynamics of the utilised Permanent Magnet Synchronous Motors (PMSMs) are studied. DP is first implemented onto the Baseline model that does not include any PMSM thermal dynamics, referred to as the Baseline DP, which acts as a benchmark since it is the conventional case. The thermal dynamics of the traction motors are then introduced in the second DP problem formulation, referred to as the Thermal DP, which is compared against the Baseline DP to evaluate the possible benefits of energy efficiency by the more informed energy management optimisation formulation. The best method is chosen to include these thermal dynamics in the overall energy management control strategy without significantly compromising computational time.Open Acces

Analyzing the Improvements of Energy Management Systems for Hybrid Electric Vehicles Using a Systematic Literature Review: How Far Are These Controls from Rule-Based Controls Used in Commercial Vehicles?

Featured Application This work is useful for researchers interested in the study of energy management systems for hybrid electric vehicles. In addition, it is interesting for institutions related to the market of this type of vehicle. The hybridization of vehicles is a viable step toward overcoming the challenge of the reduction of emissions related to road transport all over the world. To take advantage of the emission reduction potential of hybrid electric vehicles (HEVs), the appropriate design of their energy management systems (EMSs) to control the power flow between the engine and the battery is essential. This work presents a systematic literature review (SLR) of the more recent works that developed EMSs for HEVs. The review is carried out subject to the following idea: although the development of novel EMSs that seek the optimum performance of HEVs is booming, in the real world, HEVs continue to rely on well-known rule-based (RB) strategies. The contribution of this work is to present a quantitative comparison of the works selected. Since several studies do not provide results of their models against commercial RB strategies, it is proposed, as another contribution, to complete their results using simulations. From these results, it is concluded that the improvement of the analyzed EMSs ranges roughly between 5% and 10% with regard to commercial RB EMSs; in comparison to the optimum, the analyzed EMSs are nearer to the optimum than commercial RB EMSs

Intelligent Transportation Systems, Hybrid Electric Vehicles, Powertrain Control, Cooperative Adaptive Cruise Control, Model Predictive Control

Information obtainable from Intelligent Transportation Systems (ITS) provides the possibility of improving the safety and efficiency of vehicles at different levels. In particular, such information has the potential to be utilized for prediction of driving conditions and traffic flow, which allows us to improve the performance of the control systems in different vehicular applications, such as Hybrid Electric Vehicles (HEVs) powertrain control and Cooperative Adaptive Cruise Control (CACC). In the first part of this work, we study the design of an MPC controller for a Cooperative Adaptive Cruise Control (CACC) system, which is an automated application that provides the drivers with extra benefits, such as traffic throughput maximization and collision avoidance. CACC systems must be designed in a way that are sufficiently robust against all special maneuvers such as interfering vehicles cutting-into the CACC platoons or hard braking by leading cars. To address this problem, we first propose a Neural- Network (NN)-based cut-in detection and trajectory prediction scheme. Then, the predicted trajectory of each vehicle in the adjacent lanes is used to estimate the probability of that vehicle cutting-into the CACC platoon. To consider the calculated probability in control system decisions, a Stochastic Model Predictive Controller (SMPC) needs to be designed which incorporates this cut-in probability, and enhances the reaction against the detected dangerous cut-in maneuver. However, in this work, we propose an alternative way of solving this problem. We convert the SMPC problem into modeling the CACC as a Stochastic Hybrid System (SHS) while we still use a deterministic MPC controller running in the only state of the SHS model. Finally, we find the conditions under which the designed deterministic controller is stable and feasible for the proposed SHS model of the CACC platoon. In the second part of this work, we propose to improve the performance of one of the most promising realtime powertrain control strategies, called Adaptive Equivalent Consumption Minimization Strategy (AECMS), using predicted driving conditions. In this part, two different real-time powertrain control strategies are proposed for HEVs. The first proposed method, including three different variations, introduces an adjustment factor for the cost of using electrical energy (equivalent factor) in AECMS. The factor is proportional to the predicted energy requirements of the vehicle, regenerative braking energy, and the cost of battery charging and discharging in a finite time window. Simulation results using detailed vehicle powertrain models illustrate that the proposed control strategies improve the performance of AECMS in terms of fuel economy by 4\%. Finally, we integrate the recent development in reinforcement learning to design a novel multi-level power distribution control. The proposed controller reacts in two levels, namely high-level and low-level control. The high-level control decision estimates the most probable driving profile matched to the current (and near future) state of the vehicle. Then, the corresponding low-level controller of the selected profile is utilized to distribute the requested power between Electric Motor (EM) and Internal Combustion Engine (ICE). This is important because there is no other prior work addressing this problem using a controller which can adjust its decision to the driving pattern. We proposed to use two reinforcement learning agents in two levels of abstraction. The first agent, selects the most optimal low-level controller (second agent) based on the overall pattern of the drive cycle in the near past and future, i.e., urban, highway and harsh. Then, the selected agent by the high-level controller (first agent) decides how to distribute the demanded power between the EM and ICE. We found that by carefully designing a training scheme, it is possible to effectively improve the performance of this data-driven controller. Simulation results show up to 6\% improvement in fuel economy compared to the AECMS