Formula-E multi-car race strategy development—a novel approach using reinforcement learning

Electric motorsport such as Formula E is becoming more and more popular in recent years. Race strategy in such races can be very complex involving resource management, e.g. energy and thermal management, but more importantly multi-car interactions which could be both collaborative and competitive. Reinforcement Learning has been implemented in the literature for such electric racing strategy development but only accounts for one single car. In this paper, we proposed a new architecture iRaXL to implement reinforcement learning for such complex strategy development featuring hybrid action space, multi-car interactions, and non-zero-sum gaming. The iRaXL proves to be able to develop different strategies for individual competitors and also team-based objectives. In a bigger scope, this framework can be used to solve more generic problems with hybrid features such as zero/non-zero-sum games, discretized/continuous action space, and competition/collaboration interactions

Application of advanced tree search and proximal policy optimization on formula-E race strategy development

Energy and thermal management is a crucial element in Formula-E race strategy development. Most published literature focuses on the optimal management strategy for a single lap and results in sub-optimal solutions to the larger multi-lap problem. In this study, two Monte Carlo Tree Search (MCTS) enhancement techniques are proposed for multi-lap Formula-E racing strategy development. It is shown that using the bivariate Gaussian distribution enhancement, race finishing time improves by at least 0.25% and its variance reduces by more than 26%. Compared to the published conventional MCTS technique used in multi-lap problems, this proposed technique is proved to bring a remarkable enhancement with no additional computational time cost. By further enhancing the MCTS using proximal policy optimization, the final product is capable of generating more than 0.5% quicker race time solutions and improving the consistency by over 90% which makes it a very suitable method particularly when enough training time is guarantee

Formula-E race strategy development using distributed policy gradient reinforcement learning

Energy and thermal management is a crucial element in Formula-E race strategy development. In this study, the race-level strategy development is formulated into a Markov decision process (MDP) problem featuring a hybrid-type action space. Deep Deterministic Policy Gradient (DDPG) reinforcement learning is implemented under distributed architecture Ape-X and integrated with the prioritized experience replay and reward shaping techniques to optimize a hybrid-type set of actions of both continuous and discrete components. Soft boundary violation penalties in reward shaping, significantly improves the performance of DDPG and makes it capable of generating faster race finishing solutions. The new proposed method has shown superior performance in comparison to the Monte Carlo Tree Search (MCTS) with policy gradient reinforcement learning, which solves this problem in a fully discrete action space as presented in the literature. The advantages are faster race finishing time and better handling of ambient temperature rise