A Robust and Constrained Multi-Agent Reinforcement Learning Framework for Electric Vehicle AMoD Systems

Electric vehicles (EVs) play critical roles in autonomous mobility-on-demand (AMoD) systems, but their unique charging patterns increase the model uncertainties in AMoD systems (e.g. state transition probability). Since there usually exists a mismatch between the training and test (true) environments, incorporating model uncertainty into system design is of critical importance in real-world applications. However, model uncertainties have not been considered explicitly in EV AMoD system rebalancing by existing literature yet and remain an urgent and challenging task. In this work, we design a robust and constrained multi-agent reinforcement learning (MARL) framework with transition kernel uncertainty for the EV rebalancing and charging problem. We then propose a robust and constrained MARL algorithm (ROCOMA) that trains a robust EV rebalancing policy to balance the supply-demand ratio and the charging utilization rate across the whole city under state transition uncertainty. Experiments show that the ROCOMA can learn an effective and robust rebalancing policy. It outperforms non-robust MARL methods when there are model uncertainties. It increases the system fairness by 19.6% and decreases the rebalancing costs by 75.8%.Comment: 8 page

What is the Solution for State-Adversarial Multi-Agent Reinforcement Learning?

Various types of Multi-Agent Reinforcement Learning (MARL) methods have been developed, assuming that agents' policies are based on true states. Recent works have improved the robustness of MARL under uncertainties from the reward, transition probability, or other partners' policies. However, in real-world multi-agent systems, state estimations may be perturbed by sensor measurement noise or even adversaries. Agents' policies trained with only true state information will deviate from optimal solutions when facing adversarial state perturbations during execution. MARL under adversarial state perturbations has limited study. Hence, in this work, we propose a State-Adversarial Markov Game (SAMG) and make the first attempt to study the fundamental properties of MARL under state uncertainties. We prove that the optimal agent policy and the robust Nash equilibrium do not always exist for an SAMG. Instead, we define the solution concept, robust agent policy, of the proposed SAMG under adversarial state perturbations, where agents want to maximize the worst-case expected state value. We then design a gradient descent ascent-based robust MARL algorithm to learn the robust policies for the MARL agents. Our experiments show that adversarial state perturbations decrease agents' rewards for several baselines from the existing literature, while our algorithm outperforms baselines with state perturbations and significantly improves the robustness of the MARL policies under state uncertainties

Data-Driven Distributionally Robust Electric Vehicle Balancing for Autonomous Mobility-on-Demand Systems under Demand and Supply Uncertainties

Electric vehicles (EVs) are being rapidly adopted due to their economic and societal benefits. Autonomous mobility-on-demand (AMoD) systems also embrace this trend. However, the long charging time and high recharging frequency of EVs pose challenges to efficiently managing EV AMoD systems. The complicated dynamic charging and mobility process of EV AMoD systems makes the demand and supply uncertainties significant when designing vehicle balancing algorithms. In this work, we design a data-driven distributionally robust optimization (DRO) approach to balance EVs for both the mobility service and the charging process. The optimization goal is to minimize the worst-case expected cost under both passenger mobility demand uncertainties and EV supply uncertainties. We then propose a novel distributional uncertainty sets construction algorithm that guarantees the produced parameters are contained in desired confidence regions with a given probability. To solve the proposed DRO AMoD EV balancing problem, we derive an equivalent computationally tractable convex optimization problem. Based on real-world EV data of a taxi system, we show that with our solution the average total balancing cost is reduced by 14.49%, and the average mobility fairness and charging fairness are improved by 15.78% and 34.51%, respectively, compared to solutions that do not consider uncertainties.Comment: 16 page

Multi-Agent Reinforcement Learning Guided by Signal Temporal Logic Specifications

Reward design is a key component of deep reinforcement learning, yet some tasks and designer's objectives may be unnatural to define as a scalar cost function. Among the various techniques, formal methods integrated with DRL have garnered considerable attention due to their expressiveness and flexibility to define the reward and requirements for different states and actions of the agent. However, how to leverage Signal Temporal Logic (STL) to guide multi-agent reinforcement learning reward design remains unexplored. Complex interactions, heterogeneous goals and critical safety requirements in multi-agent systems make this problem even more challenging. In this paper, we propose a novel STL-guided multi-agent reinforcement learning framework. The STL requirements are designed to include both task specifications according to the objective of each agent and safety specifications, and the robustness values of the STL specifications are leveraged to generate rewards. We validate the advantages of our method through empirical studies. The experimental results demonstrate significant reward performance improvements compared to MARL without STL guidance, along with a remarkable increase in the overall safety rate of the multi-agent systems

A New Stress-reduction Model for Soil Arch in Landslides

Stabilizing piles are extensively used as an effective landslide control treatment, and the soil arching effect is the key element for the performance of the pile system. Most previous studies on soil arching effect and its application in stabilizing piles were conducted with laboratory tests and numerical simulations, while limited efforts have been dedicated to the analytical characterization of such a soil-structure interaction. In this paper, a new stress-reduction model for soil arch in landslides is established by theoretical derivation. Our model calculation has demonstrated an exponential reduction in the stress along the direction of slipping between and behind stabilizing piles and thus justifies the observations of laboratory tests and numerical simulations. Thereafter, the analytical solutions to the two key arch shape parameters, namely the inclination angle at the foothold and the thickness of soil arch, are derived based on the proposed stress-reduction model. Then, the ultimate bearing capacity of soil arch between and behind stabilizing piles is subsequently calculated, and a three-level load sharing model for landslides is thus proposed based on the stress-reduction mode. The load sharing model can well capture the stage characteristics of the interaction between landslide mass and stabilizing piles. Finally, the calculation model of spacing between stabilizing piles is established based on the proposed stress-reduction model, and it turns to be good in field application. The findings of this study can contribute to a better understanding of the soil arching effect as well as a better design of the stabilizing piles

Immunization with Fc-based recombinant Epstein-Barr virus gp350 elicits potent neutralizing humoral immune response in a BALB/c mice model

Epstein-Barr virus (EBV) was the first human virus proved to be closely associated with tumor development, such as lymphoma, nasopharyngeal carcinoma (NPC) and EBV-associated gastric carcinoma. Despite many efforts to develop prophylactic vaccines against EBV infection and diseases, no candidates have succeeded in effectively blocking EBV infection in clinical trials. Previous investigations showed that EBV gp350 plays a pivotal role in the infection of B lymphocytes. Nevertheless, using monomeric gp350 proteins as antigens has not been effective in preventing infection. Multimeric forms of the antigen are more potently immunogenic than monomers, however the multimerization elements used in previous constructs are not approved for human clinical trials. To prepare a much-needed EBV prophylactic vaccine that is potent, safe and applicable, we constructed an Fc-based form of gp350 to serve as a dimeric antigen. Here we show that the Fc-based gp350 antigen exhibits dramatically enhanced immunogenicity compared to wild-type gp350 protein. The complete or partial gp350 ectodomain was fused with the mouse IgG2a Fc domain. Fusion with the Fc domain did not impair gp350 folding, binding to a conformation-dependent neutralizing antibody and binding to its receptor by ELISA and SPR. Specific antibody titers against gp350 were notably enhanced by immunization with gp350-Fc dimers compared to gp350 monomers. Furthermore, immunization with gp350-Fc fusion proteins elicited potent neutralizing antibodies against EBV. Our data strongly suggest that an EBV gp350 vaccine based on Fc fusion proteins may be an efficient candidate to prevent EBV infection in clinical applications. Please click Additional Files below to see the full abstract