Vehicle Dispatching and Routing of On-Demand Intercity Ride-Pooling Services: A Multi-Agent Hierarchical Reinforcement Learning Approach

The integrated development of city clusters has given rise to an increasing demand for intercity travel. Intercity ride-pooling service exhibits considerable potential in upgrading traditional intercity bus services by implementing demand-responsive enhancements. Nevertheless, its online operations suffer the inherent complexities due to the coupling of vehicle resource allocation among cities and pooled-ride vehicle routing. To tackle these challenges, this study proposes a two-level framework designed to facilitate online fleet management. Specifically, a novel multi-agent feudal reinforcement learning model is proposed at the upper level of the framework to cooperatively assign idle vehicles to different intercity lines, while the lower level updates the routes of vehicles using an adaptive large neighborhood search heuristic. Numerical studies based on the realistic dataset of Xiamen and its surrounding cities in China show that the proposed framework effectively mitigates the supply and demand imbalances, and achieves significant improvement in both the average daily system profit and order fulfillment ratio

Energy Efficient and Secure Wireless Sensor Networks Design

Wireless Sensor Networks (WSNs) are emerging technologies that have the ability to sense, process, communicate, and transmit information to a destination, and they are expected to have significant impact on the efficiency of many applications in various fields. The resource constraint such as limited battery power, is the greatest challenge in WSNs design as it affects the lifetime and performance of the network. An energy efficient, secure, and trustworthy system is vital when a WSN involves highly sensitive information. Thus, it is critical to design mechanisms that are energy efficient and secure while at the same time maintaining the desired level of quality of service. Inspired by these challenges, this dissertation is dedicated to exploiting optimization and game theoretic approaches/solutions to handle several important issues in WSN communication, including energy efficiency, latency, congestion, dynamic traffic load, and security. We present several novel mechanisms to improve the security and energy efficiency of WSNs. Two new schemes are proposed for the network layer stack to achieve the following: (a) to enhance energy efficiency through optimized sleep intervals, that also considers the underlying dynamic traffic load and (b) to develop the routing protocol in order to handle wasted energy, congestion, and clustering. We also propose efficient routing and energy-efficient clustering algorithms based on optimization and game theory. Furthermore, we propose a dynamic game theoretic framework (i.e., hyper defense) to analyze the interactions between attacker and defender as a non-cooperative security game that considers the resource limitation. All the proposed schemes are validated by extensive experimental analyses, obtained by running simulations depicting various situations in WSNs in order to represent real-world scenarios as realistically as possible. The results show that the proposed schemes achieve high performance in different terms, such as network lifetime, compared with the state-of-the-art schemes

Decision-making and problem-solving methods in automation technology

The state of the art in the automation of decision making and problem solving is reviewed. The information upon which the report is based was derived from literature searches, visits to university and government laboratories performing basic research in the area, and a 1980 Langley Research Center sponsored conferences on the subject. It is the contention of the authors that the technology in this area is being generated by research primarily in the three disciplines of Artificial Intelligence, Control Theory, and Operations Research. Under the assumption that the state of the art in decision making and problem solving is reflected in the problems being solved, specific problems and methods of their solution are often discussed to elucidate particular aspects of the subject. Synopses of the following major topic areas comprise most of the report: (1) detection and recognition; (2) planning; and scheduling; (3) learning; (4) theorem proving; (5) distributed systems; (6) knowledge bases; (7) search; (8) heuristics; and (9) evolutionary programming

Modeling and Optimizing Routing Decisions for Travelers and On-demand Service Providers

This thesis investigates the dynamic routing decisions for individual travelers and on-demand service providers (e.g., regular taxis, Uber, Lyft, etc). For individual travelers, this thesis models and predicts route choice at two time-scales: the day-to-day and within-day. For day-to-day route choice, methodological development and empirical evidences are presented to understand the roles of learning, inertia and real-time travel information on route choices in a highly disrupted network based on data from a laboratory competitive route choice game. The learning of routing policies instead of simple paths is modeled when real-time travel information is available, where a routing policy is defined as a contingency plan that maps realized traffic conditions to path choices. Using data from a competitive laboratory experiment, prediction performance is then measured in terms of both one-step and full trajectory predictions. For within day route choice, a recursive logit model is formulated in a stochastic time-dependent (STD) network without sampling any choice sets. A decomposition algorithm is then proposed so that the model can be estimated in reasonable time. Estimation and prediction results of the proposed model are presented using a data set collected from a subnetwork of Stockholm, Sweden. Taxis and ride-sourcing vehicles play an important role in providing on-demand mobility in an urban transportation system. Unlike individual travelers, they do not have a clear destination when there\u27s no passenger on board. The optimal routing of a vacant taxi is formulated as a Markov Decision Process (MDP) problem to maximize long-term profit over the full working period. Two approaches are proposed to solve the problem. One is the model-based approach where a model of the state transitions of the environment is obtained from queuing-theory based passenger arrival and competing taxi distribution processes. An enhanced value iteration for solving the MDP problem is then proposed making use of efficient matrix operations. The other is the model-free Reinforcement Learning (RL) approach, which learns the best policy directly from observed trajectory data. Both approaches are implemented and tested in a mega city transportation network with reasonable running time, and a systematic comparison of the two approaches is also provided

Optimizing Transportation Systems with Information Provision, Personalized Incentives and Driver Cooperation

Poor performance of the transportation systems has many detrimental effects such as higher travel times, increased travel costs, higher energy consumption, and greenhouse gas emissions, etc. This thesis optimizes the transportation systems by addressing the traffic congestion problem and climate change impact resulting from the inefficient operation of these systems. I first focus on the key player of the transportation systems e.g., human being/traveler, and model travelers\u27 route choice behavior with real-time information. In this study, I define looking-ahead behavior in route choice as a traveler\u27s taking into account future diversion possibilities enabled by real-time information in a network with random travel times. Subjects participated in route-choice experiments in a driving simulator as well a PC-based environment. Three types of maps in increasing levels of complexity and information availability are used. Aggregate data analysis shows that network complexity negatively affects subjects\u27 ratio of choosing the risky route given an experiment environment. Higher cognitive load in the driving simulator results in a higher level of risk aversion than in the PC-based environment for the simplest map. I specify and estimate a mixed logit model with two latent classes, looking-ahead and myopic, taking into account the panel effect. The estimated latent class membership function suggests that some subjects can look ahead while others are myopic in making their route choices, and drivers learn to look ahead over time. The experiment environment plays a role in the risk attitude of myopic subjects. A bias against information is found for subjects who look ahead, however, is not significant among myopic subjects. I then shift my focus to influencing the travel patterns of individual travelers to reduce the energy and environmental impacts of the transportation sector. I present the system optimization (SO) framework of Tripod, an integrated bi-level transportation management system aimed at maximizing energy savings of the multi-modal transportation systems. From the user\u27s perspective, Tripod is a smartphone app, accessed before performing trips. The app proposes a series of alternatives each with an amount of tokens which the user can later redeem for goods or services. The role of SO is to compute the optimized set of tokens associated to the available alternatives, in order to minimize the system-wide energy consumption, under a limited token budget. I present a method to solve this complex optimization problem and describe the system architecture, the multimodal simulation-based optimization model and the heuristic method for the on-line computation of the optimized token allocation. I then present the framework with the simulation results. Finally, I optimize the systems travel time by addressing the equity issue of congestion pricing. I propose an alternative approach to an equitable and Pareto-improving transportation systems based on cooperation among travelers assisted by defector penalty. Theoretical analysis shows the existence condition of the cooperative scheme for heterogeneous value of time (VOT) of travelers. I formulate a mathematical programming problem for the optimal cooperative scheme problem in a general network with Pareto-improving constraints and practical considerations on the length the cooperation cycle. I then conduct computational tests on a simple network and evaluate the solutions in terms of efficiency improvement (total system travel time) and equitability (Gini index)

Peer-to-Peer Energy Trading in Smart Residential Environment with User Behavioral Modeling

Electric power systems are transforming from a centralized unidirectional market to a decentralized open market. With this shift, the end-users have the possibility to actively participate in local energy exchanges, with or without the involvement of the main grid. Rapidly reducing prices for Renewable Energy Technologies (RETs), supported by their ease of installation and operation, with the facilitation of Electric Vehicles (EV) and Smart Grid (SG) technologies to make bidirectional flow of energy possible, has contributed to this changing landscape in the distribution side of the traditional power grid. Trading energy among users in a decentralized fashion has been referred to as Peer- to-Peer (P2P) Energy Trading, which has attracted significant attention from the research and industry communities in recent times. However, previous research has mostly focused on engineering aspects of P2P energy trading systems, often neglecting the central role of users in such systems. P2P trading mechanisms require active participation from users to decide factors such as selling prices, storing versus trading energy, and selection of energy sources among others. The complexity of these tasks, paired with the limited cognitive and time capabilities of human users, can result sub-optimal decisions or even abandonment of such systems if performance is not satisfactory. Therefore, it is of paramount importance for P2P energy trading systems to incorporate user behavioral modeling that captures users’ individual trading behaviors, preferences, and perceived utility in a realistic and accurate manner. Often, such user behavioral models are not known a priori in real-world settings, and therefore need to be learned online as the P2P system is operating. In this thesis, we design novel algorithms for P2P energy trading. By exploiting a variety of statistical, algorithmic, machine learning, and behavioral economics tools, we propose solutions that are able to jointly optimize the system performance while taking into account and learning realistic model of user behavior. The results in this dissertation has been published in IEEE Transactions on Green Communications and Networking 2021, Proceedings of IEEE Global Communication Conference 2022, Proceedings of IEEE Conference on Pervasive Computing and Communications 2023 and ACM Transactions on Evolutionary Learning and Optimization 2023