DeepFreight: A Model-free Deep-reinforcement-learning-based Algorithm for Multi-transfer Freight Delivery

With the freight delivery demands and shipping costs increasing rapidly, intelligent control of fleets to enable efficient and cost-conscious solutions becomes an important problem. In this paper, we propose DeepFreight, a model-free deep-reinforcement-learning-based algorithm for multi-transfer freight delivery, which includes two closely-collaborative components: truck-dispatch and package-matching. Specifically, a deep multi-agent reinforcement learning framework called QMIX is leveraged to learn a dispatch policy, with which we can obtain the multi-step joint dispatch decisions for the fleet with respect to the delivery requests. Then an efficient multi-transfer matching algorithm is executed to assign the delivery requests to the trucks. Also, DeepFreight is integrated with a Mixed-Integer Linear Programming optimizer for further optimization. The evaluation results show that the proposed system is highly scalable and ensures a 100% delivery success while maintaining low delivery time and fuel consumption.Comment: This paper is presented in part at the 31st International Conference on Automated Planning and Scheduling (ICAPS 2021

Optimization of vehicular networks in smart cities: from agile optimization to learnheuristics and simheuristics

Vehicular ad hoc networks (VANETs) are a fundamental component of intelligent transportation systems in smart cities. With the support of open and real-time data, these networks of inter-connected vehicles constitute an ‘Internet of vehicles’ with the potential to significantly enhance citizens’ mobility and last-mile delivery in urban, peri-urban, and metropolitan areas. However, the proper coordination and logistics of VANETs raise a number of optimization challenges that need to be solved. After reviewing the state of the art on the concepts of VANET optimization and open data in smart cities, this paper discusses some of the most relevant optimization challenges in this area. Since most of the optimization problems are related to the need for real-time solutions or to the consideration of uncertainty and dynamic environments, the paper also discusses how some VANET challenges can be addressed with the use of agile optimization algorithms and the combination of metaheuristics with simulation and machine learning methods. The paper also offers a numerical analysis that measures the impact of using these optimization techniques in some related problems. Our numerical analysis, based on real data from Open Data Barcelona, demonstrates that the constructive heuristic outperforms the random scenario in the CDP combined with vehicular networks, resulting in maximizing the minimum distance between facilities while meeting capacity requirements with the fewest facilities.Peer ReviewedPostprint (published version

Dynamic Priority Rules for Combining On-Demand Passenger Transportation and Transportation of Goods

Urban on-demand transportation services are booming, in both passenger transportation and the transportation of goods. The types of service differ in timeliness and compensation and, until now, providers operate larger fleets separately for each type of service. While this may ensure sufficient resources for lucrative passenger transportation, the separation also leaves consolidation potentials untapped. In this paper, we propose combining both services in an anticipatory way that ensures high passenger service rates while simultaneously transporting a large number of goods. To this end, we introduce a dynamic priority policy that uses a time-dependent percentage of vehicles mainly to serve passengers. To find effective time-dependent parametrizations given a limited number of runtime-expensive simulations, we apply Bayesian Optimization. We show that our anticipatory policy increases revenue and service rates significantly while a myopic combination of service may actually lead to inferior performance compared to using two separate fleets

Approximate Multiagent Reinforcement Learning for On-Demand Urban Mobility Problem on a Large Map (extended version)

In this paper, we focus on the autonomous multiagent taxi routing problem for a large urban environment where the location and number of future ride requests are unknown a-priori, but follow an estimated empirical distribution. Recent theory has shown that if a base policy is stable then a rollout-based algorithm with such a base policy produces a near-optimal stable policy. Although, rollout-based approaches are well-suited for learning cooperative multiagent policies with considerations for future demand, applying such methods to a large urban environment can be computationally expensive. Large environments tend to have a large volume of requests, and hence require a large fleet of taxis to guarantee stability. In this paper, we aim to address the computational bottleneck of multiagent (one-at-a-time) rollout, where the computational complexity grows linearly in the number of agents. We propose an approximate one-at-a-time rollout-based two-phase algorithm that reduces the computational cost, while still achieving a stable near-optimal policy. Our approach partitions the graph into sectors based on the predicted demand and an user-defined maximum number of agents that can be planned for using the one-at-a-time rollout approach. The algorithm then applies instantaneous assignment (IA) for re-balancing taxis across sectors and a sector-wide one-at-a-time rollout algorithm that is executed in parallel for each sector. We characterize the number of taxis $m$ that is sufficient for IA base policy to be stable, and derive a necessary condition on $m$ as time goes to infinity. Our numerical results show that our approach achieves stability for an $m$ that satisfies the theoretical conditions. We also empirically demonstrate that our proposed two-phase algorithm has comparable performance to the one-at-a-time rollout over the entire map, but with significantly lower runtimes.Comment: 11 pages, 5 figures, 1 lemma, and 2 theorem

Peer-to-Peer Energy Trading in Smart Residential Environment with User Behavioral Modeling

Electric power systems are transforming from a centralized unidirectional market to a decentralized open market. With this shift, the end-users have the possibility to actively participate in local energy exchanges, with or without the involvement of the main grid. Rapidly reducing prices for Renewable Energy Technologies (RETs), supported by their ease of installation and operation, with the facilitation of Electric Vehicles (EV) and Smart Grid (SG) technologies to make bidirectional flow of energy possible, has contributed to this changing landscape in the distribution side of the traditional power grid. Trading energy among users in a decentralized fashion has been referred to as Peer- to-Peer (P2P) Energy Trading, which has attracted significant attention from the research and industry communities in recent times. However, previous research has mostly focused on engineering aspects of P2P energy trading systems, often neglecting the central role of users in such systems. P2P trading mechanisms require active participation from users to decide factors such as selling prices, storing versus trading energy, and selection of energy sources among others. The complexity of these tasks, paired with the limited cognitive and time capabilities of human users, can result sub-optimal decisions or even abandonment of such systems if performance is not satisfactory. Therefore, it is of paramount importance for P2P energy trading systems to incorporate user behavioral modeling that captures users’ individual trading behaviors, preferences, and perceived utility in a realistic and accurate manner. Often, such user behavioral models are not known a priori in real-world settings, and therefore need to be learned online as the P2P system is operating. In this thesis, we design novel algorithms for P2P energy trading. By exploiting a variety of statistical, algorithmic, machine learning, and behavioral economics tools, we propose solutions that are able to jointly optimize the system performance while taking into account and learning realistic model of user behavior. The results in this dissertation has been published in IEEE Transactions on Green Communications and Networking 2021, Proceedings of IEEE Global Communication Conference 2022, Proceedings of IEEE Conference on Pervasive Computing and Communications 2023 and ACM Transactions on Evolutionary Learning and Optimization 2023

Fleet rebalancing for expanding shared e-mobility systems : a multi-agent deep reinforcement learning approach

The electrification of shared mobility has become popular across the globe. Many cities have their new shared e-mobility systems deployed, with continuously expanding coverage from central areas to the city edges. A key challenge in the operation of these systems is fleet rebalancing, i.e., how EVs should be repositioned to better satisfy future demand. This is particularly challenging in the context of expanding systems, because i) the range of the EVs is limited while charging time is typically long, which constrain the viable rebalancing operations; and ii) the EV stations in the system are dynamically changing, i.e., the legitimate targets for rebalancing operations can vary over time. We tackle these challenges by first investigating rich sets of data collected from a real-world shared e-mobility system for one year, analyzing the operation model, usage patterns and expansion dynamics of this new mobility mode. With the learned knowledge we design a high-fidelity simulator, which is able to abstract key operation details of EV sharing at fine granularity. Then we model the rebalancing task for shared e-mobility systems under continuous expansion as a Multi-Agent Reinforcement Learning (MARL) problem, which directly takes the range and charging properties of the EVs into account. We further propose a novel policy optimization approach with action cascading, which is able to cope with the expansion dynamics and solve the formulated MARL. We evaluate the proposed approach extensively, and experimental results show that our approach outperforms the state-of-the-art, offering significant performance gain in both satisfied demand and net revenue