Cooperative Multi-Agent Reinforcement Learning for Inventory Management

With Reinforcement Learning (RL) for inventory management (IM) being a nascent field of research, approaches tend to be limited to simple, linear environments with implementations that are minor modifications of off-the-shelf RL algorithms. Scaling these simplistic environments to a real-world supply chain comes with a few challenges such as: minimizing the computational requirements of the environment, specifying agent configurations that are representative of dynamics at real world stores and warehouses, and specifying a reward framework that encourages desirable behavior across the whole supply chain. In this work, we present a system with a custom GPU-parallelized environment that consists of one warehouse and multiple stores, a novel architecture for agent-environment dynamics incorporating enhanced state and action spaces, and a shared reward specification that seeks to optimize for a large retailer's supply chain needs. Each vertex in the supply chain graph is an independent agent that, based on its own inventory, able to place replenishment orders to the vertex upstream. The warehouse agent, aside from placing orders from the supplier, has the special property of also being able to constrain replenishment to stores downstream, which results in it learning an additional allocation sub-policy. We achieve a system that outperforms standard inventory control policies such as a base-stock policy and other RL-based specifications for 1 product, and lay out a future direction of work for multiple products.Comment: 14 pages, 5 figure

Impact of travel time uncertainties on the solution cost of a two-echelon vehicle routing problem with synchronization

Two-echelon vehicle routing problems which contain synchronization between vehicles can be deeply impacted by time uncertainty, because one vehicle's delay can propagate to other vehicles. In this paper, we evaluate the deterministic solution of such a problem based on simulated travel time scenarios. The information obtained by simulation is incorporated in the optimization procedure iteratively. Computational results show that the degree of synchronization in an instance is directly correlated with the potential improvements by reoptimization. We present findings on the number of travel time scenarios required to obtain a representative picture of the stochastic solutions. In addition, we demonstrate that time dependent travel times can be aggregated on a city-wide level and linearized as a function of free flow times without major loss of reliability

Collaborative urban transportation : Recent advances in theory and practice

We thank the Leibniz Association for sponsoring the Dagstuhl Seminar 16091, at which the work presented here was initiated. We also thank Leena Suhl for her comments on an early version of this work. Finally, we thank the anonymous reviewers for the constructive comments.Peer reviewedPostprin

Minimising Total Costs of a Two-Echelon Multi-Depot Capacitated Vehicle Routing Problem (2E-MD-CVRP) that Describes the Utilisation of the Amsterdam City Canal Network for Last Mile Parcel Delivery

Part 13: Smart Methods and Techniques for Sustainable Supply Chain ManagementInternational audienceAn increase in e-shopping and (last mile) parcel deliveries has contributed to a rapid growth of urban freight transportation. This generates major impacts on city sustainability and liveability. Current solutions for urban logistics concern road traffic, but multiple Dutch cities have an extensive range of city canals that could be used for freight transportation over water. It was investigated how the city canal network of Amsterdam can be utilised for last mile parcel delivery, and what the related effects are. A MILP formulation of a Two-Echelon, Multi-Depot, Capacitated Vehicle Routing Problem (2E-MD-CVRP) was developed. The model describes a network in which ships transport parcels to pre-determined satellite locations in the city centre, where the parcels are transferred to cargo e-bikes for the last mile of the delivery to the customer. The model was optimised by minimising the total costs, using the Genetic Algorithm (GA). The algorithm was able to find solutions but could not always stay within the constrained search space. Different possible network scenarios were evaluated, describing the consequences with respect to emissions, costs, and traffic flows. The results show promising economic, social, and environmental outcomes for a network with ships and cargo e-bikes instead of delivery vans. A daily and investment cost reduction of 16% and 36% respectively and a $$CO_2$$CO2 emission reduction of 78.26% can be realised

Minimising Total Costs of a Two-Echelon Multi-Depot Capacitated Vehicle Routing Problem (2E-MD-CVRP) that Describes the Utilisation of the Amsterdam City Canal Network for Last Mile Parcel Delivery

An increase in e-shopping and (last mile) parcel deliveries has contributed to a rapid growth of urban freight transportation. This generates major impacts on city sustainability and liveability. Current solutions for urban logistics concern road traffic, but multiple Dutch cities have an extensive range of city canals that could be used for freight transportation over water. It was investigated how the city canal network of Amsterdam can be utilised for last mile parcel delivery, and what the related effects are. A MILP formulation of a Two-Echelon, Multi-Depot, Capacitated Vehicle Routing Problem (2E-MD-CVRP) was developed. The model describes a network in which ships transport parcels to pre-determined satellite locations in the city centre, where the parcels are transferred to cargo e-bikes for the last mile of the delivery to the customer. The model was optimised by minimising the total costs, using the Genetic Algorithm (GA). The algorithm was able to find solutions but could not always stay within the constrained search space. Different possible network scenarios were evaluated, describing the consequences with respect to emissions, costs, and traffic flows. The results show promising economic, social, and environmental outcomes for a network with ships and cargo e-bikes instead of delivery vans. A daily and investment cost reduction of 16% and 36% respectively and a 𝐶𝑂2 emission reduction of 78.26% can be realised

A concise guide to existing and emerging vehicle routing problem variants

Vehicle routing problems have been the focus of extensive research over the past sixty years, driven by their economic importance and their theoretical interest. The diversity of applications has motivated the study of a myriad of problem variants with different attributes. In this article, we provide a concise overview of existing and emerging problem variants. Models are typically refined along three lines: considering more relevant objectives and performance metrics, integrating vehicle routing evaluations with other tactical decisions, and capturing fine-grained yet essential aspects of modern supply chains. We organize the main problem attributes within this structured framework. We discuss recent research directions and pinpoint current shortcomings, recent successes, and emerging challenges