Deep Reinforcement Learning for a Multi-Objective Online Order Batching Problem

On-time delivery and low service costs are two important performance metrics in warehousing operations. This paper proposes a Deep Reinforcement Learning (DRL) based approach to solve the online Order Batching and Sequence Problem (OBSP) to optimize these two objectives. To learn how to balance the trade-off between two objectives, we introduce a Bayesian optimization framework to shape the reward function of the DRL agent, such that the influences of learning to these objectives are adjusted to different environments. We compare our approach with several heuristics using problem instances of real-world size where thousands of orders arrive dynamically per hour. We show the Proximal Policy Optimization (PPO) algorithm with Bayesian optimization outperforms the heuristics in all tested scenarios on both objectives. In addition, it finds different weights for the components in the reward function in different scenarios, indicating its capability of learning how to set the importance of two objectives under different environments. We also provide policy analysis on the learned DRL agent, where a decision tree is used to infer decision rules to enable the interpretability of the DRL approach

Relational Graph Attention-Based Deep Reinforcement Learning:An Application to Flexible Job Shop Scheduling with Sequence-Dependent Setup Times

This paper tackles a manufacturing scheduling problem using an Edge Guided Relational Graph Attention-based Deep Reinforcement Learning approach. Unlike state-of-the-art approaches, the proposed method can deal with machine flexibility and sequence dependency of the setup times in the Job Shop Scheduling Problem. Furthermore, the proposed approach is size-agnostic. We evaluated our method against standard priority dispatching rules based on data that reflect a realistic scenario, designed on the basis of a practical case study at the Dassault Systèmes company. We used an industry-strength large neighborhood search based algorithm as benchmark. The results show that the proposed method outperforms the priority dispatching rules in terms of makespan, obtaining an average makespan difference with the best tested priority dispatching rules of 4.45% and 12.52%.</p

Relational Graph Attention-Based Deep Reinforcement Learning:An Application to Flexible Job Shop Scheduling with Sequence-Dependent Setup Times

This paper tackles a manufacturing scheduling problem using an Edge Guided Relational Graph Attention-based Deep Reinforcement Learning approach. Unlike state-of-the-art approaches, the proposed method can deal with machine flexibility and sequence dependency of the setup times in the Job Shop Scheduling Problem. Furthermore, the proposed approach is size-agnostic. We evaluated our method against standard priority dispatching rules based on data that reflect a realistic scenario, designed on the basis of a practical case study at the Dassault Systèmes company. We used an industry-strength large neighborhood search based algorithm as benchmark. The results show that the proposed method outperforms the priority dispatching rules in terms of makespan, obtaining an average makespan difference with the best tested priority dispatching rules of 4.45% and 12.52%.</p

Tackling Uncertainty in Online Multimodal Transportation Planning Using Deep Reinforcement Learning

In this paper we tackle the container allocation problem in multimodal transportation planning under uncertainty in container arrival times, using Deep Reinforcement Learning. The proposed approach can take real-time decisions on allocating individual containers to a truck or to trains, while a transportation plan is being executed. We evaluated our method using data that reflect a realistic scenario, designed on the basis of a case study at a logistics company with three different uncertainty levels based on the probability of delays in container arrivals. The experiments show that Deep Reinforcement Learning methods outperform heuristics, a stochastic programming method, and methods that use periodic re-planning, in terms of total transportation costs at all levels of uncertainty, obtaining an average cost difference with the optimal solution within 0.37% and 0.63%

Deep Reinforcement Learning for a Multi-Objective Online Order Batching Problem

On-time delivery and low service costs are two important performance metrics in warehousing operations. This paper proposes a Deep Reinforcement Learning (DRL) based approach to solve the online Order Batching and Sequence Problem (OBSP) to optimize these two objectives. To learn how to balance the trade-off between two objectives, we introduce a Bayesian optimization framework to shape the reward function of the DRL agent, such that the influences of learning to these objectives are adjusted to different environments. We compare our approach with several heuristics using problem instances of real-world size where thousands of orders arrive dynamically per hour. We show the Proximal Policy Optimization (PPO) algorithm with Bayesian optimization outperforms the heuristics in all tested scenarios on both objectives. In addition, it finds different weights for the components in the reward function in different scenarios, indicating its capability of learning how to set the importance of two objectives under different environments. We also provide policy analysis on the learned DRL agent, where a decision tree is used to infer decision rules to enable the interpretability of the DRL approach

From Postpartum Haemorrhage Guideline to Local Protocol: A Study of Protocol Quality

Objective Postpartum hemorrhage (PPH) has a continuously rising incidence worldwide, suggesting suboptimal care. An important step in optimizing care is the translation of evidence-based guidelines into comprehensive hospital protocols. However, knowledge about the quality of these protocols is lacking. The objective of this study was to evaluate the quality of PPH-protocols on structure and content in the Netherlands. Methods We performed an observational multicenter study. Eighteen PPH-protocols from 3 University Hospitals (UH), 8 Teaching Hospitals (TH) and 7 Non-Teaching hospitals (NTH) throughout the Netherlands were acquired. The structure of the PPH-protocols was assessed using the Appraisal of Guidelines for Research and Evaluation (AGREE-II) Instrument. The content was appraised using previously developed quality indicators, based on international guidelines and Advance-Trauma-Life-Support (ATLS)-based course instructions. Results The quality of the protocols for postpartum hemorrhage for both structure and content varied widely between different hospitals, but all of them showed room for improvement. The protocols scored mainly below average on the different items of the AGREE-II instrument (8 of the 10 items scored <4 on a 1-7 scale). Regarding the content, adoption of guideline recommendations in protocols was 46 %. In addition, a timely indication of 'when to perform' a recommendation was lacking in three-fourths of the items. Conclusion This study shows that the quality of the PPH-protocols for both structure and content in the Netherlands is suboptimal. This makes adherence to the guideline and ATLS-based course instructions difficul

Evaluating Adherence to Guideline-Based Quality Indicators for Postpartum Hemorrhage Care in the Netherlands Using Video Analysis

Objective: To assess adherence to the national postpartum hemorrhage guideline and Managing Obstetric Emergencies and Trauma course instructions and its determinants in the Netherlands. Methods: A prospective observational multicenter study in 16 Dutch hospitals analyzing data from medical records of 398 women at high risk for postpartum hemorrhage, of which 293 were supplemented with data from prospective video recordings. Adherence to guideline-based quality indicators for prevention, management, and organization of postpartum hemorrhage care was measured. Indicators for prevention and management of postpartum hemorrhage were categorized according to the amount of blood loss (less than 500, greater than 500, greater than 1,000, and greater than 2,000 mL). Results: Overall, a lack of adherence was observed, particularly for the actions to be undertaken with blood loss greater than 1,000 mL (69 patients). Actions were not or only taken in a later stage when the blood loss had already increased to greater than 2,000 mL (21 patients). In almost 41% (n5119/293) of the deliveries, no active management was performed, and in almost 80% (n589/112), vital signs were not monitored (blood loss greater than 500 mL) or monitored too late with respect to blood loss. The video recordings showed that in general the actual care given was considerably underreported in medical records. Postpartum hemorrhage care in the hospitals was well organized. Fifteen hospitals had a local postpartum hemorrhage protocol, and in 12 hospitals, team trainings were organized. Regarding the determinants high-risk patient identification and type of hospital (university vs nonuniversity hospital) were mostly associated with better adherence. Conclusion: This study showed low adherence to the guideline-based quality indicators, indicating a problem with Dutch quality care. The unique video observations provided additional, valuable information at which level improvement can be made. A tailor-made implementation strategy to improve quality of postpartum hemorrhage care has been developed