Human-Machine Collaborative Optimization via Apprenticeship Scheduling

Coordinating agents to complete a set of tasks with intercoupled temporal and resource constraints is computationally challenging, yet human domain experts can solve these difficult scheduling problems using paradigms learned through years of apprenticeship. A process for manually codifying this domain knowledge within a computational framework is necessary to scale beyond the ``single-expert, single-trainee" apprenticeship model. However, human domain experts often have difficulty describing their decision-making processes, causing the codification of this knowledge to become laborious. We propose a new approach for capturing domain-expert heuristics through a pairwise ranking formulation. Our approach is model-free and does not require enumerating or iterating through a large state space. We empirically demonstrate that this approach accurately learns multifaceted heuristics on a synthetic data set incorporating job-shop scheduling and vehicle routing problems, as well as on two real-world data sets consisting of demonstrations of experts solving a weapon-to-target assignment problem and a hospital resource allocation problem. We also demonstrate that policies learned from human scheduling demonstration via apprenticeship learning can substantially improve the efficiency of a branch-and-bound search for an optimal schedule. We employ this human-machine collaborative optimization technique on a variant of the weapon-to-target assignment problem. We demonstrate that this technique generates solutions substantially superior to those produced by human domain experts at a rate up to 9.5 times faster than an optimization approach and can be applied to optimally solve problems twice as complex as those solved by a human demonstrator.Comment: Portions of this paper were published in the Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI) in 2016 and in the Proceedings of Robotics: Science and Systems (RSS) in 2016. The paper consists of 50 pages with 11 figures and 4 table

Procedural Optimization Models for Multiobjective Flexible JSSP

The most challenging issues related to manufacturing efficiency occur if the jobs to be sched-uled are structurally different, if these jobs allow flexible routings on the equipments and mul-tiple objectives are required. This framework, called Multi-objective Flexible Job Shop Scheduling Problems (MOFJSSP), applicable to many real processes, has been less reported in the literature than the JSSP framework, which has been extensively formalized, modeled and analyzed from many perspectives. The MOFJSSP lie, as many other NP-hard problems, in a tedious place where the vast optimization theory meets the real world context. The paper brings to discussion the most optimization models suited to MOFJSSP and analyzes in detail the genetic algorithms and agent-based models as the most appropriate procedural models

Adaptive Order Dispatching based on Reinforcement Learning: Application in a Complex Job Shop in the Semiconductor Industry

Heutige Produktionssysteme tendieren durch die Marktanforderungen getrieben zu immer kleineren Losgrößen, höherer Produktvielfalt und größerer Komplexität der Materialflusssysteme. Diese Entwicklungen stellen bestehende Produktionssteuerungsmethoden in Frage. Im Zuge der Digitalisierung bieten datenbasierte Algorithmen des maschinellen Lernens einen alternativen Ansatz zur Optimierung von Produktionsabläufen. Aktuelle Forschungsergebnisse zeigen eine hohe Leistungsfähigkeit von Verfahren des Reinforcement Learning (RL) in einem breiten Anwendungsspektrum. Im Bereich der Produktionssteuerung haben sich jedoch bisher nur wenige Autoren damit befasst. Eine umfassende Untersuchung verschiedener RL-Ansätze sowie eine Anwendung in der Praxis wurden noch nicht durchgeführt. Unter den Aufgaben der Produktionsplanung und -steuerung gewährleistet die Auftragssteuerung (order dispatching) eine hohe Leistungsfähigkeit und Flexibilität der Produktionsabläufe, um eine hohe Kapazitätsauslastung und kurze Durchlaufzeiten zu erreichen. Motiviert durch komplexe Werkstattfertigungssysteme, wie sie in der Halbleiterindustrie zu finden sind, schließt diese Arbeit die Forschungslücke und befasst sich mit der Anwendung von RL für eine adaptive Auftragssteuerung. Die Einbeziehung realer Systemdaten ermöglicht eine genauere Erfassung des Systemverhaltens als statische Heuristiken oder mathematische Optimierungsverfahren. Zusätzlich wird der manuelle Aufwand reduziert, indem auf die Inferenzfähigkeiten des RL zurückgegriffen wird. Die vorgestellte Methodik fokussiert die Modellierung und Implementierung von RL-Agenten als Dispatching-Entscheidungseinheit. Bekannte Herausforderungen der RL-Modellierung in Bezug auf Zustand, Aktion und Belohnungsfunktion werden untersucht. Die Modellierungsalternativen werden auf der Grundlage von zwei realen Produktionsszenarien eines Halbleiterherstellers analysiert. Die Ergebnisse zeigen, dass RL-Agenten adaptive Steuerungsstrategien erlernen können und bestehende regelbasierte Benchmarkheuristiken übertreffen. Die Erweiterung der Zustandsrepräsentation verbessert die Leistung deutlich, wenn ein Zusammenhang mit den Belohnungszielen besteht. Die Belohnung kann so gestaltet werden, dass sie die Optimierung mehrerer Zielgrößen ermöglicht. Schließlich erreichen spezifische RL-Agenten-Konfigurationen nicht nur eine hohe Leistung in einem Szenario, sondern weisen eine Robustheit bei sich ändernden Systemeigenschaften auf. Damit stellt die Forschungsarbeit einen wesentlichen Beitrag in Richtung selbstoptimierender und autonomer Produktionssysteme dar. Produktionsingenieure müssen das Potenzial datenbasierter, lernender Verfahren bewerten, um in Bezug auf Flexibilität wettbewerbsfähig zu bleiben und gleichzeitig den Aufwand für den Entwurf, den Betrieb und die Überwachung von Produktionssteuerungssystemen in einem vernünftigen Gleichgewicht zu halten

Genetic Algorithm for Job Scheduling with Maintenance Consideration in Semiconductor Manufacturing Process

This paper presents wafer sequencing problems considering perceived chamber conditions and maintenance activities in a single cluster tool through the simulation-based optimization method. We develop optimization methods which would lead to the best wafer release policy in the chamber tool to maximize the overall yield of the wafers in semiconductor manufacturing system. Since chamber degradation will jeopardize wafer yields, chamber maintenance is taken into account for the wafer sequence decision-making process. Furthermore, genetic algorithm is modified for solving the scheduling problems in this paper. As results, it has been shown that job scheduling has to be managed based on the chamber degradation condition and maintenance activities to maximize overall wafer yield.open

Controlling the order pool in make-to-order production systems

Voor ‘Make-To-Order’ (MTO, oftewel klantordergestuurde) productiesystemen is de tijd die orders moeten wachten op beschikbare productiecapaciteit cruciaal. Het beheersen van die wachttijd is van groot belang om zowel korte als betrouwbare doorlooptijden te realiseren. Daarom analyseerde en ontwierp Remco Germs regels voor orderacceptatie en ordervrijgave, om daarmee de wachttijden te beheersen. Orderacceptatie en -vrijgave zijn de twee belangrijkste mechanismen om de lengte van wachttijden te beïnvloeden en zodoende de productie te sturen. De logistieke prestatie hangt in grote mate af van specifieke kenmerken van MTO-systemen, zoals routing variabiliteit, beperkte productiecapaciteit, omsteltijden, strikte leveringsvoorwaarden en onzekerheid in het aankomstpatroon van orders. Om een beter begrip te krijgen van de afwegingen die MTO-bedrijven in dit opzicht moeten maken richt het proefschrift zich op de modellering van de belangrijkste kenmerken van MTO-systemen. De inzichten die dat oplevert worden vervolgens gebruikt om orderacceptatie- en ordervrijgaveregels te ontwikkelen die eenvoudig te begrijpen en daarom makkelijk in praktijksituaties te implementeren zijn. Deze relatief eenvoudige beslissingsregels kunnen al leiden tot significante verbeteringen in de logistieke prestaties van MTO-bedrijven. The thesis of Remco Germs analyses and develops order acceptance and order release policies to control queues in make-to-order (MTO) production systems. Controlling the time orders spend waiting in queues is crucial for realizing short and reliable delivery times, two performance measures which are of strategic importance for many MTO com-panies. Order acceptance and order release are the two most important production con-trol mechanisms to influence the length of these queues. Their performance depends on typical characteristics of MTO systems, such as random (batch) order arrival, routing variability, fixed capacities, setup times and (strict) due-dates. To better understand the underlying mechanisms of good order acceptance and order release policies the models in this thesis focus on the main characteristics of MTO systems. The insights obtained from these models are then used to develop order acceptance and order release policies that are easy to understand and thereby easy to implement in practice. The results show that these relatively simple policies may already lead to significant performance improvements for MTO companies.

Generating Dispatching Rules for the Interrupting Swap-Allowed Blocking Job Shop Problem Using Graph Neural Network and Reinforcement Learning

The interrupting swap-allowed blocking job shop problem (ISBJSSP) is a complex scheduling problem that is able to model many manufacturing planning and logistics applications realistically by addressing both the lack of storage capacity and unforeseen production interruptions. Subjected to random disruptions due to machine malfunction or maintenance, industry production settings often choose to adopt dispatching rules to enable adaptive, real-time re-scheduling, rather than traditional methods that require costly re-computation on the new configuration every time the problem condition changes dynamically. To generate dispatching rules for the ISBJSSP problem, we introduce a dynamic disjunctive graph formulation characterized by nodes and edges subjected to continuous deletions and additions. This formulation enables the training of an adaptive scheduler utilizing graph neural networks and reinforcement learning. Furthermore, a simulator is developed to simulate interruption, swapping, and blocking in the ISBJSSP setting. Employing a set of reported benchmark instances, we conduct a detailed experimental study on ISBJSSP instances with a range of machine shutdown probabilities to show that the scheduling policies generated can outperform or are at least as competitive as existing dispatching rules with predetermined priority. This study shows that the ISBJSSP, which requires real-time adaptive solutions, can be scheduled efficiently with the proposed method when production interruptions occur with random machine shutdowns.Comment: 14 pages, 10 figures. Supplementary Material not include

Self-Evaluation Applied Mathematics 2003-2008 University of Twente

This report contains the self-study for the research assessment of the Department of Applied Mathematics (AM) of the Faculty of Electrical Engineering, Mathematics and Computer Science (EEMCS) at the University of Twente (UT). The report provides the information for the Research Assessment Committee for Applied Mathematics, dealing with mathematical sciences at the three universities of technology in the Netherlands. It describes the state of affairs pertaining to the period 1 January 2003 to 31 December 2008

Deep Reinforcement Learning Techniques For Solving Hybrid Flow Shop Scheduling Problems: Proximal Policy Optimization (PPO) and Asynchronous Advantage Actor-Critic (A3C)

Well-studied scheduling practices are fundamental for the successful support of core business processes in any manufacturing environment. Particularly, the Hybrid Flow Shop (HFS) scheduling problems are present in many manufacturing environments. The current advances in the field of Deep Reinforcement Learning (DRL) attracted the attention of both practitioners and academics to investigate their adoption beyond synthetic game-like applications. Therefore, we present an approach that is based on DRL techniques in conjunction with a discrete event simulation model to solve a real-world four-stage HFS scheduling problem. The main narrative behind the presented concepts is to expose a DRL agent to a game-like environment using an indirect encoding. Two types of DRL techniques namely, Proximal Policy Optimization (PPO) and Asynchronous Advantage Actor-Critic (A3C), are evaluated for solving problems of different complexity. The computational results suggest that the DRL agents successfully learn appropriate policies for solving the investigated problem. In addition, the investigation shows that the agent can adjust their policies when we expose them to a different problem. We further evaluate the approach to solving problem instances published in the literature to establish a comparison