Reinforcement learning for an intelligent and autonomous production control of complex job-shops under time constraints

Reinforcement learning (RL) offers promising opportunities to handle the ever-increasing complexity in managing modern production systems. We apply a Q-learning algorithm in combination with a process-based discrete-event simulation in order to train a self-learning, intelligent, and autonomous agent for the decision problem of order dispatching in a complex job shop with strict time constraints. For the first time, we combine RL in production control with strict time constraints. The simulation represents the characteristics of complex job shops typically found in semiconductor manufacturing. A real-world use case from a wafer fab is addressed with a developed and implemented framework. The performance of an RL approach and benchmark heuristics are compared. It is shown that RL can be successfully applied to manage order dispatching in a complex environment including time constraints. An RL-agent with a gain function rewarding the selection of the least critical order with respect to time-constraints beats heuristic rules strictly by picking the most critical lot first. Hence, this work demonstrates that a self-learning agent can successfully manage time constraints with the agent performing better than the traditional benchmark, a time-constraint heuristic combining due date deviations and a classical first-in-first-out approach

Multi-variate time-series for time constraint adherence prediction in complex job shops

One of the most complex and agile production environments is semiconductor manufacturing, especially wafer fabrication, as products require more than several hundred operations and remain in Work-In-Progress for months leading to complex job shops. Additionally, an increasingly competitive market environment, i.e. owing to Moore’s law, forces semiconductor companies to focus on operational excellence, resiliency and, hence, leads to product quality as a decisive factor. Product-specific time constraints comprising two or more, not necessarily consecutive, operations ensure product quality at an operational level and, thus, are an industry-specific challenge. Time constraint adherence is of utmost importance, since violations typically lead to scrapping entire lots and a deteriorating yield. Dispatching decisions that determine time constraint adherence are as a state of the art performed manually, which is stressful and error-prone. Therefore, this article presents a data-driven approach combining multi-variate time-series with centralized information to predict time constraint adherence probability in wafer fabrication to facilitate dispatching. Real-world data is analyzed and different statistical and machine learning models are evaluated

Improving warehouse responsiveness by job priority management: a European distribution centre field study

Warehouses employ order cut-off times to ensure sufficient time for fulfilment. To satisfy increasing consumer’s expectations for higher order responsiveness, warehouses competitively postpone these cut-off times upholding the same pick-up time. This paper, therefore, aims to schedule jobs more efficiently to meet compressed response times. Secondly, this paper provides a data-driven decision-making methodology to guarantee the right implementation by the practitioners. Priority-based job scheduling using flow-shop models has been used mainly for manufacturing systems but can be ingeniously applied for warehouse job scheduling to accommodate tighter cut-off times. To assist warehouse managers in decision making for the practical value of these models, this study presents a computer simulation approach to decide which priority rule performs best under which circumstances. The application of stochastic simulation models for uncertain real-life operational environments contributes to the previous literature on deterministic models for theoretical environments. The performance of each rule is evaluated in terms of a joint cost criterion that integrates the objectives of low earliness, low tardiness, low labour idleness, and low work-in-process stocks. The simulation outcomes provide several findings about the strategic views for improving responsiveness. In particular, the critical ratio rule using the real-time queue status of jobs has the fastest flow-time and performs best for warehouse scenarios with expensive products and high labour costs. The case study limits the coverage of the findings, but it still closes the existent gap regarding data-driven decision-making methodology for practitioners of supply chains

Design, Implementation and Evaluation of Reinforcement Learning for an Adaptive Order Dispatching in Job Shop Manufacturing Systems

Modern production systems tend to have smaller batch sizes, a larger product variety and more complex material flow systems. Since a human oftentimes can no longer act in a sufficient manner as a decision maker under these circumstances, the demand for efficient and adaptive control systems is rising. This paper introduces a methodical approach as well as guideline for the design, implementation and evaluation of Reinforcement Learning (RL) algorithms for an adaptive order dispatching. Thereby, it addresses production engineers willing to apply RL. Moreover, a real-world use case shows the successful application of the method and remarkable results supporting real-time decision-making. These findings comprehensively illustrate and extend the knowledge on RL

Deep reinforcement learning für workload balance und Fälligkeitskontrolle in wafer fabs

Semiconductor wafer fabrication facilities (wafer fabs) often prioritize two operational objectives: work-in-process (WIP) and due date. WIP-oriented and due date-oriented dispatching rules are two commonly used methods to achieve workload balance and on-time delivery, respectively. However, it often requires sophisticated heuristics to achieve both objectives simultaneously. In this paper, we propose a novel approach using deep-Q-network reinforcement learning (DRL) for dispatching in wafer fabs. The DRL approach differs from traditional dispatching methods by using dispatch agents at work-centers to observe state changes in the wafer fabs. The agents train their deep-Q-networks by taking the states as inputs, allowing them to select the most appropriate dispatch action. Additionally, the reward function is integrated with workload and due date information on both local and global levels. Compared to the traditional WIP and due date-oriented rules, as well as heuristics-based rule in literature, the DRL approach is able to produce better global performance with regard to workload balance and on-time delivery

Serial-batch scheduling – the special case of laser-cutting machines

The dissertation deals with a problem in the field of short-term production planning, namely the scheduling of laser-cutting machines. The object of decision is the grouping of production orders (batching) and the sequencing of these order groups on one or more machines (scheduling). This problem is also known in the literature as "batch scheduling problem" and belongs to the class of combinatorial optimization problems due to the interdependencies between the batching and the scheduling decisions. The concepts and methods used are mainly from production planning, operations research and machine learning

A machine learning enhanced multi-start heuristic to efficiently solve a serial-batch scheduling problem

Serial-batch scheduling problems are widespread in several industries (e.g., the metal processing industry or industrial 3D printing) and consist of two subproblems that must be solved simultaneously: the grouping of jobs into batches and the sequencing of the created batches. This problem’s NP-hard nature prevents optimally solving large-scale problems; therefore, heuristic solution methods are a common choice to effectively tackle the problem. One of the best-performing heuristics in the literature is the ATCS–BATCS(β) heuristic which has three control parameters. To achieve a good solution quality, most appropriate parameters must be determined a priori or within a multi-start approach. As multi-start approaches performing (full) grid searches on the parameters lack efficiency, we propose a machine learning enhanced grid search. To that, Artificial Neural Networks are used to predict the performance of the heuristic given a specific problem instance and specific heuristic parameters. Based on these predictions, we perform a grid search on a smaller set of most promising heuristic parameters. The comparison to the ATCS–BATCS(β) heuristics shows that our approach reaches a very competitive mean solution quality that is only 2.5% lower and that it is computationally much more efficient: computation times can be reduced by 89.2% on average

Semiconductor Fab Scheduling with Self-Supervised and Reinforcement Learning

Semiconductor manufacturing is a notoriously complex and costly multi-step process involving a long sequence of operations on expensive and quantity-limited equipment. Recent chip shortages and their impacts have highlighted the importance of semiconductors in the global supply chains and how reliant on those our daily lives are. Due to the investment cost, environmental impact, and time scale needed to build new factories, it is difficult to ramp up production when demand spikes. This work introduces a method to successfully learn to schedule a semiconductor manufacturing facility more efficiently using deep reinforcement and self-supervised learning. We propose the first adaptive scheduling approach to handle complex, continuous, stochastic, dynamic, modern semiconductor manufacturing models. Our method outperforms the traditional hierarchical dispatching strategies typically used in semiconductor manufacturing plants, substantially reducing each order's tardiness and time until completion. As a result, our method yields a better allocation of resources in the semiconductor manufacturing process