Adaptive Order Dispatching based on Reinforcement Learning: Application in a Complex Job Shop in the Semiconductor Industry

Heutige Produktionssysteme tendieren durch die Marktanforderungen getrieben zu immer kleineren Losgrößen, höherer Produktvielfalt und größerer Komplexität der Materialflusssysteme. Diese Entwicklungen stellen bestehende Produktionssteuerungsmethoden in Frage. Im Zuge der Digitalisierung bieten datenbasierte Algorithmen des maschinellen Lernens einen alternativen Ansatz zur Optimierung von Produktionsabläufen. Aktuelle Forschungsergebnisse zeigen eine hohe Leistungsfähigkeit von Verfahren des Reinforcement Learning (RL) in einem breiten Anwendungsspektrum. Im Bereich der Produktionssteuerung haben sich jedoch bisher nur wenige Autoren damit befasst. Eine umfassende Untersuchung verschiedener RL-Ansätze sowie eine Anwendung in der Praxis wurden noch nicht durchgeführt. Unter den Aufgaben der Produktionsplanung und -steuerung gewährleistet die Auftragssteuerung (order dispatching) eine hohe Leistungsfähigkeit und Flexibilität der Produktionsabläufe, um eine hohe Kapazitätsauslastung und kurze Durchlaufzeiten zu erreichen. Motiviert durch komplexe Werkstattfertigungssysteme, wie sie in der Halbleiterindustrie zu finden sind, schließt diese Arbeit die Forschungslücke und befasst sich mit der Anwendung von RL für eine adaptive Auftragssteuerung. Die Einbeziehung realer Systemdaten ermöglicht eine genauere Erfassung des Systemverhaltens als statische Heuristiken oder mathematische Optimierungsverfahren. Zusätzlich wird der manuelle Aufwand reduziert, indem auf die Inferenzfähigkeiten des RL zurückgegriffen wird. Die vorgestellte Methodik fokussiert die Modellierung und Implementierung von RL-Agenten als Dispatching-Entscheidungseinheit. Bekannte Herausforderungen der RL-Modellierung in Bezug auf Zustand, Aktion und Belohnungsfunktion werden untersucht. Die Modellierungsalternativen werden auf der Grundlage von zwei realen Produktionsszenarien eines Halbleiterherstellers analysiert. Die Ergebnisse zeigen, dass RL-Agenten adaptive Steuerungsstrategien erlernen können und bestehende regelbasierte Benchmarkheuristiken übertreffen. Die Erweiterung der Zustandsrepräsentation verbessert die Leistung deutlich, wenn ein Zusammenhang mit den Belohnungszielen besteht. Die Belohnung kann so gestaltet werden, dass sie die Optimierung mehrerer Zielgrößen ermöglicht. Schließlich erreichen spezifische RL-Agenten-Konfigurationen nicht nur eine hohe Leistung in einem Szenario, sondern weisen eine Robustheit bei sich ändernden Systemeigenschaften auf. Damit stellt die Forschungsarbeit einen wesentlichen Beitrag in Richtung selbstoptimierender und autonomer Produktionssysteme dar. Produktionsingenieure müssen das Potenzial datenbasierter, lernender Verfahren bewerten, um in Bezug auf Flexibilität wettbewerbsfähig zu bleiben und gleichzeitig den Aufwand für den Entwurf, den Betrieb und die Überwachung von Produktionssteuerungssystemen in einem vernünftigen Gleichgewicht zu halten

Applying the big bang-big crunch metaheuristic to large-sized operational problems

In this study, we present an investigation of comparing the capability of a big bang-big crunch metaheuristic (BBBC) for managing operational problems including combinatorial optimization problems. The BBBC is a product of the evolution theory of the universe in physics and astronomy. Two main phases of BBBC are the big bang and the big crunch. The big bang phase involves the creation of a population of random initial solutions, while in the big crunch phase these solutions are shrunk into one elite solution exhibited by a mass center. This study looks into the BBBC’s effectiveness in assignment and scheduling problems. Where it was enhanced by incorporating an elite pool of diverse and high quality solutions; a simple descent heuristic as a local search method; implicit recombination; Euclidean distance; dynamic population size; and elitism strategies. Those strategies provide a balanced search of diverse and good quality population. The investigation is conducted by comparing the proposed BBBC with similar metaheuristics. The BBBC is tested on three different classes of combinatorial optimization problems; namely, quadratic assignment, bin packing, and job shop scheduling problems. Where the incorporated strategies have a greater impact on the BBBC's performance. Experiments showed that the BBBC maintains a good balance between diversity and quality which produces high-quality solutions, and outperforms other identical metaheuristics (e.g. swarm intelligence and evolutionary algorithms) reported in the literature

Combinatorial optimization and metaheuristics

Today, combinatorial optimization is one of the youngest and most active areas of discrete mathematics. It is a branch of optimization in applied mathematics and computer science, related to operational research, algorithm theory and computational complexity theory. It sits at the intersection of several fields, including artificial intelligence, mathematics and software engineering. Its increasing interest arises for the fact that a large number of scientific and industrial problems can be formulated as abstract combinatorial optimization problems, through graphs and/or (integer) linear programs. Some of these problems have polynomial-time (“efficient”) algorithms, while most of them are NP-hard, i.e. it is not proved that they can be solved in polynomial-time. Mainly, it means that it is not possible to guarantee that an exact solution to the problem can be found and one has to settle for an approximate solution with known performance guarantees. Indeed, the goal of approximate methods is to find “quickly” (reasonable run-times), with “high” probability, provable “good” solutions (low error from the real optimal solution). In the last 20 years, a new kind of algorithm commonly called metaheuristics have emerged in this class, which basically try to combine heuristics in high level frameworks aimed at efficiently and effectively exploring the search space. This report briefly outlines the components, concepts, advantages and disadvantages of different metaheuristic approaches from a conceptual point of view, in order to analyze their similarities and differences. The two very significant forces of intensification and diversification, that mainly determine the behavior of a metaheuristic, will be pointed out. The report concludes by exploring the importance of hybridization and integration methods

A self-adaptive multimeme memetic algorithm co-evolving utility scores to control genetic operators and their parameter settings

Memetic algorithms are a class of well-studied metaheuristics which combine evolutionary algorithms and local search techniques. A meme represents contagious piece of information in an adaptive information sharing system. The canonical memetic algorithm uses a fixed meme, denoting a hill climbing operator, to improve each solution in a population during the evolutionary search process. Given global parameters and multiple parametrised operators, adaptation often becomes a crucial constituent in the design of MAs. In this study, a self-adaptive self-configuring steady-state multimeme memetic algorithm (SSMMA) variant is proposed. Along with the individuals (solutions), SSMMA co-evolves memes, encoding the utility score for each algorithmic component choice and relevant parameter setting option. An individual uses tournament selection to decide which operator and parameter setting to employ at a given step. The performance of the proposed algorithm is evaluated on six combinatorial optimisation problems from a cross-domain heuristic search benchmark. The results indicate the success of SSMMA when compared to the static Mas as well as widely used self-adaptive Multimeme Memetic Algorithm from the scientific literature

Reinforcement Learning-assisted Evolutionary Algorithm: A Survey and Research Opportunities

Evolutionary algorithms (EA), a class of stochastic search methods based on the principles of natural evolution, have received widespread acclaim for their exceptional performance in various real-world optimization problems. While researchers worldwide have proposed a wide variety of EAs, certain limitations remain, such as slow convergence speed and poor generalization capabilities. Consequently, numerous scholars actively explore improvements to algorithmic structures, operators, search patterns, etc., to enhance their optimization performance. Reinforcement learning (RL) integrated as a component in the EA framework has demonstrated superior performance in recent years. This paper presents a comprehensive survey on integrating reinforcement learning into the evolutionary algorithm, referred to as reinforcement learning-assisted evolutionary algorithm (RL-EA). We begin with the conceptual outlines of reinforcement learning and the evolutionary algorithm. We then provide a taxonomy of RL-EA. Subsequently, we discuss the RL-EA integration method, the RL-assisted strategy adopted by RL-EA, and its applications according to the existing literature. The RL-assisted procedure is divided according to the implemented functions including solution generation, learnable objective function, algorithm/operator/sub-population selection, parameter adaptation, and other strategies. Finally, we analyze potential directions for future research. This survey serves as a rich resource for researchers interested in RL-EA as it overviews the current state-of-the-art and highlights the associated challenges. By leveraging this survey, readers can swiftly gain insights into RL-EA to develop efficient algorithms, thereby fostering further advancements in this emerging field.Comment: 26 pages, 16 figure

Automatic Design of Dispatching Rules for Job Shop Scheduling with Genetic Programming

Scheduling is an important planning activity in manufacturing systems to help optimise the usage of scarce resources and improve the customer satisfaction. In the job shop manufacturing environment, scheduling problems are challenging due to the complexity of production flows and practical requirements such as dynamic changes, uncertainty, multiple objectives, and multiple scheduling decisions. Also, job shop scheduling (JSS) is very common in small manufacturing businesses and JSS is considered one of the most popular research topics in this domain due to its potential to dramatically decrease the costs and increase the throughput. Practitioners and researchers have applied different computational techniques, from different fields such as operations research and computer science, to deal with JSS problems. Although optimisation methods usually show their dominance in the literature, applying optimisation techniques in practical situations is not straightforward because of the practical constraints and conditions in the shop. Dispatching rules are a very useful approach to dealing with these environments because they are easy to implement(by computers and shop floor operators) and can cope with dynamic changes. However, designing an effective dispatching rule is not a trivial task and requires extensive knowledge about the scheduling problem. The overall goal of this thesis is to develop a genetic programming based hyper-heuristic (GPHH) approach for automatic heuristic design of reusable and competitive dispatching rules in job shop scheduling environments. This thesis focuses on incorporating special features of JSS in the representations and evolutionary search mechanisms of genetic programming(GP) to help enhance the quality of dispatching rules obtained. This thesis shows that representations and evaluation schemes are the important factors that significantly influence the performance of GP for evolving dispatching rules. The thesis demonstrates that evolved rules which are trained to adapt their decisions based on the changes in shops are better than conventional rules. Moreover, by applying a new evaluation scheme, the evolved rules can effectively learn from the mistakes made in previous completed schedules to construct better scheduling decisions. The GP method using the newproposed evaluation scheme shows better performance than the GP method using the conventional scheme. This thesis proposes a new multi-objective GPHH to evolve a Pareto front of non-dominated dispatching rules. Instead of evolving a single rule with assumed preferences over different objectives, the advantage of this GPHH method is to allow GP to evolve rules to handle multiple conflicting objectives simultaneously. The Pareto fronts obtained by the GPHH method can be used as an effective tool to help decision makers select appropriate rules based on their knowledge regarding possible trade-offs. The thesis shows that evolved rules can dominate well-known dispatching rules when a single objective and multiple objectives are considered. Also, the obtained Pareto fronts show that many evolved rules can lead to favourable trade-offs, which have not been explored in the literature. This thesis tackles one of themost challenging issues in job shop scheduling, the interactions between different scheduling decisions. New GPHH methods have been proposed to help evolve scheduling policies containing multiple scheduling rules for multiple scheduling decisions. The two decisions examined in this thesis are sequencing and due date assignment. The experimental results show that the evolved scheduling rules are significantly better than scheduling policies in the literature. A cooperative coevolution approach has also been developed to reduce the complexity of evolving sophisticated scheduling policies. A new evolutionary search mechanisms and customised genetic operations are proposed in this approach to improve the diversity of the obtained Pareto fronts

Iterated local search using an add and delete hyper- heuristic for university course timetabling

Hyper-heuristics are (meta-)heuristics that operate at a higher level to choose or generate a set of low-level (meta-)heuristics in an attempt of solve difficult optimization problems. Iterated local search (ILS) is a well-known approach for discrete optimization, combining perturbation and hill-climbing within an iterative framework. In this study, we introduce an ILS approach, strengthened by a hyper-heuristic which generates heuristics based on a fixed number of add and delete operations. The performance of the proposed hyper-heuristic is tested across two different problem domains using real world benchmark of course timetabling instances from the second International Timetabling Competition Tracks 2 and 3. The results show that mixing add and delete operations within an ILS framework yields an effective hyper-heuristic approach