A Memetic Algorithm with Reinforcement Learning for Sociotechnical Production Scheduling

The following interdisciplinary article presents a memetic algorithm with applying deep reinforcement learning (DRL) for solving practically oriented dual resource constrained flexible job shop scheduling problems (DRC-FJSSP). From research projects in industry, we recognize the need to consider flexible machines, flexible human workers, worker capabilities, setup and processing operations, material arrival times, complex job paths with parallel tasks for bill of material (BOM) manufacturing, sequence-dependent setup times and (partially) automated tasks in human-machine-collaboration. In recent years, there has been extensive research on metaheuristics and DRL techniques but focused on simple scheduling environments. However, there are few approaches combining metaheuristics and DRL to generate schedules more reliably and efficiently. In this paper, we first formulate a DRC-FJSSP to map complex industry requirements beyond traditional job shop models. Then we propose a scheduling framework integrating a discrete event simulation (DES) for schedule evaluation, considering parallel computing and multicriteria optimization. Here, a memetic algorithm is enriched with DRL to improve sequencing and assignment decisions. Through numerical experiments with real-world production data, we confirm that the framework generates feasible schedules efficiently and reliably for a balanced optimization of makespan (MS) and total tardiness (TT). Utilizing DRL instead of random metaheuristic operations leads to better results in fewer algorithm iterations and outperforms traditional approaches in such complex environments.Comment: This article has been accepted by IEEE Access on June 30, 202

The 1990 progress report and future plans

This document describes the progress and plans of the Artificial Intelligence Research Branch (RIA) at ARC in 1990. Activities span a range from basic scientific research to engineering development and to fielded NASA applications, particularly those applications that are enabled by basic research carried out at RIA. Work is conducted in-house and through collaborative partners in academia and industry. Our major focus is on a limited number of research themes with a dual commitment to technical excellence and proven applicability to NASA short, medium, and long-term problems. RIA acts as the Agency's lead organization for research aspects of artificial intelligence, working closely with a second research laboratory at JPL and AI applications groups at all NASA centers

Human-Machine Collaborative Optimization via Apprenticeship Scheduling

Coordinating agents to complete a set of tasks with intercoupled temporal and resource constraints is computationally challenging, yet human domain experts can solve these difficult scheduling problems using paradigms learned through years of apprenticeship. A process for manually codifying this domain knowledge within a computational framework is necessary to scale beyond the ``single-expert, single-trainee" apprenticeship model. However, human domain experts often have difficulty describing their decision-making processes, causing the codification of this knowledge to become laborious. We propose a new approach for capturing domain-expert heuristics through a pairwise ranking formulation. Our approach is model-free and does not require enumerating or iterating through a large state space. We empirically demonstrate that this approach accurately learns multifaceted heuristics on a synthetic data set incorporating job-shop scheduling and vehicle routing problems, as well as on two real-world data sets consisting of demonstrations of experts solving a weapon-to-target assignment problem and a hospital resource allocation problem. We also demonstrate that policies learned from human scheduling demonstration via apprenticeship learning can substantially improve the efficiency of a branch-and-bound search for an optimal schedule. We employ this human-machine collaborative optimization technique on a variant of the weapon-to-target assignment problem. We demonstrate that this technique generates solutions substantially superior to those produced by human domain experts at a rate up to 9.5 times faster than an optimization approach and can be applied to optimally solve problems twice as complex as those solved by a human demonstrator.Comment: Portions of this paper were published in the Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI) in 2016 and in the Proceedings of Robotics: Science and Systems (RSS) in 2016. The paper consists of 50 pages with 11 figures and 4 table

Time-Cost Tradeoff and Resource-Scheduling Problems in Construction: A State-of-the-Art Review

Duration, cost, and resources are defined as constraints in projects. Consequently, Construction manager needs to balance between theses constraints to ensure that project objectives are met. Choosing the best alternative of each activity is one of the most significant problems in construction management to minimize project duration, project cost and also satisfies resources constraints as well as smoothing resources. Advanced computer technologies could empower construction engineers and project managers to make right, fast and applicable decisions based on accurate data that can be studied, optimized, and quantified with great accuracy. This article strives to find the recent improvements of resource-scheduling problems and time-cost trade off and the interacting between them which can be used in innovating new approaches in construction management. To achieve this goal, a state-of-the-art review, is conducted as a literature sample including articles implying three areas of research; time-cost trade off, constrained resources and unconstrained resources. A content analysis is made to clarify contributions and gaps of knowledge to help suggesting and specifying opportunities for future research

A Unified Framework for Solving Multiagent Task Assignment Problems

Multiagent task assignment problem descriptors do not fully represent the complex interactions in a multiagent domain, and algorithmic solutions vary widely depending on how the domain is represented. This issue is compounded as related research fields contain descriptors that similarly describe multiagent task assignment problems, including complex domain interactions, but generally do not provide the mechanisms needed to solve the multiagent aspect of task assignment. This research presents a unified approach to representing and solving the multiagent task assignment problem for complex problem domains. Ideas central to multiagent task allocation, project scheduling, constraint satisfaction, and coalition formation are combined to form the basis of the constrained multiagent task scheduling (CMTS) problem. Basic analysis reveals the exponential size of the solution space for a CMTS problem, approximated by O(2n(m+n)) based on the number of agents and tasks involved in a problem. The shape of the solution space is shown to contain numerous discontinuous regions due to the complexities involved in relational constraints defined between agents and tasks. The CMTS descriptor represents a wide range of classical and modern problems, such as job shop scheduling, the traveling salesman problem, vehicle routing, and cooperative multi-object tracking. Problems using the CMTS representation are solvable by a suite of algorithms, with varying degrees of suitability. Solution generating methods range from simple random scheduling to state-of-the-art biologically inspired approaches. Techniques from classical task assignment solvers are extended to handle multiagent task problems where agents can also multitask. Additional ideas are incorporated from constraint satisfaction, project scheduling, evolutionary algorithms, dynamic coalition formation, auctioning, and behavior-based robotics to highlight how different solution generation strategies apply to the complex problem space

Operational Research in Education

Operational Research (OR) techniques have been applied, from the early stages of the discipline, to a wide variety of issues in education. At the government level, these include questions of what resources should be allocated to education as a whole and how these should be divided amongst the individual sectors of education and the institutions within the sectors. Another pertinent issue concerns the efficient operation of institutions, how to measure it, and whether resource allocation can be used to incentivise efficiency savings. Local governments, as well as being concerned with issues of resource allocation, may also need to make decisions regarding, for example, the creation and location of new institutions or closure of existing ones, as well as the day-to-day logistics of getting pupils to schools. Issues of concern for managers within schools and colleges include allocating the budgets, scheduling lessons and the assignment of students to courses. This survey provides an overview of the diverse problems faced by government, managers and consumers of education, and the OR techniques which have typically been applied in an effort to improve operations and provide solutions

Reinforcement Learning-assisted Evolutionary Algorithm: A Survey and Research Opportunities

Evolutionary algorithms (EA), a class of stochastic search methods based on the principles of natural evolution, have received widespread acclaim for their exceptional performance in various real-world optimization problems. While researchers worldwide have proposed a wide variety of EAs, certain limitations remain, such as slow convergence speed and poor generalization capabilities. Consequently, numerous scholars actively explore improvements to algorithmic structures, operators, search patterns, etc., to enhance their optimization performance. Reinforcement learning (RL) integrated as a component in the EA framework has demonstrated superior performance in recent years. This paper presents a comprehensive survey on integrating reinforcement learning into the evolutionary algorithm, referred to as reinforcement learning-assisted evolutionary algorithm (RL-EA). We begin with the conceptual outlines of reinforcement learning and the evolutionary algorithm. We then provide a taxonomy of RL-EA. Subsequently, we discuss the RL-EA integration method, the RL-assisted strategy adopted by RL-EA, and its applications according to the existing literature. The RL-assisted procedure is divided according to the implemented functions including solution generation, learnable objective function, algorithm/operator/sub-population selection, parameter adaptation, and other strategies. Finally, we analyze potential directions for future research. This survey serves as a rich resource for researchers interested in RL-EA as it overviews the current state-of-the-art and highlights the associated challenges. By leveraging this survey, readers can swiftly gain insights into RL-EA to develop efficient algorithms, thereby fostering further advancements in this emerging field.Comment: 26 pages, 16 figure

Using Deep Neural Networks for Scheduling Resource-Constrained Activity Sequences

Eines der bekanntesten Planungsprobleme stellt die Planung von Aktivitäten unter Berücksichtigung von Reihenfolgenbeziehungen zwischen diesen Aktivitäten sowie Ressourcenbeschränkungen dar. In der Literatur ist dieses Planungsproblem als das ressourcenbeschränkte Projektplanungsproblem bekannt und wird im Englischen als Resource-Constrained Project Scheduling Problem oder kurz RCPSP bezeichnet. Das Ziel dieses Problems besteht darin, die Bearbeitungszeit einer Aktivitätsfolge zu minimieren, indem festgelegt wird, wann jede einzelne Aktivität beginnen soll, ohne dass die Ressourcenbeschränkungen überschritten werden. Wenn die Bearbeitungsdauern der Aktivitäten bekannt und deterministisch sind, können die Startzeiten der Aktivitäten à priori definiert werden, ohne dass die Gefahr besteht, dass der Zeitplan unausführbar wird. Da jedoch die Bearbeitungsdauern der Aktivitäten häufig nicht deterministisch sind, sondern auf Schätzungen von Expertengruppen oder historischen Daten basieren, können die realen Bearbeitungsdauern von den geschätzten abweichen. In diesem Fall ist eine reaktive Planungsstrategie zu bevorzugen. Solch eine reaktive Strategie legt die Startzeiten der einzelnen Aktivitäten nicht zu Beginn des Projektes fest, sondern erst unmittelbar an jedem Entscheidungspunkt im Projekt, also zu Beginn des Projektes und immer dann wenn eine oder mehrere Aktivitäten abgeschlossen und die beanspruchten Ressourcen frei werden. In dieser Arbeit wird eine neue reaktive Planungsstrategie für das ressourcenbeschränkte Projektplanungsproblem vorgestellt. Im Gegensatz zu anderen Literaturbeiträgen, in denen exakte, heuristische und meta-heuristische Methoden zur Anwendung kommen, basiert der in dieser Arbeit aufgestellte Lösungsansatz auf künstlichen neuronalen Netzen und maschinellem Lernen. Die neuronalen Netze verarbeiten die Informationen, die den aktuellen Zustand der Aktivitätsfolge beschreiben, und erzeugen daraus Prioritätswerte für die Aktivitäten, die im aktuellen Entscheidungspunkt gestartet werden können. Das maschinelle Lernen und insbesondere das überwachte Lernen werden für das Trainieren der neuronalen Netze mit beispielhaften Trainingsdaten angewendet, wobei die Trainingsdaten mit Hilfe einer Simulation erzeugt wurden. Sechs verschiedene neuronale Netzwerkstrukturen werden in dieser Arbeit betrachtet. Diese Strukturen unterscheiden sich sowohl in der ihnen zur Verfügung gestellten Eingabeinformation als auch der Art des neuronalen Netzes, das diese Information verarbeitet. Es werden drei Arten von neuronalen Netzen betrachtet. Diese sind neuronale Netze mit vollständig verbundenen Schichten, 1- dimensionale faltende neuronale Netze und 2-dimensionale neuronale faltende Netze. Darüber hinaus werden innerhalb jeder einzelnen Netzwerkstruktur verschiedene Hyperparameter, z.B. die Lernrate, Anzahl der Lernepochen, Anzahl an Schichten und Anzahl an Neuronen per Schicht, mittels einer Bayesischen Optimierung abgestimmt. Während des Abstimmens der Hyperparameter wurden außerdem Bereiche für die Hyperparameter identifiziert, die zur Verbesserung der Leistungen genutzt werden sollten. Das am besten trainierte Netzwerk wird dann für den Vergleich mit anderen vierunddreißig reaktiven heuristischen Methoden herangezogen. Die Ergebnisse dieses Vergleichs zeigen, dass der in dieser Arbeit vorgeschlagene Ansatz in Bezug auf die Minimierung der Gesamtdauer der Aktivitätsfolge die meisten Heuristiken übertrifft. Lediglich 3 Heuristiken erzielen kürzere Gesamtdauern als der Ansatz dieser Arbeit, jedoch sind deren Rechenzeiten um viele Größenordnungen länger. Eine Annahme in dieser Arbeit besteht darin, dass während der Ausführung der Aktivitäten Abweichungen bei den Aktivitätsdauern auftreten können, obwohl die Aktivitätsdauern generell als deterministisch modelliert werden. Folglich wird eine Sensitivitätsanalyse durchgeführt, um zu prüfen, ob die vorgeschlagene reaktive Planungsstrategie auch dann kompetitiv bleibt, wenn die Aktivitätsdauern von den angenommenen Werten abweichen