강화학습을 이용한 공항 임시폐쇄 상황에서의 항공 일정계획 복원

학위논문 (석사) -- 서울대학교 대학원 : 공과대학 산업공학과, 2021. 2. 문일경.An airline scheduler plans flight schedules with efficient resource utilization. However, unpredictable events, such as the temporary closure of an airport, disrupt planned flight schedules. Therefore, recovering disrupted flight schedules is essential for airlines. We propose Q-learning and Double Q-learning algorithms using reinforcement learning approach for the aircraft recovery problem (ARP) in cases of temporary closures of airports. We use two recovery options: delaying departures of flights and swapping aircraft. We present an artificial environment of daily flight schedules and the Markov decision process (MDP) for the ARP. We evaluate the proposed approach on a set of experiments carried out on a real-world case of a Korean domestic airline. Computational experiments show that reinforcement learning algorithms recover disrupted flight schedules effectively, and that our approaches flexibly adapt to various objectives and realistic conditions.항공사는 보유하고 있는 자원을 최대한 효율적으로 사용하여 항공 일정계획을 수립하기 위해 비용과 시간을 많이 소모하게 된다. 하지만 공항 임시폐쇄와 같은 긴급 상황이 발생하면 항공편의 비정상 운항이 발생하게 된다. 따라서 이러한 상황이 발생하였을 때, 피해를 최대한 줄이기 위해 항공 일정계획을 복원하게 된다. 본 연구는 강화학습을 이용하여 공항 임시폐쇄 상황에서 항공 일정계획 복원 문제를 푼다. 본 연구에서는 항공기 복원 방법으로 항공편 지연과 항공기 교체의 두 가지 방법을 채택하였으며, 항공 일정계획 복원 문제에 강화학습을 적용하기 위해서 마르코프 결정 과정과 강화학습 환경을 구축하였다. 본 실험을 위해 대한민국 항공사의 실제 국내선 항공 일정계획을 사용하였다. 강화학습 알고리즘을 사용하여 기존의 연구에 비해 항공 일정계획을 효율적으로 복원하였으며, 여러 현실적인 조건과 다양한 목적함수에 유연하게 적용하였다.Abstract i Contents iv List of Tables v List of Figures vi Chapter 1 Introduction 1 Chapter 2 Literature Review 7 Chapter 3 Problem statement 11 3.1 Characteristics of aircraft, flights, and flight schedule requirements 11 3.2 Definitions of disruptions and recovery options and objectives of the problem 13 3.3 Assumptions 16 3.4 Mathematical formulations 19 Chapter 4 Reinforcement learning for aircraft recovery 24 4.1 Principles of reinforcement learning 24 4.2 Environment 27 4.3 Markov decision process 29 Chapter 5 Reinforcement learning algorithms 33 5.1 Q-learning algorithm 33 5.2 Overestimation bias and Double Q-learning algorithm 36 Chapter 6 Computational experiments 38 6.1 Comparison between reinforcement learning and existing algorithms 39 6.2 Performance of the TLN varying the size of delay arcs 46 6.3 Aircraft recovery for a complex real-world case: a Korean domestic airline 48 6.4 Validation for different objectives 54 6.5 Managerial insights 57 Chapter 7 Conclusions 59 Bibliography 61 국문초록 69Maste

Multi-objective airline schedule recovery

Master'sMASTER OF ENGINEERIN

Optimising airline maintenance scheduling decisions

Airline maintenance scheduling (AMS) studies how plans or schedules are constructed to ensure that a fleet is efficiently maintained and that airline operational demands are met. Additionally, such schedules must take into consideration the different regulations airlines are subject to, while minimising maintenance costs. In this thesis, we study different formulations, solution methods, and modelling considerations, for the AMS and related problems to propose two main contributions. First, we present a new type of multi-objective mixed integer linear programming formulation which challenges traditional time discretisation. Employing the concept of time intervals, we efficiently model the airline maintenance scheduling problem with tail assignment considerations. With a focus on workshop resource allocation and individual aircraft flight operations, and the use of a custom iterative algorithm, we solve large and long-term real-world instances (16000 flights, 529 aircraft, 8 maintenance workshops) in reasonable computational time. Moreover, we provide evidence to suggest, that our framework provides near-optimal solutions, and that inter-airline cooperation is beneficial for workshops. Second, we propose a new hybrid solution procedure to solve the aircraft recovery problem. Here, we study how to re-schedule flights and re-assign aircraft to these, to resume airline operations after an unforeseen disruption. We do so while taking operational restrictions into account. Specifically, restrictions on aircraft, maintenance, crew duty, and passenger delay are accounted for. The flexibility of the approach allows for further operational restrictions to be easily introduced. The hybrid solution procedure involves the combination of column generation with learning-based hyperheuristics. The latter, adaptively selects exact or metaheuristic algorithms to generate columns. The five different algorithms implemented, two of which we developed, were collected and released as a Python package (Torres Sanchez, 2020). Findings suggest that the framework produces fast and insightful recovery solutions

Optimisation intégrée des rotations et des blocs mensuels personnalisés des équipages en transport aérien

Le problème de la construction des horaires d’équipage pour les compagnies aériennes consiste à assigner un groupe d’équipage à un ensemble planifié de segments de vols. Ce problème doit également respecter des règles de travail définies par la convention collective et les autorités du transport aérien. Le problème de la construction des horaires d’équipage a reçu une attention particulière en recherche opérationnelle car après le carburant, le coût des équipages constitue la plus grande dépense des compagnies aériennes. En raison de la grande taille du problème et de la complexité des règles de travail, ce problème est traditionnellement traité en deux étapes qui sont résolues séquentiellement : la construction de rotations et la construction de blocs mensuels. La première construit un ensemble de rotations réalisables à coût minimum afin que chaque vol prévu puisse être réalisé par un équipage. Les rotations réalisables sont celles juxtaposant des vols conformément aux règles de la convention collective entres les employés et la compagnie aérienne. La deuxième étape construit des blocs mensuels pour les membres d’équipage en combinant les rotations trouvées précédemment avec les repos, et d’autres activités. Chaque bloc mensuel doit satisfaire certaines règles définies par le contrat de travail. Les membres de l’équipage sont divisés en deux groupes selon leurs rôles et leurs responsabilités : les personnels du poste de pilotage et les personnels de la cabine des passagers. Les pilotes, les copilotes et les mécaniciens de bord font partie du personnel du poste de pilotage. Le personnel du poste de pilotage est qualifié pour piloter un avion ou une famille d’avions. Le capitaine de cabine et les agents de bord font partie des membres de la cabine des passagers. Par le passé, les chercheurs se sont concentrés sur la réduction des coûts associés au personnel du poste de pilotage car leurs salaires sont plus élevés que ceux des membres de la cabine des passagers. Dans cette thèse, nous nous concentrons uniquement sur le personnel du poste de pilotage. La construction des blocs mensuels varie pour chaque compagnie aérienne. Toutefois, on peut classer les méthodes en deux catégories : la construction des blocs anonymes (bidline) et la construction des blocs personnalisés. Pour les blocs anonymes, les horaires sont construits de manière à couvrir toutes les rotations sans connaître les préférences des employés. Les blocs sont ensuite présentés aux membres d’équipage qui sélectionnent les blocs qu’ils veulent faire. Contrairement aux blocs anonymes, les blocs personnalisés tiennent compte des préférences des membres de l’équipage. La construction de ces blocs se fait selon deux objectifs : le rostering et les blocs personnalisés avec séniorité (preferrential bidding). Le premier maximise la satisfaction globale des membres d’équipage sans considérer la séniorité. Le second priorise la satisfaction des membres ayant le plus d’ancienneté. D’un point de vue historique, la construction des blocs anonymes a été l’approche la plus utilisée par les compagnies aériennes nord-américaines alors que la construction des blocs personnalisés a été plus fréquente en Europe. Cependant, les blocs personnalisés sont aujourd’hui une approche de planification utilisée par de plus de compagnies aériennes nord-américaines car ils sont plus avantageux à la fois pour les membres de l’équipage et les compagnies aériennes. Par le passé, le problème de construction des rotations et le problème de construction des blocs mensuels ont été modélisés indépendamment. Bien que cette approche réduise la complexité du problème, elle ne considère pas les contraintes de construction de blocs mensuels lors de la construction des rotations. Ce faisant, il n’est pas possible de garantir une solution optimale pour tous les membres de l’équipage. Plus récemment, des chercheurs ont commencé à intégrer ces problèmes. Le problème de construction intégrée de rotations et de blocs mensuels anonymes pour les pilotes a été étudié par Saddoune et al. Cependant, au meilleur de nos connaissances, il n’existe pas de littérature sur le problème d’intégration de construction des rotations et des blocs mensuels personnalisés. Le premier objectif de cette thèse est de présenter une revue de la littérature sur le problème de construction des horaires d’équipage en transport aérien. De plus, nous présentons un modèle mathématique et une approche de résolution pour le problème séquentiel de construction des blocs mensuels personnalisés. Au meilleur de notre connaissance, aucun modèle permettant de prendre en compte les préférences des pilotes n’a été introduit dans la littérature. Nous avons également observé que peu de chercheurs comparent leurs méthodes sur les mêmes données. Nous proposons donc un ensemble d’instances ainsi qu’un générateur de préférences qui est disponible en ligne pour des fins de comparaison. Dans le deuxième objectif de cette thèse, nous considérons le problème intégré de construction des rotations et des blocs mensuels personnalisés. Nous proposons un algorithme heuristique qui construit simultanément des horaires mensuels pour les pilotes et copilotes, tout en respectant les préférences personnelles et les contraintes de sécurité. L’algorithme proposé alterne entre les problèmes de construction des horaires des pilotes et des copilotes afin d’obtenir des rotations similaires, même lorsque les blocs mensuels sont différents. De plus, en raison des perturbations qui arrivent souvent durant l’opération, nous nous sommes intéressés à développer un algorithme permettant d’obtenir une solution robuste ; c’est-à-dire que nous minimisons la propagation de la perturbation d’un premier vol aux autres vols et aux autres membres d’équipage. La troisième contribution de cette thèse vise à satisfaire cet aspect. Pour ce faire, nous résolvons le problème de mise à jour des blocs mensuels simultanément pour les pilotes et les copilotes. Nous visons à maintenir les services de vols et les rotations en commun pour les pilotes et les copilotes dans les solutions de mise à jour. Nous proposons ainsi un algorithme heuristique qui alterne entre le problème de mise à jour des horaires mensuels des pilotes et des copilotes. Pour résumer, cette thèse étudie le problème de construction intégrée des blocs mensuels personnalisés pour les membres de l’équipage. Nous nous concentrons à la fois sur la planification et sur la mise à jour des blocs mensuels.----------ABSTRACT : The airline crew scheduling problem assigns a group of crew members to a set of scheduled flights. This scheduling problem should respect also a set of safety regulations and collective conventions. The airline crew scheduling has received special attention in Operations Research because after fuel, the cost of crew members is the second largest cost for airlines. Due to complexity, traditionally researchers divided this problem into two steps which are solved sequentially: crew pairing and crew assignment. The former constructs a set of minimum cost anonymous feasible pairings for covering the scheduled flights while pairing régulations are taken into account. The latter combines the anonymous pairings with vacations, preassigned activities, and rest periods over a planning horizon (usually a month) to form new schedules for crew members while satisfying safety regulations. Crew members are divided into two groups based on their roles and responsibilities: the cockpit crew members and the cabin crew members. Cockpit crew members are composed of the pilot (captain), copilot (first officer), and flight engineer (for large fleets). The cockpit crew members are qualified to fly one or a family of aircraft types. The cabin crew members are the cabin captain and the flight attendants. Because cockpit crew members are paid substantially higher than cabin crew members, most of the literature has focused on cockpit crew members. In this thesis, we also focus on cockpit crew members composed of pilots and copilots. Despite crew pairings problem which always aims at constructing anonymous pairings, there are two general approaches that airlines consider when solving the crew assignment problem: constructing bidline schedules or personalized schedules. Bidline schedules are anonymous schedules for which the crew preferences and needs are not taken into account. After constructing bidline schedules for crew members, the airlines announce them to the crew members and crew members select the bidlines according to seniority order. In contrast to bidline schedules, personalized schedules consider crew member’s preferences and needs for constructing and allocating the schedules. There are two general ways for constructing personalized schedules: rostering and seniority-based. The former favors providing a maximum global satisfaction for crew members and does not take crew members seniority into account. The latter prioritizes satisfaction of more senior crew members to the junior ones. From a historical point of view, bidline scheduling has been the most common approach at North American airlines whereas personalized scheduling has been more common in Europe. However, personalized schedules are now becoming a common scheduling approach at american airlines by offering advantages for both crew members and airlines. Each of the crew pairing problem and crew assignment problem were modeled independently. This traditional sequential approach reduces the complexity of crew scheduling problem but does not guarantee a global optimum solution for crew members because the constraints of monthly schedules are not taken into account when the pairings are being constructed. More recently, researchers have started to study the integration of the crew pairing and crew assignment problems. The problem of integrated bidline scheduling for pilots has been studied by Saddoune et al. However, integrated personalized crew scheduling for pilots and copilots simultaneously has not been the subject of study so far. The first objective of this thesis is to present an extensive review of literature about airline crew scheduling problem. In addition, in the context of sequential scheduling approach, we present a mathematical model and solution approach for personalized pilot assignment problem. To the best of our knowledge, this personalized assignment model that takes into account the pilots preferences has not yet been introduced in the literature. Furthermore, we observed that researchers frequently do not compare their methods on the same data due to the lack of access to common data sets. Therefore, we made all the data sets and crew preference generators available online which will allow other researchers to do so. As the second objective in this thesis, we consider the integrated personalized crew scheduling problem that simultaneously constructs monthly schedules for pilots and copilots while respecting the personal preferences and safety constraints. In addition, we are interested to maintain the robustness of the crew schedules due to the real-life perturbations that arrive while the planned schedules are being operated. At the operational level, the pilots and copilots must have similar pairings when possible to prevent the propagation of delays throughout the schedules. We present a heuristic algorithm that alternates between the pilot and copilot scheduling problems in order to obtain similar pairings even when the monthly schedules are different. In real life, various disruption sources such as weather conditions may result in delaying or canceling the scheduled flights. These delayed or canceled flights will affect the crew schedules. Due to delay propagation, robust crew recovery problem is very significant. As the third contribution of this thesis, we solve the recovery problem simultaneously for pilots and copilots where the planned schedules are constructed using personalized scheduling approach. We aim at keeping the duties and pairings in common during the recovery solution process. This aim is satisfied by considering heuristic algorithm that alternates between pilots and copilots recovery problems. The re-scheduled flights are considered to be given as an input data.To summarize, this thesis studies integrated personalized crew scheduling problem, in both planning and operational level, which simultaneously constructs/recovers monthly schedules for both pilots and copilots

OPTIMIZATION MODELS AND METHODOLOGIES TO SUPPORT EMERGENCY PREPAREDNESS AND POST-DISASTER RESPONSE

This dissertation addresses three important optimization problems arising during the phases of pre-disaster emergency preparedness and post-disaster response in time-dependent, stochastic and dynamic environments. The first problem studied is the building evacuation problem with shared information (BEPSI), which seeks a set of evacuation routes and the assignment of evacuees to these routes with the minimum total evacuation time. The BEPSI incorporates the constraints of shared information in providing on-line instructions to evacuees and ensures that evacuees departing from an intermediate or source location at a mutual point in time receive common instructions. A mixed-integer linear program is formulated for the BEPSI and an exact technique based on Benders decomposition is proposed for its solution. Numerical experiments conducted on a mid-sized real-world example demonstrate the effectiveness of the proposed algorithm. The second problem addressed is the network resilience problem (NRP), involving an indicator of network resilience proposed to quantify the ability of a network to recover from randomly arising disruptions resulting from a disaster event. A stochastic, mixed integer program is proposed for quantifying network resilience and identifying the optimal post-event course of action to take. A solution technique based on concepts of Benders decomposition, column generation and Monte Carlo simulation is proposed. Experiments were conducted to illustrate the resilience concept and procedure for its measurement, and to assess the role of network topology in its magnitude. The last problem addressed is the urban search and rescue team deployment problem (USAR-TDP). The USAR-TDP seeks an optimal deployment of USAR teams to disaster sites, including the order of site visits, with the ultimate goal of maximizing the expected number of saved lives over the search and rescue period. A multistage stochastic program is proposed to capture problem uncertainty and dynamics. The solution technique involves the solution of a sequence of interrelated two-stage stochastic programs with recourse. A column generation-based technique is proposed for the solution of each problem instance arising as the start of each decision epoch over a time horizon. Numerical experiments conducted on an example of the 2010 Haiti earthquake are presented to illustrate the effectiveness of the proposed approach