Temporal Abstraction for Hierarchical Reinforcement Learning in Air Traffic Management

Στην παρούσα διπλωματική εργασία, παρουσιάζουμε την αποδοτικότητα και αποτελεσματικότητα ιεραρχικών πολυπρακτορικών μεθόδων ενισχυτικής μάθησης, όπου οι αποφάσεις διαμορφώνονται σε διάφορα επίπεδα αφαίρεσης, για τη σύνθεση κοινών πολιτικών με σκοπό την επίλυση προβλημάτων συμφόρησης στη διαχείριση εναέριας κυκλοφορίας, επεκτείνοντας τη δουλειά που παρουσιάζεται στο [1]. Συγκεκριμένα, στοχεύουμε στην επίλυση περιπτώσεων όπου η ζήτηση για χρήση του εναέριου χώρου υπερβαίνει την προσφορά (Demand - Capacity problems), μέσω της επιβολής καθυστερήσεων στις πτήσεις πριν το τακτικό επίπεδο των διαδικασιών. Οι πράκτορες, οι οποίοι αντιπροσωπεύουν πτήσεις, έχουν περιορισμένη πληροφόρηση σχετικά με τις αμοιβές και τις προτιμήσεις των υπόλοιπων πρακτόρων και χρειάζεται να συντονιστούν μεταξύ τους για να επιτύχουν τα καθήκοντά τους ενώ τηρούν λειτουργικούς περιορισμούς. Συγκεκριμένα μοντελοποιούμε το πρόβλημα σαν μια ιεραρχική πολυπρακτορική Μαρκοβιανή Διαδικασία Αποφάσεων και παρουσιάζουμε ιεραρχικές πολυπρακτορικές μεθόδους ενισχυτικής μάθησης που επιτρέπουν στους πράκτορες να διομορφώνουν τις δικές τους πολιτικές σε συντονισμό με τους υπόλοιπους. Εξερευνούμε την αποτελεσματικότητα εναλλακτικών ιεραρχικών μεθόδων ενισχυτικής μάθησης και εναλλακτικών σχημάτων μεταφοράς της εμπειρίας μεταξύ επιπέδων με σκοπό τη μείωση των καθυστερήσεων και του αριθμού των πτήσεων στις οποίες έχει επιβληθεί καθυστέρηση. Εκτενής πειραματική μελέτη σε μία πραγματική περίπτωση, δείχνει τη δυνατότητα των προτεινόμενων προσεγγίσεων να βελτιώσουν την υπολογιστική αποτελεσματικότητα και την ποιότητα των λύσεων για την επίλυση προβλημάτων εξισορρόπησης ζήτησης-προσφοράς (Demand - Capacity problems) σε σύγκριση και με άλλα state of the art μοντέλα και μεθόδους [1].In this thesis, we report on the efficiency and effectiveness of hierarchical multiagent reinforcement learning methods (H-MARL), where the decisions are formed at various abstraction levels, for the computation of joint policies to resolve congestion problems in the Air Traffic Management (ATM) domain extending the work presented in [1]. Specifically, we aim to resolve cases where demand of airspace use exceeds capacity (demand-capacity problems), via imposing ground delays to flights at the pre-tactical stage of operations. Agents, representing flights, have limited information about others' payoffs and preferences, and need to coordinate among themselves to achieve their tasks while adhering to operational constraints. Specifically, we formalize the problem as a hierarchical multiagent Markov Decision Process (MA-MDP) and we present hierarchical multiagent reinforcement learning methods that allow agents to form own policies in coordination with others. We explore the effectiveness of alternative hierarchical multiagent reinforcement learning methods and alternative schemes of transferring experience between levels, towards reducing delays and number of delayed flights. Extensive experimental study on a real-world case, shows the potential of the proposed approaches in increasing the computational efficiency and the quality of solutions in resolving the demand-capacity balance problems also in comparison to other state of the art flat models and methods [1]

Book of abstracts of the 24th Euro Working Group on Transportation Meeting

Sem resumo disponível.publishe

Distributed, decentralised and compensational mechanisms for platoon formation

Verkehrsprobleme nehmen mit der weltweiten Urbanisierung und der Zunahme der Anzahl der Fahrzeuge pro Kopf zu. Platoons, eine Formation von eng hintereinander fahrenden Fahrzeugen, stellen sich als mögliche Lösung dar, da bestehende Forschungen darauf hinweisen, dass sie zu einer besseren Straßenauslastung beitragen, den Kraftstoffverbrauch und die Emissionen reduzieren und Engpässe schneller entlasten können. Rund um das Thema Platooning gibt es viele Aspekte zu erforschen: Sicherheit, Stabilität, Kommunikation, Steuerung und Betrieb, die allesamt notwendig sind, um den Einsatz von Platooning im Alltagsverkehr näher zu bringen. Während in allen genannten Bereichen bereits umfangreiche Forschungen durchgeführt wurden, gibt es bisher nur wenige Arbeiten, die sich mit der logischen Gruppierung von Fahrzeugen in Platoons beschäftigen. Daher befasst sich diese Arbeit mit dem noch wenig erforschten Problem der Platoonbildung, wobei sich die vorhandenen Beispiele mit auf Autobahnen fahrenden Lastkraftwagen beschäftigen. Diese Fälle befinden sich auf der strategischen und taktischen Ebene der Planung, da sie von einem großen Zeithorizont profitieren und die Gruppierung entsprechend optimiert werden kann. Die hier vorgestellten Ansätze befinden sich hingegen auf der operativen Ebene, indem Fahrzeuge aufgrund der verteilten und dezentralen Natur dieser Ansätze spontan und organisch gruppiert und gesteuert werden. Dadurch entstehen sogenannte opportunistische Platoons, die aufgrund ihrer Flexibilität eine vielversprechende Voraussetzung für alle Netzwerkarte bieten könnten. Insofern werden in dieser Arbeit zwei neuartige Algorithmen zur Bildung von Platoons vorgestellt: ein verteilter Ansatz, der von klassischen Routing-Problemen abgeleitet wurde, und ein ergänzender dezentraler kompensatorischer Ansatz. Letzteres nutzt automatisierte Verhandlungen, um es den Fahrzeugen zu erleichtern, sich auf der Basis eines monetären Austausches in einem Platoon zu organisieren. In Anbetracht der Tatsache, dass alle Verkehrsteilnehmer über eine Reihe von Präferenzen, Einschränkungen und Zielen verfügen, muss das vorgeschlagene System sicherstellen, dass jede angebotene Lösung für die einzelnen Fahrzeuge akzeptabel und vorteilhaft ist und den möglichen Aufwand, die Kosten und die Opfer überwiegt. Dies wird erreicht, indem den Platooning-Fahrzeugen eine Form von Anreiz geboten wird, im Sinne von entweder Kostensenkung oder Ampelpriorisierung. Um die vorgeschlagenen Algorithmen zu testen, wurde eine Verkehrssimulation unter Verwendung realer Netzwerke mit realistischer Verkehrsnachfrage entwickelt. Die Verkehrsteilnehmer wurden in Agenten umgewandelt und mit der notwendigen Funktionalität ausgestattet, um Platoons zu bilden und innerhalb dieser zu operieren. Die Anwendbarkeit und Eignung beider Ansätze wurde zusammen mit verschiedenen anderen Aspekten untersucht, die den Betrieb von Platoons betreffen, wie Größe, Verkehrszustand, Netzwerkpositionierung und Anreizmethoden. Die Ergebnisse zeigen, dass die vorgeschlagenen Mechanismen die Bildung von spontanen Platoons ermöglichen. Darüber hinaus profitierten die teilnehmenden Fahrzeuge mit dem auf verteilter Optimierung basierenden Ansatz und unter Verwendung kostensenkender Anreize unabhängig von der Platoon-Größe, dem Verkehrszustand und der Positionierung, mit Nutzenverbesserungen von 20% bis über 50% im Vergleich zur untersuchten Baseline. Bei zeitbasierten Anreizen waren die Ergebnisse uneinheitlich, wobei sich der Nutzen einiger Fahrzeuge verbesserte, bei einigen keine Veränderung eintrat und bei anderen eine Verschlechterung zu verzeichnen war. Daher wird die Verwendung solcher Anreize aufgrund ihrer mangelnden Pareto-Effizienz nicht empfohlen. Der kompensatorische und vollständig dezentralisierte Ansatz weißt einige Vorteile auf, aber die daraus resultierende Verbesserung war insgesamt vernachlässigbar. Die vorgestellten Mechanismen stellen einen neuartigen Ansatz zur Bildung von Platoons dar und geben einen aussagekräftigen Einblick in die Mechanik und Anwendbarkeit von Platoons. Dies schafft die Voraussetzungen für zukünftige Erweiterungen in der Planung, Konzeption und Implementierung effektiverer Infrastrukturen und Verkehrssysteme.Traffic problems have been on the rise corresponding with the increase in worldwide urbanisation and the number of vehicles per capita. Platoons, which are a formation of vehicles travelling close together, present themselves as a possible solution, as existing research indicates that they can contribute to better road usage, reduce fuel consumption and emissions and decongest bottlenecks faster. There are many aspects to be explored pertaining to the topic of platooning: safety, stability, communication, controllers and operations, all of which are necessary to bring platoons closer to use in everyday traffic. While extensive research has already made substantial strides in all the aforementioned fields, there is so far little work on the logical grouping of vehicles in platoons. Therefore, this work addresses the platoon formation problem, which has not been heavily researched, with existing examples being focused on large, freight vehicles travelling on highways. These cases find themselves on the strategic and tactical level of planning since they benefit from a large time horizon and the grouping can be optimised accordingly. The approaches presented here, however, are on the operational level, grouping and routing vehicles spontaneously and organically thanks to their distributed and decentralised nature. This creates so-called opportunistic platoons which could provide a promising premise for all networks given their flexibility. To this extent, this thesis presents two novel platoon forming algorithms: a distributed approach derived from classical routing problems, and a supplementary decentralised compensational approach. The latter uses automated negotiation to facilitate vehicles organising themselves in a platoon based on monetary exchanges. Considering that all traffic participants have a set of preferences, limitations and goals, the proposed system must ensure that any solution provided is acceptable and beneficial for the individual vehicles, outweighing any potential effort, cost and sacrifices. This is achieved by offering platooning vehicles some form of incentivisation, either cost reductions or traffic light prioritisation. To test the proposed algorithms, a traffic simulation was developed using real networks with realistic traffic demand. The traffic participants were transformed into agents and given the necessary functionality to build platoons and operate within them. The applicability and suitability of both approaches were investigated along with several other aspects pertaining to platoon operations such as size, traffic state, network positioning and incentivisation methods. The results indicate that the mechanisms proposed allow for spontaneous platoons to be created. Moreover, with the distributed optimisation-based approach and using cost-reducing incentives, participating vehicles benefited regardless of the platoon size, traffic state and positioning, with utility improvements ranging from 20% to over 50% compared to the studied baseline. For time-based incentives the results were mixed, with the utility of some vehicles improving, some seeing no change and for others, deteriorating. Therefore, the usage of such incentives would not be recommended due to their lack of Pareto-efficiency. The compensational and completely decentralised approach shows some benefits, but the resulting improvement was overall negligible. The presented mechanisms are a novel approach to platoon formation and provide meaningful insight into the mechanics and applicability of platoons. This sets the stage for future expansions into planning, designing and implementing more effective infrastructures and traffic systems

Applications

Volume 3 describes how resource-aware machine learning methods and techniques are used to successfully solve real-world problems. The book provides numerous specific application examples: in health and medicine for risk modelling, diagnosis, and treatment selection for diseases in electronics, steel production and milling for quality control during manufacturing processes in traffic, logistics for smart cities and for mobile communications

Applications

Volume 3 describes how resource-aware machine learning methods and techniques are used to successfully solve real-world problems. The book provides numerous specific application examples: in health and medicine for risk modelling, diagnosis, and treatment selection for diseases in electronics, steel production and milling for quality control during manufacturing processes in traffic, logistics for smart cities and for mobile communications

implications to CRM and public policy

Thesis(Doctoral) --KDI School:Ph.D in Public Policy,2017With the advent of the Internet and Mobile Communications, the nature of communication has changed significantly over the past few decades .The promotion of technologies among the common people has been found to be an important element of public policy to reduce the digital divide. The rapid advancement of information technology (IT), automation systems and data communications systems leads to improvement of intelligent transport systems (ITS). ITS covers all branches of transportation and involves all dynamically interacting elements of transportation system, i.e. transport means, infrastructure, drivers and commuters. However, few researches have been carried out in the context of public sectors, especially that involving ITS. The purpose of this study is to investigate the justice dimensions that influence satisfaction and public confidence in the context of ITS and to explore implications to Citizen/Customer Relationship Management (CRM) and public policy. This study investigates the following research questions: i) Do levels of perceived justice (distributive, procedural and interactional) in ITS environment affect levels of satisfaction/dissatisfaction? ii) Do levels of satisfaction form ITS affect levels of public confidence? iii) Do levels of dissatisfaction form ITS affect levels of willingness to complain? iv) Do levels of dissatisfaction form ITS affect levels of complaining behavior? v) Do levels of complaining behavior in ITS environment affect levels of satisfaction with complaint handling when the complaints are resolved based on three dimensions (distributive, procedural and interactional)of justice? vi) Do levels of willingness to complain in ITS environment affect levels of public confidence? vii) Do levels of satisfaction with complaint handling in ITS environment affect levels of public confidence? The findings of this study imply that ITS users are more importantly perceive to equity and equality issues, or distributive justice. The employment of ITS should not be limited to the technical aspects of ITS, but should focus more attention on the subjective domain of justice. The results of this study also have important implications for public complaint handling in terms of increasing public satisfaction with ITS, which is crucial for CRM.Part I: Exploring Satisfaction/Dissatisfaction and Public Confidence in the ITS Environment; Implications to CRM and Public Policy Part II: ComparingSatisfaction/Dissatisfaction and Public Confidence in the ITS Environment in Public and Private Transportation Part III: Implementation Strategy of ITS in Developing CountriesdoctoralpublishedA. K. M. Anisur RAHMAN

Transitioning power distribution grid into nanostructured ecosystem : prosumer-centric sovereignty

PhD ThesisGrowing acceptance for in-house Distributed Energy Resource (DER) installations at lowvoltage level have gained much significance in recent years due to electricity market liberalisations and opportunities in reduced energy billings through personalised utilisation management for targeted business model. In consequence, modelling of passive customers’ electric power system are progressively transitioned into Prosumer-based settings where presidency for Transactive Energy (TE) system framework is favoured. It amplifies Prosumers’ commitments into annexing TE values during market participations and optimised energy management to earn larger rebates and incentives from TE programs. However, when dealing with mass Behind-The-Meter DER administrations, Utility foresee managerial challenges when dealing with distribution network analysis, planning, protection, and power quality security based on Prosumers’ flexibility in optimising their energy needs. This dissertation contributes prepositions into modelling Distributed Energy Resources Management System (DERMS) as an aggregator designed for Prosumer-centered cooperation, interoperating TE control and coordination as key parameters to market for both optimised energy trading and ancillary services in a Community setting. However, Prosumers are primarily driven to create a profitable business model when modelling their DERMS aggregator. Greedy-optimisation exploitations are negative concerns when decisions made resulted in detrimental-uncoordinated outcomes on Demand-Side Response (DSR) and capacity market engagements. This calls for policy decision makers to contract safe (i.e. cooperative yet competitive tendency) business models for Prosumers to maximise TE values while enhancing network’s power quality metrics and reliability performances. Firstly, digitalisation and nanostructuring of distribution network is suggested to identify Prosumer as a sole energy citizen while extending bilateral trading between Prosumer-to- Prosumer (PtP) with the involvements of other grid operators−TE system. Modelling of Nanogrid environment for DER integrations and establishment of local area network infrastructure for IoT security (i.e. personal computing solutions and data protection) are committed for communal engagements in a decentralise setting. Secondly, a multi-layered Distributed Control Framework (DCF) is proposed using Microsoft Azure cloud-edge platform that cascades energy actors into respective layers of TE control and coordination. Furthermore, modelling of flexi-edge computing architecture is proposed, comprising of Contract-Oriented Sensor-based Application Platform (COSAP) employing Multi-Agent System (MAS) to enhance data-sharing privacy and contract coalition agreements during PtP engagements. Lastly, the Agents of MAS are programmed with cooperative yet competitive intelligences attributed to Reinforcement Learning (RL) and Neural Networks (NN) algorithms to solve multimodal socio-economical and uncertainty problems that corresponded to Prosumers’ dynamic energy priorities within the TE framework. To verify the DERMS aggregator operations, three business models were proposed (i.e. greedy-profit margin, collegial-peak demand, reserved-standalone) to analyse comparative technical/physical and economic/social dimensions. Results showed that the proposed TE-valued DERMS aggregator provides participation versatility in the electricity market that enables competitive edginess when utilising Behind-The-Meter DERs in view of Prosumer’s asset scheduling, bidding strategy, and corroborative ancillary services. Performance metrics were evaluated on both domestic and industrial NG environments against IEEE Standard 2030.7-2017 & 2030.8-2018 compliances to ensure deployment practicability. Subsequently, proposed in-house protection system for DER installation serves as an add-on monitoring service which can be incorporated into existing Advance Distribution Management System (ADMS) for Distribution Service Operator (DSO) and field engineers use, ADMS aggregator. It provides early fault detections and isolation processes from allowing fault current to propagate upstream causing cascading power quality issues across the feeder line. In addition, ADMS aggregator also serves as islanding indicator that distinguishes Nanogrid’s islanding state from unintentional or intentional operations. Therefore, a Overcurrent Current Relay (OCR) is proposed using Fuzzy Logic (FL) algorithm to detect, profile, and provide decisional isolation processes using specified OCRs. Moreover, the proposed expert knowledge in FL is programmed to detect fault crises despite insufficient fault current level contributed by DER (i.e. solar PV system) which conventional OCR fails to trigger

Smart grid

Tese de mestrado integrado em Engenharia da Energia e do Ambiente, apresentada à Universidade de Lisboa, através da Faculdade de Ciências, 2016The SG concept arises from the fact that there is an increase in global energy consumption. One of the factors delaying an energetic paradigm change worldwide is the electric grids. Even though there is no specific definition for the SG concept there are several characteristics that describe it. Those features represent several advantages relating to reliability and efficiency. The most important one is the two way flow of energy and information between utilities and consumers. The infrastructures in standard grids and the SG can classified the same way but the second one has several components contributing for monitoring and management improvement. The SG’s management system allows peak reduction, using several techniques underlining many advantages like controlling costs and emissions. Furthermore, it presents a new concept called demand response that allows consumers to play an important role in the electric systems. This factor brings benefits for utilities, consumers and the whole grid but it increases problems in security and that is why the SG relies in a good protection system. There are many schemes and components to create it. The MG can be considered has an electric grid in small scale which can connect to the whole grid. To implement a MG it is necessary economic and technical studies. For that, software like HOMER can be used. However, the economic study can be complex because there are factors that are difficult to evaluate beyond energy selling. On top of that, there are legislation and incentive programs that should be considered. Two case studies prove that MG can be profitable. In the first study, recurring to HOMER, and a scenario with energy selling only, it was obtained a 106% reduction on production cost and 32% in emissions. The installer would have an $8 000 000 profit in the MG’s lifetime. In the second case, it was considered economic services related to peak load reduction, reliability, emission reduction and power quality. The DNO had a profit of$41,386, the MG owner had $29,319 profit and the consumers had a$196,125 profit. We can conclude that the MG with SG concepts can be profitable in many cases