Adaptive Railway Traffic Control using Approximate Dynamic Programming

Railway networks around the world have become challenging to operate in recent decades, with a mixture of track layouts running several different classes of trains with varying operational speeds. This complexity has come about as a result of the sustained increase in passenger numbers where in many countries railways are now more popular than ever before as means of commuting to cities. To address operational challenges, governments and railway undertakings are encouraging development of intelligent and digital transport systems to regulate and optimise train operations in real-time to increase capacity and customer satisfaction by improved usage of existing railway infrastructure. Accordingly, this thesis presents an adaptive railway traffic control system for realtime operations based on a data-based approximate dynamic programming (ADP) approach with integrated reinforcement learning (RL). By assessing requirements and opportunities, the controller aims to reduce delays resulting from trains that entered a control area behind schedule by re-scheduling control plans in real-time at critical locations in a timely manner. The present data-based approach depends on an approximation to the value function of dynamic programming after optimisation from a specified state, which is estimated dynamically from operational experience using RL techniques. By using this approximation, ADP avoids extensive explicit evaluation of performance and so reduces the computational burden substantially. In this thesis, formulations of the approximation function and variants of the RL learning techniques used to estimate it are explored. Evaluation of this controller shows considerable improvements in delays by comparison with current industry practices

A literature review of Artificial Intelligence applications in railway systems

Nowadays it is widely accepted that Artificial Intelligence (AI) is significantly influencing a large number of domains, including railways. In this paper, we present a systematic literature review of the current state-of-the-art of AI in railway transport. In particular, we analysed and discussed papers from a holistic railway perspective, covering sub-domains such as maintenance and inspection, planning and management, safety and security, autonomous driving and control, revenue management, transport policy, and passenger mobility. This review makes an initial step towards shaping the role of AI in future railways and provides a summary of the current focuses of AI research connected to rail transport. We reviewed about 139 scientific papers covering the period from 2010 to December 2020. We found that the major research efforts have been put in AI for rail maintenance and inspection, while very limited or no research has been found on AI for rail transport policy and revenue management. The remaining sub-domains received mild to moderate attention. AI applications are promising and tend to act as a game-changer in tackling multiple railway challenges. However, at the moment, AI research in railways is still mostly at its early stages. Future research can be expected towards developing advanced combined AI applications (e.g. with optimization), using AI in decision making, dealing with uncertainty and tackling newly rising cybersecurity challenges

Control Hierarchies for Critical Infrastructures in Smart Grid Using Reinforcement Learning and Metaheuristic Optimization

The objective of this work is to develop robust control framework for interdependent smart grid infrastructures comprising two critical infrastructures: 1) power distribution networks that are characterized by high penetration of distributed energy resources (DERs), and 2) DC-rail transportation systems in congested urban areas. The rising integration of DERs into the power grid is causing a paradigm shift in the power distribution network. Consequently, new control challenges for efficient and robust operation of the power grid have surfaced. For instance, the intermittency of renewable energy resources necessitates coordinated control of power flows, voltage regulators, and protection device settings of the online resources in the system. This can be achieved with the help of an active distribution network, equipped with a distribution management system that provides online solutions to control problems, in real-time, by having full or partial observability. Besides, the increasing electrification of other critical infrastructures, such as transportation and communication, necessitates controllers that can accommodate the over-arching control requirements of interdependent critical infrastructures. Present control approaches lack the amalgamation of active distribution management and the flexibility to accommodate other critical infrastructures. In this work two levels of control hierarchies, viz. 1) Primary Controller and 2) Secondary Controller, have been designed for the power distribution system. These controllers can provide active distribution management, which can be expanded for seamless integration of other critical infrastructures. Besides, a real-time simulation-in-the-loop testbed has been developed, so that both transient and steady state performance of the controllers can be evaluated simultaneously. This testbed has been developed using OPAL-RT and DSpace. The effectiveness of these controllers have been tested in three types of active distribution networks: 1) A modified IEEE 5-bus system equipped with a grid-connected microgrid that consists of two DERs, 2) A modified IEEE 13-bus system equipped with an islanded community microgrid (C_-Grid) comprising four DERs, and 3) A modified IEEE-30 bus system comprising five grid-connected distributed generations (DGs). The DERs used for this work are battery energy storage systems and photovoltaic systems. The Primary Controller has been designed for regulating voltage, frequency and current in the system, while maintaining stability of these parameters, in both grid-tied and islanded operating modes. These design approaches consider multiple points of coupling among the DERs, which is lacking in the existing literature that is primarily focused on single point of common coupling. Besides, this work shows a method of incorporating communication latency, which may exist between Primary and Secondary Controller, into the control design. This facilitates performance analysis of the primary controller, when it is subjected to communication latency, and accordingly develop mitigation techniques. The Secondary Controller has been designed using a reinforcement learning technique called Adaptive Critic Design (ACD). ACD can facilitate seamless integration of a power distribution network with other critical infrastructures. The ACD based algorithm functions as a distribution management system where its control objectives are to balance load and generation, to take preventive or corrective measures for mitigating failures and improving system resiliency, to minimize the cost of energy incurred by the loads by dispatching the DERs, and to maintain their state of health. Alongside DERs, the impact of DC-rail transportation on the power distribution network has been investigated here, with an objective of efficiently and economically reducing congestion in power substations. Hybrid energy storage systems, comprising battery, supercapacitor and flywheel, have been used as wayside energy storage technologies for this purpose. These storage technologies reduce congestion by supplying energy during acceleration and coasting of the trains, and replenish their energy by recapturing the regenerative braking energy during deceleration of the same. The trains consume/regenerate energy at a very high rate during acceleration/deceleration, thereby requiring storage technologies with both high energy and power densities. Hence, the three aforementioned storage technologies have been investigated for both standalone and hybrid operations, by considering system performance and resiliency alongside the percentage of energy recovered, without comprising the cost-recuperation over time. This has been achieved using a two-stage method, where the first stage comprises the development of detailed mathematical model of the rail system and the storage technologies. This mathematical model has been optimized using Genetic Algorithm, in order to obtain optimal combinations of type and size of the storage technologies for minimum cost, within the system constraints. In the second stage, a detailed simulation model has been developed by capturing all the dynamics of the transportation network, which could not be entirely represented in the mathematical model. The optimal sizes obtained from the first stage have been used in the second stage to evaluate their performance and accordingly adjust their values. Thus, the mathematical model provides initial values in a large search space, and these values are further tuned based on the results from simulation model

Deep reinforcement learning for multi-modal embodied navigation

Ce travail se concentre sur une tâche de micro-navigation en plein air où le but est de naviguer vers une adresse de rue spécifiée en utilisant plusieurs modalités (par exemple, images, texte de scène et GPS). La tâche de micro-navigation extérieure s’avère etre un défi important pour de nombreuses personnes malvoyantes, ce que nous démontrons à travers des entretiens et des études de marché, et nous limitons notre définition des problèmes à leurs besoins. Nous expérimentons d’abord avec un monde en grille partiellement observable (Grid-Street et Grid City) contenant des maisons, des numéros de rue et des régions navigables. Ensuite, nous introduisons le Environnement de Trottoir pour la Navigation Visuelle (ETNV), qui contient des images panoramiques avec des boîtes englobantes pour les numéros de maison, les portes et les panneaux de nom de rue, et des formulations pour plusieurs tâches de navigation. Dans SEVN, nous formons un modèle de politique pour fusionner des observations multimodales sous la forme d’images à résolution variable, de texte visible et de données GPS simulées afin de naviguer vers une porte d’objectif. Nous entraînons ce modèle en utilisant l’algorithme d’apprentissage par renforcement, Proximal Policy Optimization (PPO). Nous espérons que cette thèse fournira une base pour d’autres recherches sur la création d’agents pouvant aider les membres de la communauté des gens malvoyantes à naviguer le monde.This work focuses on an Outdoor Micro-Navigation (OMN) task in which the goal is to navigate to a specified street address using multiple modalities including images, scene-text, and GPS. This task is a significant challenge to many Blind and Visually Impaired (BVI) people, which we demonstrate through interviews and market research. To investigate the feasibility of solving this task with Deep Reinforcement Learning (DRL), we first introduce two partially observable grid-worlds, Grid-Street and Grid City, containing houses, street numbers, and navigable regions. In these environments, we train an agent to find specific houses using local observations under a variety of training procedures. We parameterize our agent with a neural network and train using reinforcement learning methods. Next, we introduce the Sidewalk Environment for Visual Navigation (SEVN), which contains panoramic images with labels for house numbers, doors, and street name signs, and formulations for several navigation tasks. In SEVN, we train another neural network model using Proximal Policy Optimization (PPO) to fuse multi-modal observations in the form of variable resolution images, visible text, and simulated GPS data, and to use this representation to navigate to goal doors. Our best model used all available modalities and was able to navigate to over 100 goals with an 85% success rate. We found that models with access to only a subset of these modalities performed significantly worse, supporting the need for a multi-modal approach to the OMN task. We hope that this thesis provides a foundation for further research into the creation of agents to assist members of the BVI community to safely navigate

The Machine as Art/ The Machine as Artist

The articles collected in this volume from the two companion Arts Special Issues, “The Machine as Art (in the 20th Century)” and “The Machine as Artist (in the 21st Century)”, represent a unique scholarly resource: analyses by artists, scientists, and engineers, as well as art historians, covering not only the current (and astounding) rapprochement between art and technology but also the vital post-World War II period that has led up to it; this collection is also distinguished by several of the contributors being prominent individuals within their own fields, or as artists who have actually participated in the still unfolding events with which it is concerne

Facing Forward: Art & Theory from a Future Perspective

Contributors to this volume include participants in the Facing Forward Project of 2011-12, which started as a collaboration between the Stedelijk Museum Amsterdam, the Amsterdam School for Cultural Analysis at the University of Amsterdam, ..