FLATLAND: A study of Deep Reinforcement Learning methods applied to the vehicle rescheduling problem in a railway environment

In the field of Reinforcement Learning the task is learning how agents should take sequences of actions in an environment in order to maximize a numerical reward signal. This learning process employed in combination with neural networks has given rise to Deep Reinforcement Learning (DRL), that is nowadays applied in many domains, from video games to robotics and self-driving cars. This work investigates possible DRL approaches applied to Flatland, a multi-agent railway simulation where the main task is to plan and reschedule train routes in order to optimize the traffic flow within the network. The tasks introduced in Flatland are based on the Vehicle Rescheduling Problem, for which determining an optimal solution is a NP-complete problem in combinatorial optimization and determining acceptably good solutions using heuristics and deterministic methods is not feasible in realistic railway systems. In particular, we analyze the tasks of navigation of a single agent inside a map, that from a starting position has to reach a target station in the minimum number of time steps and the generalization of this task to a multi-agent setting, with the new issue of conflicts avoidance and resolution between agents. To solve the problem we developed specific observations of the environment, so as to capture the necessary information for the network, trained with Deep Q-Learning and variants, to learn the best action for each agent, that leads to the solution that maximizes the total reward. The positive results obtained on small environments offer ideas for various interpretations and possible future developments, showing that Reinforcement Learning has the potential to solve the problem under a new perspective

Active Object Classification from 3D Range Data with Mobile Robots

This thesis addresses the problem of how to improve the acquisition of 3D range data with a mobile robot for the task of object classification. Establishing the identities of objects in unknown environments is fundamental for robotic systems and helps enable many abilities such as grasping, manipulation, or semantic mapping. Objects are recognised by data obtained from sensor observations, however, data is highly dependent on viewpoint; the variation in position and orientation of the sensor relative to an object can result in large variation in the perception quality. Additionally, cluttered environments present a further challenge because key data may be missing. These issues are not always solved by traditional passive systems where data are collected from a fixed navigation process then fed into a perception pipeline. This thesis considers an active approach to data collection by deciding where is most appropriate to make observations for the perception task. The core contributions of this thesis are a non-myopic planning strategy to collect data efficiently under resource constraints, and supporting viewpoint prediction and evaluation methods for object classification. Our approach to planning uses Monte Carlo methods coupled with a classifier based on non-parametric Bayesian regression. We present a novel anytime and non-myopic planning algorithm, Monte Carlo active perception, that extends Monte Carlo tree search to partially observable environments and the active perception problem. This is combined with a particle-based estimation process and a learned observation likelihood model that uses Gaussian process regression. To support planning, we present 3D point cloud prediction algorithms and utility functions that measure the quality of viewpoints by their discriminatory ability and effectiveness under occlusion. The utility of viewpoints is quantified by information-theoretic metrics, such as mutual information, and an alternative utility function that exploits learned data is developed for special cases. The algorithms in this thesis are demonstrated in a variety of scenarios. We extensively test our online planning and classification methods in simulation as well as with indoor and outdoor datasets. Furthermore, we perform hardware experiments with different mobile platforms equipped with different types of sensors. Most significantly, our hardware experiments with an outdoor robot are to our knowledge the first demonstrations of online active perception in a real outdoor environment. Active perception has broad significance in many applications. This thesis emphasises the advantages of an active approach to object classification and presents its assimilation with a wide range of robotic systems, sensors, and perception algorithms. By demonstration of performance enhancements and diversity, our hope is that the concept of considering perception and planning in an integrated manner will be of benefit in improving current systems that rely on passive data collection

A Survey of Monte Carlo Tree Search Methods

Monte Carlo tree search (MCTS) is a recently proposed search method that combines the precision of tree search with the generality of random sampling. It has received considerable interest due to its spectacular success in the difficult problem of computer Go, but has also proved beneficial in a range of other domains. This paper is a survey of the literature to date, intended to provide a snapshot of the state of the art after the first five years of MCTS research. We outline the core algorithm's derivation, impart some structure on the many variations and enhancements that have been proposed, and summarize the results from the key game and nongame domains to which MCTS methods have been applied. A number of open research questions indicate that the field is ripe for future work

Task-Oriented Active Sensing via Action Entropy Minimization

This work is licensed under a Creative Commons Attribution 4.0 International License.In active sensing, sensing actions are typically chosen to minimize the uncertainty of the state according to some information-theoretic measure such as entropy, conditional entropy, mutual information, etc. This is reasonable for applications where the goal is to obtain information. However, when the information about the state is used to perform a task, minimizing state uncertainty may not lead to sensing actions that provide the information that is most useful to the task. This is because the uncertainty in some subspace of the state space could have more impact on the performance of the task than others, and this dependence can vary at different stages of the task. One way to combine task, uncertainty, and sensing, is to model the problem as a sequential decision making problem under uncertainty. Unfortunately, the solutions to these problems are computationally expensive. This paper presents a new task-oriented active sensing scheme, where the task is taken into account in sensing action selection by choosing sensing actions that minimize the uncertainty in future task-related actions instead of state uncertainty. The proposed method is validated via simulations

One-dimensional lattice gasses with soft interaction

Eukaryotic DNA must undergo several levels of organized compaction in order to be packaged within the spatial confines of the cell nucleus. The first level of this packaging involves the formation of nucleosomes by wrapping DNA around histone-octamers. The arrangement of nucleosomes along the length of the DNA has important influences on the way higher levels of packaging are organized. In addition to this structural role, the positioning of nucleo- somes along the genome –and in relation to one-another– has important implications for the regulation of genes. Tightly-packaged nucleosomes tend to occlude promoter regions from transcription machinery, while looser configurations tend to be up-regulated. At this level, nucleosome positioning can be treated as an effective one-dimensional system. Many factors contribute to the positioning of nucleosomes along the DNA: genetic sequence, active remodellers, and competition for binding sites with other binding proteins and with one-another all play a role. How to disentangle these effects is a central question that will be explored in this work using yeast as a model organism. In the process, however, more general physical questions will arise regarding the kinetics of one-dimensional adsorption/desorption processes. The over-arching goal is to provide a bridge from biophysical, data-driven work to more pure statistical physics; thus the work is comprised mainly of 5 somewhat separate, but related projects. This thesis will begin with an overview of background information and introductory observa- tions in Chapter 1 to provide context. Chapter 2 will then focus on equilibrium properties of nucleosome positioning. Experimental nucleosome data from a dozen different species of yeast will be used to model the pattern of nucleosome formation near a ‘barrier’ –in this case, the strongly positioned +1 nucleoseome nearest (downstream) to the transcription start site. It will be shown that accounting for ‘softness’ in nucleosomes, due to known biophysical effects, allows for a unified model of nucleosome positioning. Since nucleosomes are rela- tively structurally consistent across very different species, this represents a model that is both parsimonious and physically sound. The published work studying the nucleosome po- sitioning patterns of a dozen species of yeast is included and relies on equilibrium statistical mechanics, as well as a Monte Carlo numeric scheme to account for active processes. While histones clearly dominate the landscape of DNA binding positions, important loci ad- mit binding by other proteins such as transcription factors which serve to regulate genetic transcription and influence nucleosomal patterning. In Chapter 3, we consider the interac- tion of small transcription factors which bind specifically to loci on the DNA and shift the positioning of the neighboring nucleosome, with a corresponding domino effect on other nu- cleosomes in the vicinity. Such shifts in nucleosome patterns can create nucleosome-mediated cooperativity between transcription factors, even when separated by intervening nucleosomes. Next, in Chapter 4, we will consider the role of the genetic sequence in nucleosome positioning, an effect which has also been the subject of considerable research. We will refer to this as the energetic ‘landscape’ of the genome and present a new way of inferring this sequence- preference from nucleosome positioning data. We will see that the experimentally observed density patterns in yeast, together with the interaction-energy of neighboring nucleosomes that was derived in Chapter 2, can be used to quantify this sequence preference. This effort, however, is complicated by the lack of specific data characterizing the 2-body correlation between neighboring nucleosomes. For this reason, the ‘amoeba’ optimization algorithm is adapted to fit the available data, as described in Chapter 4. In Chapter 5, the focus will shift to the dynamics of one-dimensional filling. It will be shown that the kinetic process of equilibration through one-dimensional reversible adsorption is qualitatively different, and much faster, when one allows for soft-interaction of neighboring particles. It has long been known that ‘hard rods’ adsorbing randomly in 1 dimension undergo a jamming phenomenon which can only be resolved into densely packed arrays through very slow collective rearrangement processes. Upon introduction of softness to the nucleosome model, however, jamming is circumvented by a new phase we term ‘cramming’; equilibration can then proceed orders of magnitude faster. This will be reviewed with specific application to the problem of nucleosome adsorption which has been of interest recently in light of new experimental work and the attached publication highlights the main findings. Finally, the dynamics of one-dimensional adsorption-desorption with soft-interacting particles are considered in a more general way. With finite neighbor interactions, a rich new set of dynamics emerges, including a curious non-monotonic density trace in time. The theoretical underpinnings of this effect will be provided in a manuscript, accepted for publication, that concludes this text.Eukaryotische DNA muss mehrere Stufen einer organisierten Kompaktifizierung durchlaufen, um in die räumlichen Grenzen eines Zellkerns zu passen. Die erste Stufe dieser Kompaktifi- zierung beinhaltet den Aufbau von Nukleosomen durch die Verbindung von DNA und Histon- Oktameren. Die Anordnung dieser Nukleosome entlang der DNA hat wichtige Einflüsse auf die Organisation höherer Kompaktifizierungsstufen. Zusätzlich zu ihrer strukturellen Funktion hat die Positionierung von Nukleosomen entlang eines Genoms, sowie die Wechselwirkung von Nukleosomen untereinander wichtige Implikationen für die Regulation von Genen. Dicht gepackte Nukleosome neigen dazu, Promotorregionen von der Transkription auszuschließen, während eine lockere Packung von Nukleosomen in der Regel zur Hochregulation der entsprechenden Gene führt. In dieser ersten Stufe kann die Positionierung von Nukleosomen effektiv als ein eindimensio- nales System beschrieben werden. Viele Faktoren tragen zur Positionierung von Nukleosomen entlang einer DNA bei. Hierzu zählen die Nukleotidsequenz der DNA, aktive “Chromatin remodellers”, sowie der Wettbewerb um Bindungsstellen zwischen Nukleosomen untereinan- der und mit anderen Bindungsproteinen. Die Einordung der einzelnen Faktoren ist zentraler Bestandteil dieser Arbeit, wobei Hefe als Modellorganismus dient. Im Verlauf der Arbeit wer- den allgemeine physikalische Fragen hinsichtlich der Kinetik eindimensionaler Adsorptions- und Desorptions-Prozesse aufgeworfen. Das übergreifende Ziel ist daher die Errichtung einer Brücke zwischen Daten getriebener biophysikalischer Forschung und statistischer Physik. Infolgedessen besteht diese Dissertation aus fünf verschiedenen, jedoch untereinander verwandten Projekten. Diese Arbeit beginnt mit einem Überblick über Hintergründe und einführende Beobachtungen in Kapitel 1. In Kapitel 2 liegt der Fokus auf den Gleichgewichtseigenschaften der Nukleosom- Positionierung. Experimentelle Nukleosom-Daten von einem Dutzend verschiedener Hefearten werden verwendet, um die Anordnung von Nukleosomen in der Nähe einer Barriere zu modellieren. Bei der Barriere handelt es sich um das stark positionierte +1 Nukleosom mit geringstem Abstand zur Transkriptionsstartstelle (abwärts). Es wird gezeigt, dass die Berücksichtigung von “Weichheit” der Nukleosomen, aufgrund von bekannten biophysikalischen Effekten, eine einheitliche Modellierung der Nukleosom-Position- ierung ermöglicht. Da die Struktur von Nukleosomen verhältnismäßig konsistent zwischen verschiedenen Arten ist, ist das beschriebene Modell sowohl minimalistisch als auch physika- lisch sinnvoll. Angefügt an das Kapitel ist eine veröffentlichte Arbeit über die Anordnung von Nukleosomen in einem Dutzend verschiedener Hefearten. Diese Verffentlichung basiert auf der statistischen Physik des Gleichgewichts, sowie auf numerischen Monte-Carlo-Methoden zur Berücksichtigung von aktiven Prozessen. Obwohl DNA-Bindungspositionen durch Histone dominiert sind, gibt es wichtige Orte an denen eine Bindung von anderen Proteinen, wie zum Beispiel von Transkriptionsfaktoren, möglich ist. Diese Faktoren dienen zur Regulation der Transkription und beeinflussen die Anordnung der Nukleosome. In Kapitel 3 betrachten wir die Wechselwirkung kleiner Transkriptionsfaktoren, die mit spezifischen Loci binden und die Positionierung der benachbarten Nukleosomen verschieben kann. Ein Dominoeffekt auf andere benachbarte Nukleosomen wird ebenfalls beobachtet. Eine solche Verschiebung in der Anordnung von Nukleosomen induziert eine Kooperativität zwischen Transkriptionsfaktoren, deren Reichweite mehrere Nukleosome umfassen kann. Im Kapitel 4 wird die Rolle der genetischen Sequenz auf die Nukleosom-Positionierung na ̈her betrachtet, die schon vielfach Gegenstand der Forschung war. Hier wird die genetische Sequenz als energetische Landschaft betrachtet und ein neuer Weg zur Bestimmung von Sequenz- Präferenzen aus Daten ber die Nukleosom-Positionierung dargelegt. Es wird gezeigt, dass die experimentell beobachteten Dichteverteilungen in Hefe, kombiniert mit der in Kapitel 2 hergeleiteten Interaktionsenergie zwischen benachbarten Nukleosomen, genutzt werden können, um die Sequenz-Präferenzen zu quantifizieren. Dieser Prozess wird jedoch durch den derzei- tigen Mangel an spezifischen Daten über zwei-Körper-Korrelationen zwischen benachbarten Nukleosomen erschwert. Aus diesem Grund ist der “Amöben” Optimierungsalgorithmus auf die verfügbaren Daten angepasst, wie in Kapitel 4 beschrieben. In Kapitel 5 wird der Fokus in Richtung der Dynamik des eindimensionalen Füllens verschoben. Es wird gezeigt, dass der kinetische Prozess der Gleichgewichtseinstellung durch eindimensionale reversible Adsorption qualitativ anders und sehr viel schneller ist, wenn wei- che Interaktionen zwischen benachbarten Teilchen erlaubt sind. Es ist seit langem bekannt, dass die Adsorption von “hard rods” in einer Dimension ein “jamming” Phänomen verursacht, das nur durch sehr langsame, kollektive Umordnungsprozesse zu organisierten “Arrays” mit hoher Dichte gelöst werden kann. Mit der Einführung von weichen Wechselwirkungen im Nukleosom-Modell wird jamming durch eine neue Phase, die wir als “cramming” bezeichnen, umgangen; der Übergang ins Gleichgewicht erfolgt auf Zeitskalen, die um Größenordnungen kürzer sind. Dieses Prinzip wird hinsichtlich seiner Anwendung auf die Anordnung von Nukleosomen präsentiert. Die wichtigsten Erkenntnisse hierzu sind in der angehängten Publikation beinhaltet. Abschließend wird die Dynamik des eindimensionalen Adsorption- und Desorptions-Problems von weichwechselwirkenden Teilchen in einer allgemeineren Weise betrachtet. Mit endlichen nachbarschaftlichen oder benachbarten Wechselwirkungen entsteht eine reichhaltige Dynamik, einschließlich eines seltsam, nicht-monotonen Dichteverlaufs in der Zeit. Die theoretischen Grundlagen dieses Effekts werden in einem Manuskript, das diesen Text schlussfolgernd abschließt, präsentiert

CRH*: A Deadlock Free Framework for Scalable Prioritised Path Planning in Multi-Robot Systems

Multi-robot system is an ever growing tool which is able to be applied to a wide range of industries to improve productivity and robustness, especially when tasks are distributed in space, time and functionality. Recent works have shown the benefits of multi-robot systems in fields such as warehouse automation, entertainment and agriculture. The work presented in this paper tackles the deadlock problem in multi-robot navigation, in which robots within a common work-space, are caught in situations where they are unable to navigate to their targets, being blocked by one another. This problem can be mitigated by efficient multi-robot path planning. Our work focused around the development of a scalable rescheduling algorithm named Conflict Resolution Heuristic A* (CRH*) for decoupled prioritised planning. Extensive experimental evaluation of CRH* was carried out in discrete event simulations of a fleet of autonomous agricultural robots. The results from these experiments proved that the algorithm was both scalable and deadlock-free. Additionally, novel customisation options were included to test further optimisations in system performance. Continuous Assignment and Dynamic Scoring showed to reduce the make-span of the routing whilst Combinatorial Heuristics showed to reduce the impact of outliers on priority orderings

Elements of Engineering Excellence

The inspiration for this Contract Report (CR) originated in discussions with the director of Marshall Space Flight Center (MSFC) Engineering who asked that we investigate the question: "How do you achieve excellence in aerospace engineering?" Engineering a space system is a complex activity. Avoiding its inherent potential pitfalls and achieving a successful product is a challenge. This CR presents one approach to answering the question of how to achieve Engineering Excellence. We first investigated the root causes of NASA major failures as a basis for developing a proposed answer to the question of Excellence. The following discussions integrate a triad of Technical Understanding and Execution, Partnership with the Project, and Individual and Organizational Culture. The thesis is that you must focus on the whole process and its underlying culture, not just on the technical aspects. In addition to the engineering process, emphasis is given to the need and characteristics of a Learning Organization as a mechanism for changing the culture