Numerical Solution of Optimal Control Problems with Explicit and Implicit Switches

This dissertation deals with the efficient numerical solution of switched optimal control problems whose dynamics may coincidentally be affected by both explicit and implicit switches. A framework is being developed for this purpose, in which both problem classes are uniformly converted into a mixed–integer optimal control problem with combinatorial constraints. Recent research results relate this problem class to a continuous optimal control problem with vanishing constraints, which in turn represents a considerable subclass of an optimal control problem with equilibrium constraints. In this thesis, this connection forms the foundation for a numerical treatment. We employ numerical algorithms that are based on a direct collocation approach and require, in particular, a highly accurate determination of the switching structure of the original problem. Due to the fact that the switching structure is a priori unknown in general, our approach aims to identify it successively. During this process, a sequence of nonlinear programs, which are derived by applying discretization schemes to optimal control problems, is solved approximatively. After each iteration, the discretization grid is updated according to the currently estimated switching structure. Besides a precise determination of the switching structure, it is of central importance to estimate the global error that occurs when optimal control problems are solved numerically. Again, we focus on certain direct collocation discretization schemes and analyze error contributions of individual discretization intervals. For this purpose, we exploit a relationship between discrete adjoints and the Lagrange multipliers associated with those nonlinear programs that arise from the collocation transcription process. This relationship can be derived with the help of a functional analytic framework and by interrelating collocation methods and Petrov–Galerkin finite element methods. In analogy to the dual-weighted residual methodology for Galerkin methods, which is well–known in the partial differential equation community, we then derive goal–oriented global error estimators. Based on those error estimators, we present mesh refinement strategies that allow for an equilibration and an efficient reduction of the global error. In doing so we note that the grid adaption processes with respect to both switching structure detection and global error reduction get along with each other. This allows us to distill an iterative solution framework. Usually, individual state and control components have the same polynomial degree if they originate from a collocation discretization scheme. Due to the special role which some control components have in the proposed solution framework it is desirable to allow varying polynomial degrees. This results in implementation problems, which can be solved by means of clever structure exploitation techniques and a suitable permutation of variables and equations. The resulting algorithm was developed in parallel to this work and implemented in a software package. The presented methods are implemented and evaluated on the basis of several benchmark problems. Furthermore, their applicability and efficiency is demonstrated. With regard to a future embedding of the described methods in an online optimal control context and the associated real-time requirements, an extension of the well–known multi–level iteration schemes is proposed. This approach is based on the trapezoidal rule and, compared to a full evaluation of the involved Jacobians, it significantly reduces the computational costs in case of sparse data matrices

Learning to Run challenge solutions: Adapting reinforcement learning methods for neuromusculoskeletal environments

In the NIPS 2017 Learning to Run challenge, participants were tasked with building a controller for a musculoskeletal model to make it run as fast as possible through an obstacle course. Top participants were invited to describe their algorithms. In this work, we present eight solutions that used deep reinforcement learning approaches, based on algorithms such as Deep Deterministic Policy Gradient, Proximal Policy Optimization, and Trust Region Policy Optimization. Many solutions use similar relaxations and heuristics, such as reward shaping, frame skipping, discretization of the action space, symmetry, and policy blending. However, each of the eight teams implemented different modifications of the known algorithms.Comment: 27 pages, 17 figure

Nonlinear Model Predictive Control for Motion Generation of Humanoids

Das Ziel dieser Arbeit ist die Untersuchung und Entwicklung numerischer Methoden zur Bewegungserzeugung von humanoiden Robotern basierend auf nichtlinearer modell-prädiktiver Regelung. Ausgehend von der Modellierung der Humanoiden als komplexe Mehrkörpermodelle, die sowohl durch unilaterale Kontaktbedingungen beschränkt als auch durch die Formulierung unteraktuiert sind, wird die Bewegungserzeugung als Optimalsteuerungsproblem formuliert. In dieser Arbeit werden numerische Erweiterungen basierend auf den Prinzipien der Automatischen Differentiation für rekursive Algorithmen, die eine effiziente Auswertung der dynamischen Größen der oben genannten Mehrkörperformulierung erlauben, hergeleitet, sodass sowohl die nominellen Größen als auch deren ersten Ableitungen effizient ausgewertet werden können. Basierend auf diesen Ideen werden Erweiterungen für die Auswertung der Kontaktdynamik und der Berechnung des Kontaktimpulses vorgeschlagen. Die Echtzeitfähigkeit der Berechnung von Regelantworten hängt stark von der Komplexität der für die Bewegungerzeugung gewählten Mehrkörperformulierung und der zur Verfügung stehenden Rechenleistung ab. Um einen optimalen Trade-Off zu ermöglichen, untersucht diese Arbeit einerseits die mögliche Reduktion der Mehrkörperdynamik und andererseits werden maßgeschneiderte numerische Methoden entwickelt, um die Echtzeitfähigkeit der Regelung zu realisieren. Im Rahmen dieser Arbeit werden hierfür zwei reduzierte Modelle hergeleitet: eine nichtlineare Erweiterung des linearen inversen Pendelmodells sowie eine reduzierte Modellvariante basierend auf der centroidalen Mehrkörperdynamik. Ferner wird ein Regelaufbau zur GanzkörperBewegungserzeugung vorgestellt, deren Hauptbestandteil jeweils aus einem speziell diskretisierten Problem der nichtlinearen modell-prädiktiven Regelung sowie einer maßgeschneiderter Optimierungsmethode besteht. Die Echtzeitfähigkeit des Ansatzes wird durch Experimente mit den Robotern HRP-2 und HeiCub verifiziert. Diese Arbeit schlägt eine Methode der nichtlinear modell-prädiktiven Regelung vor, die trotz der Komplexität der vollen Mehrkörperformulierung eine Berechnung der Regelungsantwort in Echtzeit ermöglicht. Dies wird durch die geschickte Kombination von linearer und nichtlinearer modell-prädiktiver Regelung auf der aktuellen beziehungsweise der letzten Linearisierung des Problems in einer parallelen Regelstrategie realisiert. Experimente mit dem humanoiden Roboter Leo zeigen, dass, im Vergleich zur nominellen Strategie, erst durch den Einsatz dieser Methode eine Bewegungserzeugung auf dem Roboter möglich ist. Neben Methoden der modell-basierten Optimalsteuerung werden auch modell-freie Methoden des verstärkenden Lernens (Reinforcement Learning) für die Bewegungserzeugung untersucht, mit dem Fokus auf den schwierig zu modellierenden Modellunsicherheiten der Roboter. Im Rahmen dieser Arbeit werden eine allgemeine vergleichende Studie sowie Leistungskennzahlen entwickelt, die es erlauben, modell-basierte und -freie Methoden quantitativ bezüglich ihres Lösungsverhaltens zu vergleichen. Die Anwendung der Studie auf ein akademisches Beispiel zeigt Unterschiede und Kompromisse sowie Break-Even-Punkte zwischen den Problemformulierungen. Diese Arbeit schlägt basierend auf dieser Grundlage zwei mögliche Kombinationen vor, deren Eigenschaften bewiesen und in Simulation untersucht werden. Außerdem wird die besser abschneidende Variante auf dem humanoiden Roboter Leo implementiert und mit einem nominellen modell-basierten Regler verglichen

Spacecraft Trajectory Optimization: A review of Models, Objectives, Approaches and Solutions

This article is a survey paper on solving spacecraft trajectory optimization problems. The solving process is decomposed into four key steps of mathematical modeling of the problem, defining the objective functions, development of an approach and obtaining the solution of the problem. Several subcategories for each step have been identified and described. Subsequently, important classifications and their characteristics have been discussed for solving the problems. Finally, a discussion on how to choose an element of each step for a given problem is provided.La Caixa, TIN2016-78365-

Global optimization in systems biology: stochastic methods and their applications

Mathematical optimization is at the core of many problems in systems biology: (1) as the underlying hypothesis for model development, (2) in model identification, or (3) in the computation of optimal stimulation procedures to synthetically achieve a desired biological behavior. These problems are usually formulated as nonlinear programing problems (NLPs) with dynamic and algebraic constraints. However the nonlinear and highly constrained nature of systems biology models, together with the usually large number of decision variables, can make their solution a daunting task, therefore calling for efficient and robust optimization techniques. Here, we present novel global optimization methods and software tools such as cooperative enhanced scatter search (eSS), AMIGO, or DOTcvpSB, and illustrate their possibilities in the context of modeling including model identification and stimulation design in systems biology.This work was supported by the Spanish MICINN project ”MultiSysBio” (ref. DPI2008-06880-C03-02), and by CSIC intramural project ”BioREDES” (ref. PIE-201170E018).Peer reviewe

Numerical solution of optimal control problems with implicitly defined discontinuities with applications in engineering

In the thesis on hand we treat optimal control problems for implicitly discontinuous dynamical processes. We give a general model formulation which includes implicitly given state dependent discontinuities in the right hand sides of the DAE system. The formulation is adapted to real-world applications from chemical and biotechnological engineering. The resulting problems are large scale constrained problems of optimal control with implicitly given discontinuities of a priori unknown chronology and number. Our solution approach builds on the direct multiple shooting approach which allows the combination of appropriate DAE solvers with modern simultaneous optimization strategies. To solve the underlying optimization problem we apply SQP methods. We explain our strategy to provide sensitivity information at the presence of implicitly given discontinuities for large scale models. Efficient techniques for the derivative generation of the right hand sides particularly adapted to structural sparsity pattern changes of the adjacent Jacobians are presented. We formulate an algorithm to treat the optimization problem which depends on the chronology and number of discontinuities occuring in a digraph given by the successive trajectories of the SQP steps. We explain our modeling of a complex rack-in process of a distillation column and present the models of two biotechnological processes. Each of the models is equipped with characteristical implicit state dependent discontinuities of a priori unknown chronology. In numerical experiments we show the efficient applicability of our algorithms to the presented chemical process and to the two biotechnological applications. We apply our approach to optimal feedback control of a biotechnological application with implicit discontinuities

Efficient meta-heuristics for spacecraft trajectory optimization

Meta-heuristics has a long tradition in computer science. During the past few years, different types of meta-heuristics, specially evolutionary algorithms got noticeable attention in dealing with real-world optimization problems. Recent advances in this field along with rapid development of high processing computers, make it possible to tackle various engineering optimization problems with relative ease, omitting the barrier of unknown global optimal solutions due to the complexity of the problems. Following this rapid advancements, scientific communities shifted their attention towards the development of novel algorithms and techniques to satisfy their need in optimization. Among different research areas, astrodynamics and space engineering witnessed many trends in evolutionary algorithms for various types of problems. By having a look at the amount of publications regarding the development of meta-heuristics in aerospace sciences, it can be seen that a high amount of efforts are dedicated to develop novel stochastic techniques and more specifically, innovative evolutionary algorithms on a variety of subjects. In the past decade, one of the challenging problems in space engineering, which is tackled mainly by novel evolutionary algorithms by the researchers in the aerospace community is spacecraft trajectory optimization. Spacecraft trajectory optimization problem can be simply described as the discovery of a space trajectory for satellites and space vehicles that satisfies some criteria. While a space vehicle travels in space to reach a destination, either around the Earth or any other celestial body, it is crucial to maintain or change its flight path precisely to reach the desired final destination. Such travels between space orbits, called orbital maneuvers, need to be accomplished, while minimizing some objectives such as fuel consumption or the transfer time. In the engineering point of view, spacecraft trajectory optimization can be described as a black-box optimization problem, which can be constrained or unconstrained, depending on the formulation of the problem. In order to clarify the main motivation of the research in this thesis, first, it is necessary to discuss the status of the current trends in the development of evolutionary algorithms and tackling spacecraft trajectory optimization problems. Over the past decade, numerous research are dedicated to these subjects, mainly from two groups of scientific communities. The first group is the space engineering community. Having an overall look into the publications confirms that the focus in the developed methods in this group is mainly regarding the mathematical modeling and numerical approaches in dealing with spacecraft trajectory optimization problems. The majority of the strategies interact with mixed concepts of semi-analytical methods, discretization, interpolation and approximation techniques. When it comes to optimization, usually traditional algorithms are utilized and less attention is paid to the algorithm development. In some cases, researchers tried to tune the algorithms and make them more efficient. However, their efforts are mainly based on try-and-error and repetitions rather than analyzing the landscape of the optimization problem. The second group is the computer science community. Unlike the first group, the majority of the efforts in the research from this group has been dedicated to algorithm development, rather than developing novel techniques and approaches in trajectory optimization such as interpolation and approximation techniques. Research in this group generally ends in very efficient and robust optimization algorithms with high performance. However, they failed to put their algorithms in challenge with complex real-world optimization problems, with novel ideas as their model and approach. Instead, usually the standard optimization benchmark problems are selected to verify the algorithm performance. In particular, when it comes to solve a spacecraft trajectory optimization problem, this group mainly treats the problem as a black-box with not much concentration on the mathematical model or the approximation techniques. Taking into account the two aforementioned research perspectives, it can be seen that there is a missing link between these two schemes in dealing with spacecraft trajectory optimization problems. On one hand, we can see noticeable advances in mathematical models and approximation techniques on this subject, but with no efforts on the optimization algorithms. On the other hand, we have newly developed evolutionary algorithms for black-box optimization problems, which do not take advantage of novel approaches to increase the efficiency of the optimization process. In other words, there seems to be a missing connection between the characteristics of the problem in spacecraft trajectory optimization, which controls the shape of the solution domain, and the algorithm components, which controls the efficiency of the optimization process. This missing connection motivated us in developing efficient meta-heuristics for solving spacecraft trajectory optimization problems. By having the knowledge about the type of space mission, the features of the orbital maneuver, the mathematical modeling of the system dynamics, and the features of the employed approximation techniques, it is possible to adapt the performance of the algorithms. Knowing these features of the spacecraft trajectory optimization problem, the shape of the solution domain can be realized. In other words, it is possible to see how sensitive the problem is relative to each of its feature. This information can be used to develop efficient optimization algorithms with adaptive mechanisms, which take advantage of the features of the problem to conduct the optimization process toward better solutions. Such flexible adaptiveness, makes the algorithm robust to any changes of the space mission features. Therefore, within the perspective of space system design, the developed algorithms will be useful tools for obtaining optimal or near-optimal transfer trajectories within the conceptual and preliminary design of a spacecraft for a space mission. Having this motivation, the main goal in this research was the development of efficient meta-heuristics for spacecraft trajectory optimization. Regarding the type of the problem, we focused on space rendezvous problems, which covers the majority of orbital maneuvers, including long-range and short-range space rendezvous. Also, regarding the meta-heuristics, we concentrated mainly on evolutionary algorithms based on probabilistic modeling and hybridization. Following the research, two algorithms have been developed. First, a hybrid self adaptive evolutionary algorithm has been developed for multi-impulse long-range space rendezvous problems. The algorithm is a hybrid method, combined with auto-tuning techniques and an individual refinement procedure based on probabilistic distribution. Then, for the short-range space rendezvous trajectory optimization problems, an estimation of distribution algorithm with feasibility conserving mechanisms for constrained continuous optimization is developed. The proposed mechanisms implement seeding, learning and mapping methods within the optimization process. They include mixtures of probabilistic models, outlier detection algorithms and some heuristic techniques within the mapping process. Parallel to the development of algorithms, a simulation software is also developed as a complementary application. This tool is designed for visualization of the obtained results from the experiments in this research. It has been used mainly to obtain high-quality illustrations while simulating the trajectory of the spacecraft within the orbital maneuvers.La Caixa TIN2016-78365R PID2019-1064536A-I00 Basque Government consolidated groups 2019-2021 IT1244-1