Reinforcement Learning for Mobile Robot Collision Avoidance in Navigation Tasks

Collision avoidance is fundamental for mobile robot navigation. In general, its solutions include: {\it map-based} and {\it mapless approaches.} In the map-based approach, robots pre-plan collision-free paths based on an environment map and follow their paths during navigation. On the other hand, the mapless approach requires robots to avoid collisions without referencing to an environment map. This thesis first studies the map-based approach for multiple robots to collectively build environment maps. In this study, a robot following a pre-planned path may encounter unexpected obstacles, such as other moving robots and obstacles inaccurately presented on an environment map. This motivates us to study mapless collision avoidance in the second part of the thesis. Mapless collision avoidance requires a robot to infer an optimal action based on sensor data and operate in real time. Inferring an optimal action in a timely manner is computationally expensive, particularly when a robot has limited on-board computing resources. To avoid the expensive online action inferring, this thesis presents a reinforcement learning approach which learns policies for mapless collision avoidance under real-world settings. We first propose a Real-Time Actor-Critic Architecture (RTAC) to support asynchronous reinforcement learning under real-time constraint. Based on RTAC, we propose asynchronous reinforcement learning methods for mapless collision avoidance of various numbers of robots under different environment configurations. Through extensive experiments, we demonstrate that RTAC serves as a solid foundation to support multi-task and multi-agent learning for mapless collision avoidance under asynchronous settings

Low Complexity Regularization of Linear Inverse Problems

Inverse problems and regularization theory is a central theme in contemporary signal processing, where the goal is to reconstruct an unknown signal from partial indirect, and possibly noisy, measurements of it. A now standard method for recovering the unknown signal is to solve a convex optimization problem that enforces some prior knowledge about its structure. This has proved efficient in many problems routinely encountered in imaging sciences, statistics and machine learning. This chapter delivers a review of recent advances in the field where the regularization prior promotes solutions conforming to some notion of simplicity/low-complexity. These priors encompass as popular examples sparsity and group sparsity (to capture the compressibility of natural signals and images), total variation and analysis sparsity (to promote piecewise regularity), and low-rank (as natural extension of sparsity to matrix-valued data). Our aim is to provide a unified treatment of all these regularizations under a single umbrella, namely the theory of partial smoothness. This framework is very general and accommodates all low-complexity regularizers just mentioned, as well as many others. Partial smoothness turns out to be the canonical way to encode low-dimensional models that can be linear spaces or more general smooth manifolds. This review is intended to serve as a one stop shop toward the understanding of the theoretical properties of the so-regularized solutions. It covers a large spectrum including: (i) recovery guarantees and stability to noise, both in terms of $\ell^2$-stability and model (manifold) identification; (ii) sensitivity analysis to perturbations of the parameters involved (in particular the observations), with applications to unbiased risk estimation ; (iii) convergence properties of the forward-backward proximal splitting scheme, that is particularly well suited to solve the corresponding large-scale regularized optimization problem

Computational structure‐based drug design: Predicting target flexibility

The role of molecular modeling in drug design has experienced a significant revamp in the last decade. The increase in computational resources and molecular models, along with software developments, is finally introducing a competitive advantage in early phases of drug discovery. Medium and small companies with strong focus on computational chemistry are being created, some of them having introduced important leads in drug design pipelines. An important source for this success is the extraordinary development of faster and more efficient techniques for describing flexibility in three‐dimensional structural molecular modeling. At different levels, from docking techniques to atomistic molecular dynamics, conformational sampling between receptor and drug results in improved predictions, such as screening enrichment, discovery of transient cavities, etc. In this review article we perform an extensive analysis of these modeling techniques, dividing them into high and low throughput, and emphasizing in their application to drug design studies. We finalize the review with a section describing our Monte Carlo method, PELE, recently highlighted as an outstanding advance in an international blind competition and industrial benchmarks.We acknowledge the BSC-CRG-IRB Joint Research Program in Computational Biology. This work was supported by a grant from the Spanish Government CTQ2016-79138-R.J.I. acknowledges support from SVP-2014-068797, awarded by the Spanish Government.Peer ReviewedPostprint (author's final draft

Local adaptation to hummingbirds and bees in Salvia stachydifolia : insights into pollinator shifts in a Southern Andean sage

Differences among populations in pollinator assemblages can lead to local adaptation mosaics in which plants evolve different floral morphologies and attractive traits. Mountain habitats may promote local adaptation because of differences in environmental conditions with altitude, causing changes in pollinators, and because mountaintops can act as isolated habitats. We studied if the differences in floral shape, size and nectar traits in Salvia stachydifolia can be attributed to variations in the relative contribution of hummingbirds and insects.• Methods We studied eight populations of S. stachydifolia in natural and under common garden conditions, to assess whether population differences have a genetic component. We recorded pollinators, their behaviour and visitation rates, and characterized pollinator assemblages. In addition, we measured nectar volume and concentration, and collected flowers to describe floral shape and size variation using geometric morphometric methods. We then applied an unsupervised learning algorithm to identify ecotypes based on morphometric traits. Finally, we explored whether populations with different pollinator assemblages had different climatic and/or elevationpreferences.• Key Results We found that variation in the identity of the main pollinators was associated with differences among populations in all traits, as expected under a local adaptation scenario. These differences persisted in the common garden, suggesting that they were not due to phenotypic plasticity. We found S. stachydifolia populations were pollinated either by bees, by hummingbirds or had mixed pollination. We identified two ecotypes that correspond to the identity of the main pollinator guilds, irrespective of climate or altitude.• Conclusions Variation in S. stachydifolia floral traits did not follow any evident association with bioclimatic factors, suggesting that populations may have diverged as the product of historical isolation on mountaintops. We suggest that differences among populations point to incipient speciation and an ongoing pollinator shift.Fil: Izquierdo, Juliana Verónica. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba. Instituto Multidisciplinario de Biología Vegetal. Universidad Nacional de Córdoba. Facultad de Ciencias Exactas Físicas y Naturales. Instituto Multidisciplinario de Biología Vegetal; ArgentinaFil: Costas, Santiago. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba. Instituto Multidisciplinario de Biología Vegetal. Universidad Nacional de Córdoba. Facultad de Ciencias Exactas Físicas y Naturales. Instituto Multidisciplinario de Biología Vegetal; ArgentinaFil: Castillo, Santiago. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba. Instituto Multidisciplinario de Biología Vegetal. Universidad Nacional de Córdoba. Facultad de Ciencias Exactas Físicas y Naturales. Instituto Multidisciplinario de Biología Vegetal; ArgentinaFil: Baranzelli, Matias Cristian. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba. Instituto Multidisciplinario de Biología Vegetal. Universidad Nacional de Córdoba. Facultad de Ciencias Exactas Físicas y Naturales. Instituto Multidisciplinario de Biología Vegetal; ArgentinaFil: Sazatornil, Federico David. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba. Instituto Multidisciplinario de Biología Vegetal. Universidad Nacional de Córdoba. Facultad de Ciencias Exactas Físicas y Naturales. Instituto Multidisciplinario de Biología Vegetal; ArgentinaFil: Benitez Vieyra, Santiago Miguel. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba. Instituto Multidisciplinario de Biología Vegetal. Universidad Nacional de Córdoba. Facultad de Ciencias Exactas Físicas y Naturales. Instituto Multidisciplinario de Biología Vegetal; Argentin

Design and analysis of adaptive noise subspace estimation algorithms

Ph.DDOCTOR OF PHILOSOPH

State estimation, system identification and adaptive control for networked systems

A networked control system (NCS) is a feedback control system that has its control loop physically connected via real-time communication networks. To meet the demands of `teleautomation', modularity, integrated diagnostics, quick maintenance and decentralization of control, NCSs have received remarkable attention worldwide during the past decade. Yet despite their distinct advantages, NCSs are suffering from network-induced constraints such as time delays and packet dropouts, which may degrade system performance. Therefore, the network-induced constraints should be incorporated into the control design and related studies. For the problem of state estimation in a network environment, we present the strategy of simultaneous input and state estimation to compensate for the effects of unknown input missing. A sub-optimal algorithm is proposed, and the stability properties are proven by analyzing the solution of a Riccati-like equation. Despite its importance, system identification in a network environment has been studied poorly before. To identify the parameters of a system in a network environment, we modify the classical Kalman filter to obtain an algorithm that is capable of handling missing output data caused by the network medium. Convergence properties of the algorithm are established under the stochastic framework. We further develop an adaptive control scheme for networked systems. By employing the proposed output estimator and parameter estimator, the designed adaptive control can track the expected signal. Rigorous convergence analysis of the scheme is performed under the stochastic framework as well

Neural networks in control engineering

The purpose of this thesis is to investigate the viability of integrating neural networks into control structures. These networks are an attempt to create artificial intelligent systems with the ability to learn and remember. They mathematically model the biological structure of the brain and consist of a large number of simple interconnected processing units emulating brain cells. Due to the highly parallel and consequently computationally expensive nature of these networks, intensive research in this field has only become feasible due to the availability of powerful personal computers in recent years. Consequently, attempts at exploiting the attractive learning and nonlinear optimization characteristics of neural networks have been made in most fields of science and engineering, including process control. The control structures suggested in the literature for the inclusion of neural networks in control applications can be divided into four major classes. The first class includes approaches in which the network forms part of an adaptive mechanism which modulates the structure or parameters of the controller. In the second class the network forms part of the control loop and replaces the conventional control block, thus leading to a pure neural network control law. The third class consists of topologies in which neural networks are used to produce models of the system which are then utilized in the control structure, whilst the fourth category includes suggestions which are specific to the problem or system structure and not suitable for a generic neural network-based-approach to control problems. Although several of these approaches show promising results, only model based structures are evaluated in this thesis. This is due to the fact that many of the topologies in other classes require system estimation to produce the desired network output during training, whereas the training data for network models is obtained directly by sampling the system input(s) and output(s). Furthermore, many suggested structures lack the mathematical motivation to consider them for a general structure, whilst the neural network model topologies form natural extensions of their linear model based origins. Since it is impractical and often impossible to collect sufficient training data prior to implementing the neural network based control structure, the network models have to be suited to on-line training during operation. This limits the choice of network topologies for models to those that can be trained on a sample by sample basis (pattern learning) and furthermore are capable of learning even when the variation in training data is relatively slow as is the case for most controlled dynamic systems. A study of feedforward topologies (one of the main classes of networks) shows that the multilayer perceptron network with its backpropagation training is well suited to model nonlinear mappings but fails to learn and generalize when subjected to slow varying training data. This is due to the global input interpretation of this structure, in which any input affects all hidden nodes such that no effective partitioning of the input space can be achieved. This problem is overcome in a less flexible feedforward structure, known as regular Gaussian network. In this network, the response of each hidden node is limited to a -sphere around its center and these centers are fixed in a uniform distribution over the entire input space. Each input to such a network is therefore interpreted locally and only effects nodes with their centers in close proximity. A deficiency common to all feedforward networks, when considered as models for dynamic systems, is their inability to conserve previous outputs and states for future predictions. Since this absence of dynamic capability requires the user to identify the order of the system prior to training and is therefore not entirely self-learning, more advanced network topologies are investigated. The most versatile of these structures, known as a fully recurrent network, re-uses the previous state of each of its nodes for subsequent outputs. However, despite its superior modelling capability, the tests performed using the Williams and Zipser training algorithm show that such structures often fail to converge and require excessive computing power and time, when increased in size. Despite its rigid structure and lack of dynamic capability, the regular Gaussian network produces the most reliable and robust models and was therefore selected for the evaluations in this study. To overcome the network initialization problem, found when using a pure neural network model, a combination structure· _in which the network operates in parallel with a mathematical model is suggested. This approach allows the controller to be implemented without any prior network training and initially relies purely on the mathematical model, much like conventional approaches. The network portion is then trained during on-line operation in order to improve the model. Once trained, the enhanced model can be used to improve the system response, since model exactness plays an important role in the control action achievable with model based structures. The applicability of control structures based on neural network models is evaluated by comparing the performance of two network approaches to that of a linear structure, using a simulation of a nonlinear tank system. The first network controller is developed from the internal model control (IMC) structure, which includes a forward and inverse model of the system to be controlled. Both models can be replaced by a combination of mathematical and neural topologies, the network portion of which is trained on-line to compensate for the discrepancies between the linear model _ and nonlinear system. Since the network has no dynamic ·capacity, .former system outputs are used as inputs to the forward and inverse model. Due to this direct feedback, the trained structure can be tuned to perform within limits not achievable using a conventional linear system. As mentioned previously the IMC structure uses both forward and inverse models. Since the control law requires that these models are exact inverses, an iterative inversion algorithm has to be used to improve the values produced by the inverse combination model. Due to deadtimes and right-half-plane zeroes, many systems are furthermore not directly invertible. Whilst such unstable elements can be removed from mathematical models, the inverse network is trained directly from the forward model and can not be compensated. These problems could be overcome by a control structure for which only a forward model is required. The neural predictive controller (NPC) presents such a topology. Based on the optimal control philosophy, this structure uses a model to predict several future outputs. The errors between these and the desired output are then collected to form the cost function, which may also include other factors such as the magnitude of the change in input. The input value that optimally fulfils all the objectives used to formulate the cost function, can then be found by locating its minimum. Since the model in this structure includes a neural network, the optimization can not be formulated in a closed mathematical form and has to be performed using a numerical method. For the NPC topology, as for the neural network IMC structure, former system outputs are fed back to the model and again the trained network approach produces results not achievable with a linear model. Due to the single network approach, the NPC topology furthermore overcomes the limitations described for the neural network IMC structure and can be extended to include multivariable systems. This study shows that the nonlinear modelling capability of neural networks can be exploited to produce learning control structures with improved responses for nonlinear systems. Many of the difficulties described are due to the computational burden of these networks and associated algorithms. These are likely to become less significant due to the rapid development in computer technology and advances in neural network hardware. Although neural network based control structures are unlikely to replace the well understood linear topologies, which are adequate for the majority of applications, they might present a practical alternative where (due to nonlinearity or modelling errors) the conventional controller can not achieve the required control action