Neural Network iLQR: A New Reinforcement Learning Architecture

As a notable machine learning paradigm, the research efforts in the context of reinforcement learning have certainly progressed leaps and bounds. When compared with reinforcement learning methods with the given system model, the methodology of the reinforcement learning architecture based on the unknown model generally exhibits significantly broader universality and applicability. In this work, a new reinforcement learning architecture is developed and presented without the requirement of any prior knowledge of the system model, which is termed as an approach of a "neural network iterative linear quadratic regulator (NNiLQR)". Depending solely on measurement data, this method yields a completely new non-parametric routine for the establishment of the optimal policy (without the necessity of system modeling) through iterative refinements of the neural network system. Rather importantly, this approach significantly outperforms the classical iterative linear quadratic regulator (iLQR) method in terms of the given objective function because of the innovative utilization of further exploration in the methodology. As clearly indicated from the results attained in two illustrative examples, these significant merits of the NNiLQR method are demonstrated rather evidently.Comment: 13 pages, 9 figure

Application of multilevel control techniques to classes of distributed parameter plants

This study concerns the application of a combination of multilevel hierarchical systems analysis techniques and Pontryagin\u27s minimum principle (multilevel control) to the problem of controlling optimally two classes of dynamic distributed parameter plants representing concentrations balances in streams, rivers and estuaries. The concentrations treated in this study are those deemed the most effective indicators of water quality, dissolved oxygen (DO) and biochemical oxygen demand (BOD). One class of plants treated in this study consists of linear continuous distributed parameter plants represented mathematically by sets of simultaneous partial differential equations. Optimal control of a plant of this class is initiated by applying spatial discretization followed by a combination of multilevel techniques and Pontryagin\u27s minimum principle for lumped parameter systems. This approach reduces the original problem of optimally controlling a distributed parameter plant to a hierarchy of subproblems comprised of ordinary differential and algebraic equations that can be solved iteratively. A general two-dimensional plant representative of a class of two-step discrete dynamic distributed parameter plants is derived from mass balances at the faces of a model of a volume element of a waterway. The resulting set of simultaneous finite-difference equations represents dynamic balances of concentrations at a finite number of spatial points in a reach of a waterway at selected time instants. Application of Pontryagin\u27s minimum principle for discrete systems in conjunction with multilevel hierarchical systems analysis techniques reduces the problem of controlling such a plant optimally to a hierarchy of subproblems to be solved iteratively. Implicit in the application of optimal control to a plant is the selection of a suitable performance index functional with which to measure the relative optimality of each solution iteration. A variety of performance indices based upon physical considerations is utilized in conjunction with several different control modes for a number of plants representative of the two classes treated in this study. Subproblem hierarchies corresponding to both continuous and discrete distributed parameter plants representing concentrations balances in waterway reaches subject to multilevel optimal control are aggregated into super hierarchies. These super hierarchies possess at least one more level than those corresponding to the single reaches and represent, in this context, the concentrations balances in multireach or regional portions of waterways. Sufficient boundary, initial and final conditions are presented for numerical solution of the subproblem hierarchies developed in this study. Flow charts for the corresponding digital computer programs also are depicted. A proof of consistency between the ordinary differential equations of the spatially discretized plant and the partial differential equations of the continuous distributed parameter plant that it approximates is developed for a representative plant. A proof of convergence of the solutions of the equations of the same spatially discretized plant also is developed. Stability analyses are conducted for representative continuous and discrete distributed parameter plants. The optimal control of the spatially discretized continuous distributed parameter plant is formulated as a linear regulator problem and the associated performance index is utilized as a Liapunov function. The optimal control of the discrete distributed parameter plant with time-varying mean volume flow rate is formulated as the problem of optimal control of a nonstationary system which is treated by transforming the nonstationary system to an equivalent stationary system. The z-transform is applied to the finite-difference equations of the plant to facilitate evaluation of the effect of the presence of transport lags. The relationship between structural characteristics and computational efficiency of subproblem hierarchies is analyzed. Multilevel hierarchical systems analysis techniques are applied to the sensitivity analysis of a spatially discretized distributed parameter plant subject to multilevel optimal control. The combination of discretization and multilevel techniques is shown to reduce the generation of trajectory sensitivity coefficients for an optimally controlled distributed parameter plant to generation of trajectory sensitivity coefficients for a series of lumped parameter plants under optimal control. A normalized performance index sensitivity function also is developed for the same plant. Numerical results of multilevel optimization are presented for various control modes and configurations applied to plants representing: single reaches of a tidal river, four contiguous reaches of a tidal river, six contiguous reaches of a tidal river with taper and waste dischargers, and single reaches of an estuary. The study culminates with the application of one of the single reach subproblem hierarchies for a discrete distributed parameter plant under multilevel optimal control and multilevel hierarchical systems analysis techniques to the problem of minimizing total treatment cost for a multireach portion of a tidal river. This demonstrates the feasibility and efficiency of the multilevel approach to the solution of dynamic systems optimization problems of regional scope

NASA/NBS (National Aeronautics and Space Administration/National Bureau of Standards) standard reference model for telerobot control system architecture (NASREM)

The document describes the NASA Standard Reference Model (NASREM) Architecture for the Space Station Telerobot Control System. It defines the functional requirements and high level specifications of the control system for the NASA space Station document for the functional specification, and a guideline for the development of the control system architecture, of the 10C Flight Telerobot Servicer. The NASREM telerobot control system architecture defines a set of standard modules and interfaces which facilitates software design, development, validation, and test, and make possible the integration of telerobotics software from a wide variety of sources. Standard interfaces also provide the software hooks necessary to incrementally upgrade future Flight Telerobot Systems as new capabilities develop in computer science, robotics, and autonomous system control

A novel numerical framework for simulation of multiscale spatio-temporally non-linear systems in additive manufacturing processes.

New computationally efficient numerical techniques have been formulated for multi-scale analysis in order to bridge mesoscopic and macroscopic scales of thermal and mechanical responses of a material. These numerical techniques will reduce computational efforts required to simulate metal based Additive Manufacturing (AM) processes. Considering the availability of physics based constitutive models for response at mesoscopic scales, these techniques will help in the evaluation of the thermal response and mechanical properties during layer-by-layer processing in AM. Two classes of numerical techniques have been explored. The first class of numerical techniques has been developed for evaluating the periodic spatiotemporal thermal response involving multiple time and spatial scales at the continuum level. The second class of numerical techniques is targeted at modeling multi-scale multi-energy dissipative phenomena during the solid state Ultrasonic Consolidation process. This includes bridging the mesoscopic response of a crystal plasticity finite element framework at inter- and intragranular scales and a point at the macroscopic scale. This response has been used to develop an energy dissipative constitutive model for a multi-surface interface at the macroscopic scale. An adaptive dynamic meshing strategy as a part of first class of numerical techniques has been developed which reduces computational cost by efficient node element renumbering and assembly of stiffness matrices. This strategy has been able to reduce the computational cost for solving thermal simulation of Selective Laser Melting process by ~100 times. This method is not limited to SLM processes and can be extended to any other fusion based additive manufacturing process and more generally to any moving energy source finite element problem. Novel FEM based beam theories have been formulated which are more general in nature compared to traditional beam theories for solid deformation. These theories have been the first to simulate thermal problems similar to a solid beam analysis approach. These are more general in nature and are capable of simulating general cross-section beams with an ability to match results for complete three dimensional analysis. In addition to this, a traditional Cholesky decomposition algorithm has been modified to reduce the computational cost of solving simultaneous equations involved in FEM simulations. Solid state processes have been simulated with crystal plasticity based nonlinear finite element algorithms. This algorithm has been further sped up by introduction of an interfacial contact constitutive model formulation. This framework has been supported by a novel methodology to solve contact problems without additional computational overhead to incorporate constraint equations averting the usage of penalty springs