Development of Advanced Verification and Validation Procedures and Tools for the Certification of Learning Systems in Aerospace Applications

Adaptive control technologies that incorporate learning algorithms have been proposed to enable automatic flight control and vehicle recovery, autonomous flight, and to maintain vehicle performance in the face of unknown, changing, or poorly defined operating environments. In order for adaptive control systems to be used in safety-critical aerospace applications, they must be proven to be highly safe and reliable. Rigorous methods for adaptive software verification and validation must be developed to ensure that control system software failures will not occur. Of central importance in this regard is the need to establish reliable methods that guarantee convergent learning, rapid convergence (learning) rate, and algorithm stability. This paper presents the major problems of adaptive control systems that use learning to improve performance. The paper then presents the major procedures and tools presently developed or currently being developed to enable the verification, validation, and ultimate certification of these adaptive control systems. These technologies include the application of automated program analysis methods, techniques to improve the learning process, analytical methods to verify stability, methods to automatically synthesize code, simulation and test methods, and tools to provide on-line software assurance

TRA-956: IMPROVING INTERSECTION THROUGHPUT USING CONNECTED VEHICLES

This paper proposes a connected vehicle based approach to improve the throughput at signalized intersections and ultimately increase the mobility of a transportation system. Connected vehicle technology demonstrates tremendous potential for improving safety and mobility, as it enables the real-time sharing of vehicle data, including position, speed, acceleration, etc., not only among vehicles but also between vehicles and infrastructure. The proposed approach takes advantage of such real-time data to develop a strategy that maximizes throughput of an isolated intersection locally. Accordingly, the problem is formulated as a two-step centralized optimization. There are two main processes in this method: optimization for vehicles in motion, and optimization for stopped vehicles. The first step maximizes the intersection throughput of vehicles in motion using advisory acceleration. The second one minimizes the total delay of the stopped vehicles by adjusting the positions at which vehicles stop. A case study is also presented to show the efficiency of the proposed approach, which improves the traffic flow throughput of an isolated signalized intersection and reduces the total delay of all vehicles

Value-Gradient Learning

This thesis presents an Adaptive Dynamic Programming method, Value-Gradient Learning, for solving a control optimisation problem, using a neural network to represent a critic function in a large continuous-valued state space. The algorithm developed, called VGL(λ), requires a learned differentiable model of the environment. VGL(λ) is an extension of Dual Heuristic Programming (DHP) to include a bootstrapping parameter, λ, analogous to that used in the reinforcement learning algorithm TD(λ). Online and batch-mode implementations of the algorithm are provided, and its theoretical relationships to its precursor algorithms, DHP and TD(λ), are described. A theoretical result is given which shows that to achieve trajectory optimality in a continuous-valued state space, the critic must learn the value-gradient, and this fact affects any critic-learning algorithm. The connection of this result to Pontryagin's Minimum Principle is made clear. Hence it is proven that learning this value-gradient directly will obviate the need for local exploration of the value function, and this motivates value-gradient learning methods in terms of automatic local value exploration and improved learning speed. Empirical results for the algorithm are given for several benchmark problems, and the improved speed, convergence, and ability to work without local value exploration, is demonstrated in comparison to its precursor algorithms, TD(λ) and DHP. A convergence proof for one instance of the VGL(λ) algorithm is given, which is valid for control problems with a greedy policy, and a general nonlinear function approximator to represent the critic. This is a non-trivial accomplishment, since most or all other related algorithms can be made to diverge under similar conditions, and new divergence proofs demonstrating this for certain algorithms are given in the thesis. Several technical problems must be overcome to make a robust VGL(λ) implementation, and these solutions are described. These include implementing an efficient greedy policy, implementing trajectory clipping correctly, and the efficient computation of second-order gradients with a neural network

Modeling Dynamic Systems for Multi-Step Prediction with Recurrent Neural Networks

This thesis investigates the applicability of Recurrent Neural Networks (RNNs) and Deep Learning methods for multi-step prediction of robotic systems. The unmodeled dynamics and simplifying assumptions in classic modeling methods result in models that yield rapidly diverging predictions when the model is used in an iterative fashion, i.e., for multi-step prediction. However, the effect of the unmodeled dynamics can be captured by collecting datasets of the system. Deep Learning provides a strong set of tools to extract patterns from data, however, large datasets are commonly required for the methods to work well. Collecting a large amount of data from a robotic system can be a cumbersome and expensive approach. In this work, Deep Learning methods, particularly RNNs, are studied and employed for the purpose of learning models of two aerial vehicles from experimental data. The feasibility of employing RNNs is first studied to learn a model of a quadrotor based on a simulated dataset, which yields a Multi Layer Fully Connected (MLFC) architecture. Models can be learned for multi-step prediction, recovering excellent predictions over 500 timesteps in the presence of simulated disturbances to the robot and noise on the measurements. To learn models from experimental data, the RNN state initialization problem is defined and formulated. It is shown that the RNN state initialization problem can be addressed by creating and training an initialization network jointly with the multi-step prediction network, and the combination can be used in a black-box modeling approach such that the model produces predictions which are immediately accurate. The RNN based black-box methods are trained on an experimental dataset gathered from a quadrotor and a publicly available helicopter dataset. The quadrotor dataset, which encompasses approximately 4 hours of flight data in various regimes, has been released and is now available publicly online. Finally, a hybrid network, which combines the proposed RNN based black-box models with a physics based quadrotor model into a single RNN-based modeling system is introduced. The proposed hybrid network solves many of the limitations of the existing state of the art in long-term prediction for robotics systems. Trained on the quadrotor dataset, the hybrid model provides accurate body angular rate and velocity predictions of the vehicle over almost 2 seconds which is suitable to be used in a variety of model-based controller applications

Advances in Reinforcement Learning

Reinforcement Learning (RL) is a very dynamic area in terms of theory and application. This book brings together many different aspects of the current research on several fields associated to RL which has been growing rapidly, producing a wide variety of learning algorithms for different applications. Based on 24 Chapters, it covers a very broad variety of topics in RL and their application in autonomous systems. A set of chapters in this book provide a general overview of RL while other chapters focus mostly on the applications of RL paradigms: Game Theory, Multi-Agent Theory, Robotic, Networking Technologies, Vehicular Navigation, Medicine and Industrial Logistic

Quality analysis modelling for development of a process controller in resistance spot welding using neural networks techniques

Student Number : 9811923K - PhD thesis - School of Mechanical Engineering - Faculty of Engineering and the Built EnvironmentMethods are presented for obtaining models used for predicting welded sample resistance and effective weld current (RMS) for desired weld diameter (weld quality) in the resistance spot welding process. These models were used to design predictive controllers for the welding process. A suitable process model forms an important step in the development and design of process controllers for achieving good weld quality with good reproducibility. Effective current, dynamic resistance and applied electrode force are identified as important input parameters necessary to predict the output weld diameter. These input parameters are used for the process model and design of a predictive controller. A three parameter empirical model with dependent and independent variables was used for curve fitting the nonlinear halfwave dynamic resistance. The estimates of the parameters were used to develop charts for determining overall resistance of samples for any desired weld diameter. Estimating resistance for samples welded in the machines from which dataset obtained were used to plot the chart yielded accurate results. However using these charts to estimate sample resistance for new and unknown machines yielded high estimation error. To improve the prediction accuracy the same set of data generated from the model were used to train four different neural network types. These were the Generalised Feed Forward (GFF) neural network, Multilayer Perceptron (MLP) network, Radial Basis Function (RBF) and Recurrent neural network (RNN). Of the four network types trained, the MLP had the least mean square error for training and cross validation of 0.00037 and 0.00039 respectively with linear correlation coefficient in testing of 0.999 and maximum estimation error range from 0.1% to 3%. A prediction accuracy of about 97% to 99.9%. This model was selected for the design and implementation of the controller for predicting overall sample resistance. Using this predicted overall sample resistance, and applied electrode force, a second model was developed for predicting required effective weld current for any desired weld diameter. The prediction accuracy of this model was in the range of 94% to 99%. The neural network predictive controller was designed using the MLP neural network models. The controller outputs effective current for any desired weld diameter and is observed to track the desired output accurately with same prediction accuracy of the model used which was about 94% to 99%. The controller works by utilizing the neural network output embedded in Microsoft Excel as a digital link library and is able to generate outputs for given inputs on activating the process by the push of a command button