Practical recommendations for gradient-based training of deep architectures

Learning algorithms related to artificial neural networks and in particular for Deep Learning may seem to involve many bells and whistles, called hyper-parameters. This chapter is meant as a practical guide with recommendations for some of the most commonly used hyper-parameters, in particular in the context of learning algorithms based on back-propagated gradient and gradient-based optimization. It also discusses how to deal with the fact that more interesting results can be obtained when allowing one to adjust many hyper-parameters. Overall, it describes elements of the practice used to successfully and efficiently train and debug large-scale and often deep multi-layer neural networks. It closes with open questions about the training difficulties observed with deeper architectures

Multi-layer functional approximation of non-linear unsteady aerodynamic response

Non-linear unsteady aerodynamic effects present major modelling difficulties in the analysis of aeroelastic response and in the subsequent design of appropriate controllers. As the direct use of the basic fluid mechanic equations is still not practical for aeroelastic applications, approximate models of the non-linear unsteady aerodynamic response are required. A rigorous mathematical framework, that can account for the complex non-linearities and time-history effects of the unsteady aerodynamic response, is provided by the use of functional representations. A recent development, based on functional approximation theory, has provided a new functional form; namely, multi-layer functionals. Moreover, the multi-layer functional representation for time-invariant, infinite memory systems is shown to be realisable in terms of temporal neural networks. In this work, a multi-layer functional representation of non-linear motion-induced unsteady aerodynamic response is presented. A discrete-time, finite memory temporal neural network, in the form of a finite impulse response (FIR) neural network, is used as a practical realisation of a multi-layer functional. This model form permits the identification of parametric input-output models of the non-linear motion-induced unsteady aerodynamic response. Identification of an appropriate FIR neural network model is facilitated by means of a supervised training process using multiple sets of motion-induced unsteady aerodynamic response. The training process is based on a conventional genetic algorithm to optimise the FIR neural network architecture, and is combined with a simplification of the simulated annealing algorithm to update weight and bias values