Christiansen, Niels Hørbye

Høgsberg, Jan Becker

Job, Jonas Hultmann

Klyver, Katrine

It is shown how the procedure know as optimal brain surgeon can be used to trim and optimize artificial neural networks in nonlinear structural dynamics. Beside optimizing the neural network, and thereby minimizing computational cost in simulation, the surgery procedure can also serve as a quickinput parameter study based on one simulation only

Online Research Database In Technology

                       General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.  • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal   If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.     Downloaded from orbit.dtu.dk on: Dec 20, 2017Optimal Brain Surgeon on Artificial Neural Networks inChristiansen, Niels Hørbye; Job, Jonas Hultmann; Klyver, Katrine; Høgsberg, Jan BeckerPublished in:Proceedings of the 25th Nordic Seminar on Computational MechanicsPublication date:2012Link back to DTU OrbitCitation (APA):Christiansen, N. H., Job, J. H., Klyver, K., & Høgsberg, J. B. (2012). Optimal Brain Surgeon on Artificial NeuralNetworks in. In Proceedings of the 25th Nordic Seminar on Computational MechanicsProceedings of the 25th Nordic Seminar on Computational MechanicsK. Persson, J. Revstedt, G. Sandberg, M. Wallin (Eds.)c©Lund University, 2012Optimal Brain Surgeon on Artificial Neural Networks inNonlinear Structural DynamicsNiels H. Christiansen1,2, Jonas Hultmann Job3, Katrine Klyver4, and Jan Høgsbrg1(1)Department of Mechanical Engineering, Technical University of Denmark, Kgs. Lyngby, Den-mark(2)DNV Denmark, Hellerup, Denmark(3)Rambøll Danmark - Offshore Wind, Copenhagen, Denmark(4)NOV Flexibles, Broendby, DenmarkSummary. It is shown how the procedure know as optimal brain surgeon can be used to trim and op-timize artificial neural networks in nonlinear structural dynamics. Beside optimizing the neural network,and thereby minimizing computational cost in simulation, the surgery procedure can also serve as a quickinput parameter study based on one simulation only.Key words: Nonlinear structural dynamics, Artificial neural networks, Optimal brain surgeon.IntroductionTime domain simulation of nonlinear systems using finite element method (FEM) analysis canbe computationally very expensive. One method that have shown to be very time saving is theuse of artificial neural networks (ANN). In the literature it has been shown on various typesof structures how this method can reduce calculation time by up to two orders of magnitude[1, 2]. However, a major problem with this method is the difficulty of developing a robustalgorithm that automatically generates an optimized ANN for a given structure. In this paperit is shown how a trained ANN can be further optimized and reduced in size by the procedureknown as optimal brain surgeon (OBS). This procedure prunes the least salient network weightsin an ANN. Besides trimming the network this operation can be used as a quick ranking ofthe importance of the various inputs that go into the analysis. Dynamic response of structuresoften depends on several external loads. By applying the OBS procedure to a trained ANN itis possible to determine which of these external force components that play the least significantrole and also evaluate if one or more components can be ignored completely in the analysis.Hence, the OBS procedure can serve as an input parameter study based on one simulation only.Artificial neural networks and optimal brain surgeonArtificial neural networks are in principal mathematical models that replicate the human brain’simpressive ability to do high speed pattern recognition. They serve as tools which performnonlinear mapping between a given input and a corresponding output without time consumingequilibrium iterations. There exists vast literature on the fundamentals of neural networks andthe philosophy behind. Since the ANN is not the main object in this paper it will be left to thereader to consuls e.g. [3] for further information on that topic. Here it will just be mentionedthat an ANN with one input layer, one hidden layer with tangent hyperbolic as activationyt(t− 1, ..., t− 4)WoWiyt(t)BiasBiasInputOutputHiddenfy(t, ..., t− 3)fz(t, ..., t− 3)xyzfy(t)fz(t)Figure 1. Left: ANN with one hidden layer. Right: Model of cantilevered beam.function and a liner output layer as sketched in the left part of figure 1 is used in the followingstudy.OBS is a procedure that prunes the least salient network weights in a trained ANN. Whentraining an ANN the first derivative of the error function is used to find the direction in whichto step in weight space in order to minimize the error. The OBS procedure uses the secondderivative to find the curvature of the error function in weight space. That information can beused to arrange all network weights and to find out which weight is least salient and thereby’cheapest’ to delete [4]. The saliency of a weight q is calculated byLq =12w2q[H−1]qq, (1)where H−1 is the inverse of the Hessian matrix (H ≡ ∂2E/∂w2) containing all second orderderivatives. Having found the least salient weight we can update the rest of the weights usingδw = −wq[H−1]qqH−1 · eq, (2)where eq is the unit vector in weight space. The inverse Hessian is calculated directly usinga slightly modified version of the Sherman-Morrison formula (3) and a single sequential passthrough the training data 1 ≤ m ≤ P .H−1m+1 = H−1m −H−1m Xm+1 ·XTm+1 ·H−m1P +XTm+1H−1m Xm+1, (3)where P is the number of training data sets.So the procedure is to take a network that has been trained to a local minimum, calculateH−1 using (3), find the network weight that gives the smallest saliency using (1), and determinethe adjustment of the network weights (2). This procedure can in principal be repeated until allweights are deleted. However, after each deletion of the cheapest weight one can evaluate theincrease in error and see if it exceeds an acceptable level. In that case the procedure must bestopped.Structural modelTo demonstrate the OBS method a 3D FEM model of a cantilevered beam is set up. Thebeam has the properties listed in table 1 and is shown to the right in figure 1. The model usesthe co-rotational beam element formulation as described in [5]. Hence, it is a nonlinear modelable to handle large rotations and deflections. The beam model is divided into five elements ofequal length and is damped through Rayleigh damping. The beam is subject to two stochasticload histories working in two different directions at the beam tip. The two load histories arel E Iy Iz A ρ10 m 1 · 105 MPa 1 · 104 mm4 1 · 105 mm4 1 · 105 mm2 1 · 105 kg/m3Table 1. Beam propertiesdifferent but have equal characteristics in terms of mean value, standard deviation and dominantfrequency. The load has a mean of 2.7 kN with a standard deviation of 0.3 kN and a dominatingfrequency around 0.7 Hz. The response of the beam is calculated by Newmark’s method ofdirect integration. The Newton-Raphson method is used to achieve force equilibrium in eachtime step, i.e. to update system matrices in accordance with the procedures described in [5].Time step size is 0.1 sec. and equilibrium is assumed when force and displacement residuals arebelow 1× 10−6.Data generated by the FEM is used to train an ANN to predict the horizontal responseyt(t) of the beam tip. A 400 sec. simulated response history is divided into 3 sets. The firstset is used for training the ANN, the second set is used for testing and the last set is saved todemonstrate the accuracy of the pruned ANN on new data. The ANN has 40 neurons in thehidden layer and 13 neurons in the input layer. The network input vector consists of the fourprevious time steps of the tip response and the two loads, as indicated in figure 1. This gives a[13 × 40] weight matrix Wi between input and hidden layer and a [40 × 1] weight matrix Wobetween hidden and output layer.One step in the OBS procedure is to evaluate the network error each time a network weightis deleted. Depending on the structure and the type of analysis carried out one must decidean error increase tolerance for when to stop the deletion of weights. Figure 2 shows how thenetwork error increases with the number of deleted weights. It is seen that it is possible to deleteabout half the weights without increasing the network error considerably.0 100 200 300 400 50010−1110−1010−910−8TesterrorNumber of pruned weightsFigure 2. Test error vs. number of pruned weights.Figure 3 shows response histories calculated by the FEM model and the pruned ANN. It isseen that even though almost half of the network connections have been deleted the ANN stillpredicts the horizontal deflection very accurately. Note that the ANN in this case simulatesabout two orders of magnitude faster than the FEM. If we examine the input weight matrix Wicontaining all connection weights between the input layer and the hidden layer it is possible todo some interpretation of the importance of the input variables.As the procedure deletes the least important weights and since the loads on the beam arearranged in the network input vector it is possible to evaluate the importance of the loads andwhether one or more inputs can be neglected. The pruned weight matrix Wi,prune is writtenbelow. As actual values of individual weights are unimportant for this evaluation of the prunedmatrix positive and negative weights are marked by +/− while deleted weights are denoted 0. Itis seen that the load in the z−direction is almost completely ruled out by the OBS and thereby0 20 40 60 80 100 120 1406812x 10−3  t [s]y t(t) FEMANN20Figure 3. Comparison of calculated response after pruning of 250 network weights.possibly can be completely neglected in the response simulation. This is as one intuitively wouldexpect since fz works vertically and the analysis considers the horizontal deflection.It should be noted that two runs with the OBS procedure will not give identical prunedweight matrices. This is due to the fact that the ANN is initiated with random weights beforetraining. However, the simulation will still be as accurate as the one shown in figure 3 and theconclusions drawn from the interpretation of Wi,pruned will be the same despite some minorchanges in the matrix.+ - + - - 0 - - 0 - - - + - 0 - 0 - + + + + - - 0 + + - - + - 0 - + 0 + - 0 - -fy+ 0 + - 0 + + 0 + - - 0 0 0 + + 0 + 0 0 0 0 + 0 - 0 - 0 - + 0 0 + 0 - 0 0 - 0 0+ - + - - - - - - - - - + - - - 0 0 + + + + - - + 0 + - - 0 - 0 - + + + - + - -0 - + 0 - - - - - - - - + - - - 0 + + + 0 + - - 0 0 0 - 0 0 0 0 - + + + - + - -0 0 0 0 0 + 0 0 0 0 0 - 0 - + 0 0 - 0 0 0 + 0 0 0 0 0 - 0 0 0 0 0 0 0 0 0 0 0 0fz0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0Wi,pruned⇒ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 00 0 0 0 + - + + 0 0 0 + - + - 0 0 - 0 - 0 - 0 + 0 0 + + 0 0 0 0 0 0 + - 0 + + +0 + 0 0 0 + + 0 + 0 0 0 0 0 + + 0 + - 0 0 0 + + - 0 - 0 0 0 0 0 + 0 - - 0 - + 0yt0 - + 0 0 - - - - 0 0 0 + - - - 0 - + + 0 + - - + 0 + 0 0 0 0 0 - 0 + + - + - -0 + 0 0 + + + + + 0 0 + - + + + 0 + - - - - + + - 0 - + 0 0 + 0 + - - - + - + ++ - + - - - - - - - - - + - - - - - + + + + - - + + + - - + - - - + + + - + - -0 - 0 0 0 - - 0 - 0 0 0 0 0 - - 0 - 0 0 0 0 - - + 0 + 0 0 0 0 0 - 0 + + - + 0 0}BiasConcluding remarksThe example presented in this paper demonstrates that an trained ANN can be trimmed andreduced considerably in size by the OBS procedure. The above analysis was carried out on avery simple structure and with simple load patterns. However, it may be possible to use thismethod on more complex structures subject to more complicated loads. In that case the OBSprocedure can serve as a quick input parameter study that can help reducing the computationalcost on nonlinear simulations.References[1] R. Guarize, N.A.F. Matos, L.V.S. Sagrilo, and E.C.P. Lima. Neural networks in dynamic re-sponse analysis of slender marine structures. Applied Ocean Research, (29):191–198, January2008.[2] N. H. Christiansen, J. Høgsberg, and O. Winter. Artificial neural networks for nonlineardynamic response simulation in mechanical systems. Nordic Seminar on ComputationalMechanics, (24):77–80, November 2011.[3] Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, October 2006.[4] B. Hassibi, D.G. Stork, and G.J. Wolff. Optimal brain damage and general network pruning.1993 IEEE International Conference on Neural Networks, vols 1-3, 293-299, 1993.[5] Steen Krenk. Non-linear Modeling and Analysis of Solids and Structures. Cambridge Uni-versity Press, August 2009.

Optimal Brain Surgeon on Artificial Neural Networks in

http://orbit.dtu.dk/ws/files/41012379/NSCM-25_Niels.pdf

Optimal Brain Surgeon on Artificial Neural Networks in

Abstract

Similar works

Full text

Available Versions

Online Research Database In Technology