A new recurrent neural network based on B-spline function approximation is presented. The network can be easily trained and its training converges more quickly than that for other recurrent neural networks. Moreover, an adaptive weight updating algorithm for the recurrent network is proposed. It can speed up the training process of the network greatly and its learning speed is more quickly than existing algorithms, e.g., back-propagation algorithm. Examples are presented comparing the adaptive weight updating algorithm and the constant learning rate method, and illustrating its application to modelling of nonlinear dynamic system.published_or_final_versio

Chan, CW

Cheung, KC

Jin, H

Zhang, HY

HKU Scholars Hub

Title B-spline recurrent neural network and its application tomodelling of non-linear dynamic systemsAuthor(s) Chan, CW; Cheung, KC; Jin, H; Zhang, HYCitation The 1998 American Control Conference, Philadelphia, PA., 24-26June 1998. In Conference Proceedings, 1998, v. 1, p. 78-82Issued Date 1998URL http://hdl.handle.net/10722/46636Rights©1998 IEEE. Personal use of this material is permitted. However,permission to reprint/republish this material for advertising orpromotional purposes or for creating new collective works forresale or redistribution to servers or lists, or to reuse anycopyrighted component of this work in other works must beobtained from the IEEE.Proceedings of  the American Control Conference Philadelphia, Pennsylvania June 1998 B-Spline Recurrent Neurall Network and Its Application to Modelling C. W. Chan*, K. C. Cheung", Hong Jint, H. Y. Zhangt * Department of Mechanical Engineering, University of Hong Kong Pokfulam road, Hong Kong, Email: mechan@hkucc.hku.hk f Department of Automatic Control, Beijing University of Aeronautics and Astronautics Beijing 100083, P. R. China, Email: buaa301 @mimicnc.aC.cn The authors are currently with the Department of Mechanical Engineering, University of Hong Kong Abstract: A new recurrent neural network based on B- spline function approximation is presented. The network can be easily trained and its training converges more quickly than that for other recurrent neural networks. Moreover, an adaptive weight updating algorithm for the recurrent network is proposed. It can speed up the training process of the network greatly and its learning speed is more quickly than existing algorithms, e.g., back-propagation algorithm. Examples are presented comparing the adaptive weight updating algorithm and the constant learning rate method, and illustrating its application to modelling of nonlinear dynamic system. Keywords: Recurrent Neural Network, B-Spline network, Adaptive Learning Algorithm, State Estimation, System Modelling. 1. Introduction Recurrent neural networks(RNN) have proved to be valuable tools and have been studied extensively in modelling and control of non-linear dynamic systems, e.g., the designs of non-linear dynamic system identifier and controller[ 13 [2][3]. When the state variables of non-linear dynamic system are not accessible, the identification of the system and the system states[4](i.e., observer's problem), is substantially more complex than in the case where the states are accessible. The critical issues in the application of recurrent networks are the choice of the network architecture, i.e., number of network elements, and the location of feedback loops, or RNN based on Gaussian radial basis function[5] with multilayer feedforward neural network, etc.[ 1][6] and the development of suitable training procedures. A single or multiple hidden layer networks with feedback loop induces the dynamic complexity while the number of hidden layer elements are associated with the degree of non-linearity. In addition, the training of recurrent network is of much more complex than the training of feedforward networks. Another problem associated with recurrent networks is that the convergence of learning can be very slow and training errors are not always guaranteed to reduce to previously defined tolerances. By using constructive training methods, it is possible that the neural network could possibly build itself little by little, and speed up the whole training process. So a good architecture and quickly learning algorithm are important for us to apply RNN better. In this paper, a new recurrent neural network architecture is proposed, which is based on B-Spline Neural Network(BSNN)[7] and denoted by BSRNN(B-Spline Recurrent Neural Network). Basis spline function has several advantages. For example, it has a good ability of approximation; it only needs local adjustment of weights for every input; it needs less computation and storage than other basis functions(e.g., Gaussian function and Berstein function etc.); moreover, the derivative of basis spline function can be readily obtained. Further, the training of the BSNN is more quickly than other networks(e.g., multilayer feedforword network) and has been applied for guidance[ 81, fault detection and isolation[9], Kalman filter initialisation[ 101. Here we discuss the recurrent networks based on BSNN. Our aim is to model nonlinear dynamic systems better and faster by applying the basis spline function for training recurrent networks. It is, to our knowledge, that few works have focused on BSRNNs. Other parts of this paper are arranged as follows: In 2"d and 3'd sections, we briefly discuss BSNN and BSRNN respectively, and give the BSRNN models and their architectures for two kinds of non-linear systems. In 4" and 5* sections, we discuss the learning algorithm of weights and the determination of the amounts of weights to update. In 6* section, an adaptive weight updating algorithm is proposed. This adaptive updating method can increase the learning speed greatly, make the learning process converge quickly and overcome the ad hoc choice of learning rate in weight updating. In 7th section, an example is given to show 0-7803-4530-4198 $10.00 0 1998 AACC 78 the application of BS,RNN to the modelling of nonlinear dynamic systems. Finally, we give the conclusion about BSRNN. 2. B-Spline neural network Let x ( t )  E R "  be the input vector of the network. By using the operation of tensor product, we can calculate the multivariate basis function vector, i.e., transformed input vector[7], s(x) = (s, ( x )  ... sq (x))' or so-called box spline[ 111 of x and the jth multivariate B-spline function is denoted by s,(x) . The number q of multivariate B-spline functions is dependent on the order of basis spline function and the number of inner knots of the interval given for every component of x. The space {s(x)} is so-called transformed input space. These q multivariate B-spline functions can be considered as net inpu1.s of the hidden layer, and the output of network ?( t )  (Fig.1)) is, i ( t >  = x;=, w s, (x ( t>> (1) where w, is the weight valueo= I,. . .,q). W I  h:&fl Fig. 1: BSNN(B-Spline Neural Network) When x is a univariate variable, equation (1) can be considered as a B-spline curve[ 121, otherwise, as a B-spline surface[ 131, where ( w1 , - , wq ) can be also considered as control points[l4], so the process of adjusting the weights is that of adjusting the control points. The main advantage of the B-spline formulation over other curve fitting(e.g., the Bezier curve) is local control of the curve shape, i.e., the shape of the curve clhanges only in the vicinity of few changed control points[l2]. As in the training of BSNN, only a few weights need to be adjusted, much computation time can be saved. The p-order B-spline function s, (x) can be determined by using following B-spline recurrence relationship[ 131: (3) I Ti  5 x < for i=l, ..., n 0 otherwise s,p (x) = where zi is the i-th knot. 3. B-Spline recurrent neural network In BSNN, the training ie undertaken by supervising learning, i.e., the input of network is known and output of network need to be compared to the actual value which is accessible. But for some applications, only output of a system can be obtained. When we need to know the inner state of a system, recurrent neural network is a good choice. In this section, two kinds of recurrent neural network based on BSNN are described for two classes of non-linear models. These two models are given as follows: Model I: x( i  + 1) = f ( x ( i ) ,  x(i - l), . . . x ( i  --n+ 1); (4) u(i), u(i - l),... , u(i - m+ 1)) Model 11: xn-r (i + 1) = f , ,-r  (xI (i),. . . , x ~ - ~ - ,  (9, xn-r (0,. ., x,, (9; X"( i+ l )  =fn(xl(i) ,..., x"-~-l(i) ,xn-r(i) , . . . ,x"(i);  u(i), u(i - l), . .e, u(i -m+ 1)) u(i), u(i - l),..., u(i -m+ 1)) ( 5 )  I where r is given. For these two models, we can use BSNN and BSRNN to implement. For example of Model I, its BSNN and BSRNN models can be described by following equations: BSNN: (6) E(i + 1) = BSNN ( ~ ( i ) ,  ~ ( i  - l),. . * ~ ( i  - n + 1); u( i ) ,u( i - l ) ; . . ,u( i -m+I))  BSRNN( Fig. 2) : (7) i ( i +  1) = BSRNN(i(i), i ( i  - l ) ,**- i ( i  -n+ 1); u(i), u(i - l), ... , u(i - m+ 1)) (6) and (7) are called series-parallel model and parallel model respectively [ 11. @ci -m+l) Fig.2: BSRNN of Model I Fig.3: BSRNN of Model I1 Fig.3 gives the BSRNN of Model 11. In these two BSRN", the output of network is fed back as an input. In this case, the transformed input vector is dependent on the weight 79 values of the previous step, which influences the updating of the weights. This issue will be discussed next. 4. Learning algorithm for BSRNN In training BSRNN, the steepest descent gradient algorithm can be used to adjust weights of network. For example, in Model I, a natural performance criterion for a recurrent network is the sum of the squared of errors between the target sequence and the outputs of the network, where y(i)  is the real output of system(4) at time i, i ( i )  is the output of the BSRNN for estimating y(i). The least-squares approximation to y(i)  is to minimise E by finding the optimal weight values {wi , i  = I , . . .  , q }  . A typical learning rule is the so-called dynamic back propagation algorithm[ 151 and the weights are adjusted by, where q is the learning rate or step size. Weight adjustments can be performed at each time or in a batch mode. E = Ct, ( ~ ( i )  - j ( i ) ) *  (8) rl . way1 wold (9) WneW = - 5. Determination of the partial derivatives with respect to the weights The partial derivative of (8) with respect to wt can be written as follows: (10) dE -- ay, where k= 1 , .  ... q. From ( 1), Because the transformed input vector of the BSRNN is dependent on the weights of the network, in contrast to that of the BSNN or other non-recurrent radial basis function neural networks, the second term in the right hand side of (11) is not equal to zero. It needs to know how large the effect caused by this factor is when calculating (10). In matrix formation, (1 1) becomes s1 (x(1)) * * +  s1 (x(m>) s,(x(l)) --.  s,(x(" ... . . . . . .  ... ] ... 1 % 1 where the determination of is, ( x ( i ) ) p w ,  is very complex. As the 2nd term in the right hand side of equation (12) requires a heavy calculation load, Considerable saving in computing time can be made if this term is ignored. It is shown in the following example that ignoring this term affects only the rate of convergence of the estimates of the weights. Example 1: Consider a model as follows: x(k) = 0.5sin(x(k - 1)) - O.Ix(k - 2) + u ( k )  (13) where u(k) is zero-mean white noise with the standard deviation of CT = 0.01 . Given x(0 )  = 0, x ( l )  = 1 . Fig.4 gives curves of errors performance with or without considering the second term of (12). (a) the number of inner knots N ,  = 1 ;  (b) inner knots N ,  = 2 .  It shows that consideration of the 2nd term in (12) has no obvious advantage. y ( k )  = 100x(k) (14) ....................... 4 0 0  2ooo ?b 2b 3b 4; 5 0  E p o c h  I term I" ( 5 . 3 )  ................... 500 ' 0  ~b I b  2b 2 5  30 1 7 E p o c h  Fig.4: Performance indexes 6. Adaptive Weight Updating Algorithm The proposed learning algorithm is based on the gradient decent method, and is similar to the backpropagation algorithm. Hence it suffers from the same drawback as that in the back propagation algorithm, i.e., slow learning rate. This slowness arises largely from the lack of suitable methods in selecting the step size 7 7 .  It is that if 77 is sufficiently small, the overall learning process will be stable[l6], but the cost for small 77 is a very long time for convergence. But if the step size is chosen too large, there is a possibility of divergence. In this section, we give an optimal value of learning rate for both BSNN and BSRNN. Firstly, we discuss the determination of learning rate of BSNN. Let Y = [ ~ ( I )  ... y(m>l', P) =[y(,'(1) ... y'"((m)l', A f ( k )  = Y - f',), S = [s(l) s(m)]' . where k denotes the training epoch time, s ( i )  = [sI1 ... s,,]' is composed ... of q B-spline function values of the i-th input vector. An adaptive learning step size is derived as 6 = IIS'Af(k-l) 11' /ll,S'Af(k-l) ll$s where Ilxll: := x'Ax . The derivation is given in the Appendix. As for the BSRNN, if the matrix S is used as an approximation instead of the right hand side of (12), then (15) also applies. The following example compares the (15) 80 adaptive weight updating algorithm with ordinary constant learning rate algorithm by using BSNN. Example 2: Consider :v = sin' x . Since no information on the learning rate is available, we tried different learning rates and find q = 0.01 is best. Table 1 gives epoch number I( N e  ) and its final performance index (E)  for different numbler N ,  of inner knots and a constant learning rate 7 = 0.01 by using lSt-order B-spline function to train the BSNN, the samples of variable x = 0:0.1:27~, and the initial weights were zero. The condition of stopping learning is: E < eps or ( E ( k ) -  E ( k  - l ) ) / E ( k  -1 )  < eps ,  where eps = 1.e - 6 . Table 2 gives the results under the same condition except the adaptive learning rate given by (15) is used and the initial learning rate is 17 = 0.01 . Table 1: Using a constant learning rate 7 = 0.01 I Nk I Ne I E I I ;; 1 ;;i4 I 0.0022 1 2.233e-4 I 40 I 2581 I 3.600e-5 I Table 2: Usin the acla tive learnin rate (15) m-1 -1 20 20 0.0022 2.2327e-4 3.5986e-5 Clearly, the learning is much faster when the adaptive learning rate is used. 7. Example Example 3: Modelling of ARMA models Model: x ( k  +1) = O.Olx(k)+0.4sin(3x(k))+05u(k)  y ( k )  = sin(x(k)) +O.lx(k - 1) Let x1 ( k )  = x ( k )  , x:! ( k )  = x ( k  - 1) , the architecture of BSRNN is shown in Fig.5. Simulation conditions: initial input x, (1) = 1 , x 2  (1) = 1 , initial learning rate ~ ( 0 )  = 0.95, eps=O.OOl, N ,  = 5  for every variable of input, u(i)  = sin(2ni/l0) + sin(2ni/25) , sample number m=100 and 1"-order B-spline function for training. Fig.5: Architecture of BSRNN Fig.6 gives some simulation results. (a) real state with its estimate of network; (b) estimate error curve of state; (c) learning rate curve; (d) performance index curve; (e) real measurement with its estimate; (f) measurement estimate error curve. When epoch time N e  = 720, E=0.00099942, its corresponding learning rate is q = 2.84775. I 1 1 1 0 , .  . . . . . . (d). Perfomanes index 0 5 10 IS 20 25 30 15 E w r h  81 0 02 If). Eslimaleerror of measurement: Y ( k ) - p ( k )  Fig.6 8. Conclusion Recurrent neural networks based on B-spline neural networks are proposed in this paper. In order to decrease the computational burden of the learning algorithm, an adaptive learning rate of updating weights is used. Simulation results have shown that a great reduction of learning time is achieved. Acknowledgment The work was supported by the Research Grants Council of Hong Kong. Reference [l]. Narendra K. S .  and K. Parthasarathy(l990), Identification and Control of Dynamic Systems Using Neural Networks, IEEE Trans. on Neural Networks, Vol.1, [2]. Yip P. P. C. and Y. H. Pao(1994), A Recurrent Network Net Approach to One -Step Ahead Control Problems, IEEE Trans. on Systems, Man, and Cybernetics, Vo1.24, No.4, [3]. Park Y. M., M. S .  Choi and K.Y. Lee, An Optimal Tracking Neuro-Controller for Nonlinear Dynamic Systems, IEEE Trans. on Neural Networks, Vo1.7, No.5, 1996, 1099- 11 10. [4]. Levin A. U. and K. S .  Narendra(l996), Control of Nonlinear Dynamical Systems Using Neural Networks---- Part 11: Observability, Identification, and Control, IEEE Trans. On Neural Networks, Vo1.7, No.1, pp30-42. [5 ] .  Obradovic D, On-line Training of Recurrent Neural Networks with Continuous Topology Adaptation, IEEE Trans. on Neural Networks, Vo1.7, No. 1996, 222-228. [6]. Lee S.  W. and H. H. Song, A New Recurrent Neural - Network Architecture for Visual Pattern Recognition, IEEE Trans. on Neural Networks, Vol.8, No.2, 1997, pp331-340. [7] .  Brown M. and C .  Harris(1994), Neurofuzzy Adaptive Modelling and Control, Prentice Hall International (UK) Limited. [8]. Doyle R. S.  and C. J. Harris(1996), Multi-sensor data fusion for helicopter guidance using neuro-fuzzy estimation algorithms, The Aeronautical Journal of the Royal Aeronautical Society, June/July, pp240-25 1. [9]. H. Benkhedda and R. J. Patton, B-spline Network Integrated Qualitative and Quantitative Fault Detection, IFAC 13th Triennial World Congress, San Francisco, USA, 1996. NO. 1, pp4-27. ~ ~ 6 7 8 - 6 8 3 .  82 [lo]. Roberts J. M., D. J. Mills, D. Charnley and C. J. Harris( 1995), Improved Kalman Filter Initialisation using Neurofuzzy Estimation, 1995/6 Research Journal: Image, Speech and Intelligent Systems, The University of Southampton, pplO.1-10.10 [ 111. Chui C. K., Multivariate Spline, Society for Industrial and Applied Mathematics, 1988. [12]. Newman W. M. and R. F. Sprod1(1979), Principles of Interactive Computer Graphics, McGraw-Hill, Inc. [13]. Piegl L. and W. Tiller, The NURBS Book, Springer- Verlag Berlin Heidelberg, 1995. [14]. Wang C. H., W. Y. Wang, T. T. Lee and P. S .  Tseng, Fuzzy B-spline Membership Function (BMF) and Its Applications in Fuzzy-neural Control, IEEE Trans. on Systems, Man, and Cybernetics, Vo1.25, No.5, 1995, [15]. Levin A. U. and K. S .  Narendra(1993), Control of Nonlinear Dynamical Systems Using Neural Networks: Controllability and Stabilization, IEEE Trans. On Neural Networks, Vo1.4, No.2, pp192-206. [16]. Narendra K.S. and S .  M. Li, Neural Networks in Control System, Neural Networks Theory, Technology, and Applications (Ed. By P. K. Simpson), The Institute of Electrical and Electronics Engineers ,Inc., New York City, New York, 1996. ~ ~ 8 4 1 - 8 5 1 .  Appendix: Derivation of (15) The performance index at the k-th iteration can be written as follows: For the BSNN, the 2”d term of right hand side in (1 1) is equal to zero, so (9) can be rewritten as follows (A21 Here we have not considered the coefficient 2 in (10). Because j ( k )  (i) = s’(i)w(k) and noting that E ( k )  = x;l(y(i)-j(k)(i))z (AI) w ( k )  = w(k - 1) + vS’A?~-’) j j ( k )  (i) - j j (k-l )  (i) = qsr(i)StAf(k-I) Differentiating this quadratic equation with respect to parameter , (15) can be easily derived. 

B-spline recurrent neural network and its application to modelling of non-linear dynamic systems

http://hub.hku.hk/bitstream/10722/46636/1/31907.pdf

B-spline recurrent neural network and its application to modelling of non-linear dynamic systems

Abstract

Similar works

Full text

Available Versions

HKU Scholars Hub