Good generalization results are obtained from neurofuzzy networks if its structure is suitably chosen. To select the structure of neurofuzzy networks, the authors proposed a construction algorithm that is derived from the Support Vector Regression. However, the modeling errors are assumed to be uncorrelated. In this paper, systems with correlated modeling errors are considered. The correlated noise is modeled separately by a recurrent network. The overall network is referred to as the support vector recurrent neurofuzzy networks. The prediction error method is used to train the networks, where the derivatives are computed by a sensitivity model. The performance of proposed networks is illustrated by an example involving a nonlinear dynamic system corrupted by correlated noise.published_or_final_versio

Chan, CW

Chan, WC

Cheung, KC

Harris, CJ

HKU Scholars Hub

Title Support vector recurrent neurofuzzy networks in modelingnonlinear systems with correlated noiseAuthor(s) Chan, WC; Chan, CW; Cheung, KC; Harris, CJCitation Annual Conference Of The North American Fuzzy InformationProcessing Society - Nafips, 2001, v. 1, p. 545-550Issued Date 2001URL http://hdl.handle.net/10722/46664Rights©2001 IEEE. Personal use of this material is permitted. However,permission to reprint/republish this material for advertising orpromotional purposes or for creating new collective works forresale or redistribution to servers or lists, or to reuse anycopyrighted component of this work in other works must beobtained from the IEEE.Support vector recurrent neurofuzzy networks in modeling nonlinear systems with correlated noise W. C. Chan, C. W. Chan, K. C. Cheung Department of Mechanical Engineering The University of Hong Kong Pokhlam Road, Hong Kong, China Abstract Good generalization results are obtained l?om neurofkzy networks if its structure is suitably chosen. To select the structure of neurofuzzy networks, the authors proposed a construction algorithm that is derived fiom the Support Vector Regression. However, the modeling errors are assumed to be uncorrelated. In this paper, systems with correlated modeling errors are considered. The comlated noise is modeled separately by a recurrent network. The overall network is referred to as the support vector recurrent neurofuzzy networks. The prediction error method is used to train the networks, where the derivatives are computed by a sensitivity model. The performance of proposed networks is illustrated by an example involving a nonlinear dynamic system corrupted by correlated noise. Keywords: Support vectors, recurrent neurofuzzy network, sensitivity model, correlated noise. 1. Introduction Neurofuzzy networks were shown to have the transparency of fuzzy systems and the modeling ability of neural networks [ 121. However, the goodness of the approximation depends not only on the training algorithm, but also on the structure of the network, i.e., the number of basis functions and their locations in the input space. If a-priori knowledge of the system is available, the structure of the network can be suitably chosen tiom this knowledge. If, however, no a-priori knowledge is available, the structure of the network is usually chosen from the training data, which a well- known problem. Support Vector Regression (SVR) developed from statistical learning theory is a linear- in-weights network [5].  The structure and the parameters of the network are obtained by minimizing a cost function that consists of the modeling errors and the complexity of the network. The structure of the SVR is now selected for a given error bound from the training data, which is known as the Support Vector (SV). It is shown that the SVR can be transformed to a C. J. Harris Department of Electronics and Computing University of Southampton, UK radial basis function (RBF) neural network [13,15]. The structure of the transformed RBF network is the same as that of the SVR obtained objectively for a given error bound. However, the noise of the system is assumed to be a white noise [13]. In this paper, this result is extended to systems with correlated noise. To model the correlated noise, an additional recurrent neurofuzzy network is used, yielding the support vector recurrent neurofbzy network (SVRNFN). The training of the SVRNFN is derived fiom the predictipn error method [6]. Newton-Raphson technique is used to optimize the cost fiinction, with the derivatives obtained fkom the sensitivity model. The organization of the paper is as follows. In section 2, the modeling of dynamic systems with correlated noise is presented. A brief description of the SVR algorithm is given in section 3, fiom which the SVRNFN is derived in section 4. The training of the SVRNFN derived l?om the prediction error method is derived in section 5. The performance of the SVRNFN and the proposed training algorithm is illustrated by an example given in section 6. 2. Modelling of Nonlinear Stochastic Systems Consider the Nonlinear Auto-Regressive Moving Average model with exogenous input (NARMAX) given by [8,10], y(k)= F(y(k-I), ..., y(k -mbu(k-l)...,u(k - p i  e(k-1 ),..., e(k-n))+e(k) (1) where yfi), U&) and e&)- N(0, d) are respectively the output, input and white noise; m, p and n, the respective orders; and F(o), a smooth nonlihear function. There is a wide class of nonlinear stochastic systems that can be modeled by NARMAX (1) around an equilibrium point subject to the following conditions [SI: A 1 : a state-space realization of the system exists, A2: a linearized model exists when operating close to the given equilibrium point. 0-7803-7078-3/0lf$l0~00 (C)U)ol EEE. Page: 545 For simplicity, it is assumed that the correlated noise is independent of y&), U&), hence (1) can be simplified to where f() and g() are smooth nonlinear functions, X(k)  = b(k - 1) ..., y(k - M) u(k - 1) ..., ~ ( k  - p)]' and E(k) = [e(k - l),...,e(k - n)r . The model given by (2) is referred to as the simplified NARMAX model [lo]. Since the correlated noise is bounded with a mean of zero, hence we have, N= f (X(k ) )+  g(E(k))+ 4) (2) g@)= 0 (3) 3G E %+ S.Z. lg(E)I < G V E  E '8'' (4) 3. Support Vector Regression Rewrite (2), where d(k) = g(E(k)) + e(k) is the correlated noise. The nonlinearity fix@)) can be approximated by a Support Vector Machine (SVM) as given below [5]. where N is the length of data for determining the structure of f(*) ; x(~F[x~(R), .  . . J,AN, m=m,+m.; K(X(k),  X(i)) is a kernel satisfying the Mercer's condition [2]; ETi and ai are the Lagrange multipliers, and b is the bias. The following fourth order B-spline kernels is used [2], Y ( 4  = f (X(k) )+  4 k )  ( 5 )  f ( N k ) ) =  Cf,(zi - n , ) ~ ( ~ t k b ~ ( i ) ) + b  (6) where B4 (4 = (22 +X)3 /6~3  -22 I < -A (21 - x)' /sa3 a I x 2 a  0 otherwise where h is the distance between two consecutive knots. The estimate of Ei and ai are obtained by minimizing the cost function, L(zi ,ai )= 3 cyj=, (zi -ai )(zj -aj ) K ( x ( ~ ) x ( . ~ ) )  - C(".1 (q - ai )Ai>+ (q + ai )E (9) subjectto Os& S C I N  and 0 2 %  S C I N ,  where& and C are the given precision level and regularization constant respectively. Minimizing L(.,.) yields the following cases: If Iy(i)- f ( ~ ( i ) ) (  e E ,  then 6 = ai = o (10) O<E, < C I N  (1 1) ai = o  If y(i) - f ( X ( i ) )  = E , then E, = C I N  a, = o  El = o  If y(i)- f ( ~ ( i ) )  > E ,  then If f ( ~ ( i ) ) -  y(i) > E , then a, = C I N  Condition (10) implies that some of the Lagrange multipliers are zero. Consequently, the number of kernels in the SVM (6) can be reduced. Assuming n data points satisfy cases (11)-(14), the SVM is simplified as, f ( ~ ( k ) ) =  Q jK(X(k ) ,  x 'O' ) )+~ (15) where {X'(l),. .., X ( n ) )  is a subset of the training data referred to as the Support Vectors (SV) [ 1-51, and { aI,. . ,a,,) are given by, The bias 6 is computed fiom Conditions (1 1 )-( 14), the so-called Karush-Kuhn-Tucker conditions [ 1,2]. The model (8) is known as the support vector regression (SVR). It is shown that (13) and (14) do not occur if C is sufficiently large, implying that the estimate of d(k) is bounded by ix [14]. Note that an optimal structure offiX(k)) is suitably chosen subject to (4) if E is less than G. s = @(i) : Iy(i) - f(x(i)] 2 E )  A = ki -ai - : I&)- f(x(i))I 2 €1 (16) (17) 4. Support Vector Recurrent Neurofuzzy NeurofUzzy networks are often used in modeling nonlinear systems for given input-output data. It is shown that under certain conditions, they are transparent and that the linguistic rules can be deduced fiom the network [12]. A recurrent network is used to implement a stochastic system represented by an ARMA model [9]. In this work, recurrent neurofuzzy networks are used to implement the NARMAX model. Since A.) and g(.) are well-defined, they can be approximated by the following neurofuzzy networks. Network i(W) = z;:., b, 4 (m)- zZ1 b, 4 (2) (1 9) where &i and bi are the estimated weights; n and n', the respective numbers of weights; A,(X(R)), the normalized kernel is given by, 0-7803-7@78-3/0l/$lO.~ (C)Ml IEEE Page: 546 and Bi (E(k)) is the tensor product of the univariate B- spline fimctions [ 11,121, Bi (E(R))= n;l1 s:, (4’ - i)) (21) where si(.) is given by, (22) i f  X E I X J - I , X J )  otherwise It should be noted in (19) that &I) = 0 , satisfying (3).  From ( 1  8 )  and (1 9), the prediction ofy(k) is, %k) = c;=, 4 A, (X(k))+ x;:, B, B, (E(k))  (23) As ?(X(k)) is a Support Vector Neurofuzzy Network derived fiom the SVR, (23) is referred to as the support vector recurrent neurofuzzy network (SVRNFN), as shown in figure 1. The training of the network will be discussed in the next section. 5. Training of the SVRNFN The prediction error (PE) method is a general parameter estimation method, which can be implemented by the Newton-Raphson technique [6 ] .  In this paper, a training algorithm is proposed for training the SVRNFN, which is derived following closely the PE method. Similarly, the following assumptions are made. S1: Let ek=[e(k), ..., e(0)JT and sz” be ealgebra generated by (ek, U”>. So E[e(k) I C f l ]  = 0 . ~ 2 :  I 0k-112 SI,  6 > 0. Let DM be a compact set, and M, the model set that consists of model (23), M =Cw(e):~(k,e)=rT(k)~~eE D ~ ]  (24) where r ( k )  =[A,(k) ... A,(k) B,(k)...B,,.(k)]T and i = [a, ... a, p, . . . pne ]’. The training algorithm aims at finding the model with the smallest mean square errors fiom M, (25) Since the mean of e(k) is zero, the neurohzzy network is trained fiom the input-output data, {U(]), I N 2  J(8)  =&=,e (4 ..., u(N), y(l), ..., y(N)} by minimizing the cost function (25). The Newton-Raphson method is used here to estimate the weights in (24). The prediction error of SVRNFN (23) is given by, where the weight 6 is computed iteratively by the Newton-Raphson technique, where 1 is the number of iterations, 6 = [el e2. ..en+,], rtk)=[rl(k) rdk) ... r n + f l ~ k ) ~ T ,  Y = [e( l ) ,  e(2), ..., e(N)IT, and cb and S are e ( k )  = y ( k )  - r (k )6  (26) e(1+1)=8(1)-(cbTcb+s)cbyI (27) The partial derivatives of the prediction errors with respect to 8 can be generated by the sensitivity model, as given below [7] .  5.1 Sensitivity model The derivatives of prediction errors of the SVRNFN can be derived by dynamic back propagation [7] .  Define k = j  (29) 0, otherwise Differentiating (21) with respect to e(i) gives The derivative of an univariate B-spline function can be obtained recursively as follows [ 11, 121. Assuming that the initial conditions of the first and second partial derivatives are zero, differentiating (26) with respect to Bi, yields, 0-7803-7U78-3/0l/$l0.00 (C)u)ol BEE. Page: 547 This linear dynamic model is known as the sensitivity model, with the fmt partial derivative c3e(k)/LWi as the output, and the basis function as the input [7]. An advantage of using the neurofuzzy networks is that the coefficients of the dynamic model are given by the derivatives of the basis hctions,  therefore, . the computation time will not be too large for small me. Define i = n + 1, ..., n+ n’ and ci (kd= j )  k-12 j 2 k-me (33) {? otherwise Differentiating (32) with respect to Oi, a sensitivity model of the second partial derivative is obtained. 5.2 Training Procedure The procedure to train the recurrent neural networks is summarized below. (1) Choose suitable X(k), E@), me, E, h and C. (2) Obtain SV by minimizing L(.) given by (9). (3) Initialize weights of j(X(&)) by that of SVR. (4) Compute e(k), for k- 1,. . . f l  by (26). (5) Compute @and S by (32) and (34). (6) Update the parameters &1+1) by (27). (7) Repeat (4)-(6) until IAJ(i)J(6] < E ’ ,  say lo3 .  6. Examples Consider the following nonlinear system given in [ 121, y ( k )  = (0.&0.5e-Y2(k-1) )y(k-1)(0.3 + 0.9e-y2(k-’))y(k-2) (35) where r(k) - N(0, 0. 12), and p is a constant set to 0.5, 0,s and 1 respectively. The iterative map of (v(k - 2), y(k  - 1)) with an initial condition of (0.1, 0-1), when there is no noise, diverges from an unstable equilibrium near the origin towards a periodic attractor, as shown in Fig. 2. In the example, 1000 input-output data are generated from (35) with an initial condition of (0,O). Since most of the data are on the periodic attractor, as shown in Fig. 2, only the first 50 data points are required to select the SV. The distance between the inner knots L is set to 1.25, and the regularization constant C, to 100. The precision E is chosen to be 0.122, 0.122 and 0.149 for the respective value of p. Note that E for p = 1 is larger, as the variance of the correlated noise is larger. From Step 2 in training procedure, 16 SV are selected from + 0.1 sin(ny(k-1))- ps(k-l)+ €(k) the 50 data points, which are marked by circles in Fig. 3. In the SVRNFN, 16 normalized B-spline kemels are used to model &), and 2 triangular B-spline functions are used to model i(*). The initial estimate of a is now computed, as given in Step 3. The Newton-Raphson technique is now applied to train the SVRNFN with derivatives computed ftom the sensitivity model. The estimate iterative map ofA.) in all three cases are shown in Figs. 4 to 6. The autocorrelation functions of the prediction errors for p = 0.5, as shown in Fig. 7, indicated that the prediction errors are statistically uncorrelated. The autocorrelation functions from the other two cases are similar, indicating that good generalization results are obtained. 7. Conclusion The SVRNFN is proposed to model dynamic systems with correlated noise. The main feature of the SVRNFN is that the structure of the sub-network modeling the nonlinear system is determined by the SV, which is selected similar to that in the SVR. A recurrent neurofuzzy network is used to model the correlated noise. The training algorithm of the SVRNFN is derived based on the prediction error method. The Newton-Raphson technique is proposed to train the SVRNFN, and the derivation of the derivatives using the sensitivity model is given. The SVRNFN is applied to model a nonlinear system corrupted by correlated noise. Good generalization results are obtained using the SVRNFN, and the training algorithms based on the Newton-Raphson technique. 8. References B. Schblkopf, P. Bartlett, A.J. Smola, and R. Williamson, “Shrinking the Tube: A New Support Vector Regression Algorithm,” Advances in Neural Information Processing Systems, 11, M.S. Keams, S.A. Solla, and D.A. Cohn, (eds.), MIT Press, Cambridge, MA, 1999. C.J.C. Burges, “Geometry and Invariance in Kemel Based Methods,” Advanced in Kernel Methoh - Support Vector Learning, B. Schblkopf, C. Burges, and A. Smola (eds.), MIT Press, Cambridge, USA, 1998, pp 89-1 16. P.M.L. Drezet, and R.F. Harrison, “Support Vector Machines for System Identification,” UKACC International Conference on CONTROL ‘98, 1-4 September 1998, pp 688- 692. ’ S.R. Gunn, and M. Brown, “SUPANOVA - A Sparse, Transparent Modelling Approach,” Page: 548 Proc. IEEE Neural Networks for Signal Processing Workshop, 1999, pp 2 1-30. V. Vapnik, The nature of statistical learning theory, Springer-Verlag New York, 1995. L. Ljung, “Analysis of a General Recursive Prediction Error Identification Algorithm,” Automatica, 17, 1981, pp 89-99. K.S. Narendra, and K. Parthasamthy, “Gradient methods for the optimization of dynamical systems containing neural networks,” Trans. IEEE on Neural Networks, 2, 1991, pp 252- 262. LJ. Leontaritis, and S.A. Billings, “Input- output parametric models for nonlinear systems Part 11: stochastic nonlinear systems,” Int. J .  Control, 41, 1985, pp 329-344. J.T. COMOr, R.D. Martin, and L.E. Atlas, “Recurrent Neural Networks and Robust Time Series Prediction,” Trans. IEEE on Neural Networks, 5 ,  1991, pp 240-253. S .  Chen, S.A. Billings, C.F.N. Cowan, and P.M. Grant, “Practical identification of input Xfi )  Vectors space support network output t NARMAX models using radial basis functions,” Int. J. Control, 52, 1990, pp 1327- 1350. [Ill L. Piegl, and W. Tiller, The NURBS Book, Springer-Verlag, Berlin Heidelberg, 1995. 1121 M. Brown, and C.J. Harris, Neurofiuzy Adaptive Modelling and Control, Prentice Hall, 1994. W.C. Chan, C.W. Chan, K.C. Cheung and C.J. Harris, “Modelling of nonlinear dynamic systems using support vector neural networks,” Preprints IFAC Qmposium AIRTC 2000, Budapest, Hungary, 2-4 October 2000, pp 2 17-222. V. Vapnik, Statistical Learning Theov, John Wiley & Sons, New York, 1998. W.C Chan, C.W. Chan, K.C. Cheung, and C.J. Harris, “On the modelling of nonlinear dynamic systems using support vector neural networks,” Engineering Applications of Artificial intelligence, 14,2001, pp 105-1 13. [13] [I41 [15] -1.5 -1 -0.5 0 0.5 1 1.5 Fig. 2 The iterative map ofA.). 0-7803-7078-3/0U$l0.00 (C)uH)l IEEE. Page: 549 1'5 r------ -0.5. -1 0'. 0 . 0, . .  ' g.@ O . -1.51 "'! 0.2 -1.5 -1 -0.5 0 0.5 1 1.5 Fig. 3 Support Vectors foryO.5 and 170.122. ' Fig. 4 Estimated iterative map offi.) for ~ 0 . 5 .  Fig. 5 Estimated iterative map ofA.) for y 0 . 8 .  1.5 1 0.5 0 -0.5 -1 -1.5 -1.5 -1 -0.5 0 0.5 1 1.5 Fig. 6 Estimated iterative map offf.) for p=l . 11 1 0.8 R 1 01- 4.2 I 1 0 5 10 15 20 Fig. 7 The autocorrelation functions of e(k) forp4.5. 0-7803-7078-3/0U$10.00 (C)Uw)l IEEE. Page: 550 

Support vector recurrent neurofuzzy networks in modeling nonlinear systems with correlated noise

http://hub.hku.hk/bitstream/10722/46664/1/68089.pdf

Support vector recurrent neurofuzzy networks in modeling nonlinear systems with correlated noise

Abstract

Similar works

Full text

Available Versions

HKU Scholars Hub