Fuzzy heterogeneous neural networks are recently introduced models based on neurons accepting heterogeneous inputs (i.e. mixtures of numerical and non-numerical information possibly with missing data) with either crisp or imprecise character, which can be coupled with classical neurons. This paper compares the effectiveness of this kind of networks with time-delay and recurrent architectures that use classical neuron models and training algorithms in a signal forecasting problem, in the context of finding models of the central nervous system controllers.Peer ReviewedPostprint (author's final draft

Alquézar Mancho, René

Belanche Muñoz, Luis Antonio

Valdés Ramos, Julio José

English

Fuzzy heterogeneous neural networks are recently introduced models based on neurons accepting heterogeneous inputs (i.e. mixtures of numerical and non-numerical information possibly with missing data) with either crisp or imprecise character, which can be coupled with classical neurons. This paper compares the effectiveness of this kind of networks with time-delay and recurrent architectures that use classical neuron models and training algorithms in a signal forecasting problem, in the context of finding models of the central nervous system controllers.Peer Reviewe

UPCommons

Fuzzy heterogeneous neural networks for signal forecasting

UPCommons. Portal del coneixement obert de la UPC

Fuzzy Heterogeneous Neural Networksfor Signal ForecastingLlus Belanche, Julio J. Valdes and Rene AlquezarDept. Llenguatges i Sistemes Informatics, Universitat Politecnica de Catalunyafbelanche, valdes, alquezarg@lsi.upc.esBarcelona, SpainAbstractFuzzy heterogeneous neural networks are recently introduced models basedon neurons accepting heterogeneous inputs (i.e. mixtures of numericaland non-numerical information possibly with missing data) with eithercrisp or imprecise character, which can be coupled with classical neu-rons. This paper compares the eectiveness of this kind of networks withtime-delay and recurrent architectures that use classical neuron modelsand training algorithms in a signal forecasting problem, in the context ofnding models of the central nervous system controllers.1 IntroductionA fuzzy heterogeneous neuron is dened as a mapping h :^Hn! Rout R,satisfying h() = 0 ( is the empty set). Here R denotes the reals and^Hnisa cartesian product of an arbitrary number of source sets. Source sets may befamilies of extended reals^R = R[ fXg, extended fuzzy sets^Fi= Fi[ fXg,and extended nite sets of the form^Oi= Oi[ fXg;^Mi= Mi[ fXg, whereeach of the Oihas a full order relation, while the Mihave not. In all cases,the special symbol X denotes the unknown element (missing information) andit behaves as an incomparable element w.r.t. any ordering relation. Accordingto this denition, neuron inputs are possibly empty arbitrary tuples, composedby n elements among which there might be reals, fuzzy sets, ordinals, nominalsand missing data [1], [2]. Heterogeneous neurons are classied according to thenature of their image set (which need not be restricted to a subset of the reals).In the present study, since the image set is given byRoutthe model is of the realkind, which is easily coupled with other classical neuron models (i.e. acceptingonly real inputs), thus leading to hybrid networks in a straightforward way.These networks have been used successfully in classication problems reportedelsewhere [1], but their potential of application in other elds was not yetassessed experimentally. The purpose of this paper is to explore further theperformance of fuzzy heterogeneous networks (in hybrid architectures) in asignal forecasting task concerning the central nervous system control.The paper is organized as follows. Section 2 reviews the concept of fuzzy he-terogeneous neurons and their use in conguring hybrid networks, while section3 describes the problem at hand, covering also the dierent neural paradigmscompared to the one presented, the experiment setup and the obtained results.Finally, section 4 presents the conclusions.Belanche, L.; Valdés, J.; Alquézar, R. Fuzzy heterogeneous neural networks for signal forecasting. A: International Conference on Artificial Neural Networks. "ICANN 98: proceedings of the 8th International Conference on Artificial Neural Networks, Skövde, Sweden, 2-4 September 1998". Berlín: Springer, 1998, p. 1089-1094. The final authenticated version is available online at https://doi.org/10.1007/978-1-4471-1599-1_1712 Heterogeneous Neural NetworksA particular class of heterogeneous networks (HNNs) is constructed by con-sidering h as the composition of two mappings, that is, h = f  s , such thats :^Hn! R0 R and f : R0! Rout R. The mapping h can be consid-ered as a n-ary function parameterized by a n-ary tuple^~w ^Hnrepresentingneuron's weights, i.e. h(^~x;^~w) = f(s(^~x;^~w)). In particular, function s repre-sents a similarity and f a squashing non-linear function with its image in [0; 1].Accordingly, the neuron is sensitive to the degree of similarity between its in-puts {composed in general by a mixture of continuous and discrete quantitiespossibly with missing data{ and its weights. More precisely, s is understoodas a similarity index, or proximity relation (transitivity considerations are putaside). That is, a binary, reexive and symmetric function s(x; y) with im-age on [0; 1] such that s(x; x) = 1 (strong reexivity). The concrete instanceof the model under study in the present paper uses as aggregation function aGower-like similarity index in which the computation for heterogeneous entitiesis constructed as a weighted combination of partial similarities over subsets ofvariables. This coecient has its values in the real interval [0; 1] and for anytwo objects i; j given by tuples of cardinality n, is given by sij=Pnk=1gijkijkPnk=1ijkwhere gijkis a similarity score for objects i; j according to their value for vari-able k. These scores are in the interval [0; 1] and are computed according todierent schemes for numeric and qualitative variables. The factor ijkis a bi-nary function expressing whether objects i, j are comparable or not accordingto their values w.r.t. variable k. Gower's original denitions [3] for real-valuedand discrete variables are kept, although other similarity functions are possi-ble. For variables representing fuzzy sets, similarity relations from the point ofview of fuzzy theory have been dened elsewhere [4] and dierent choices arepossible. In our case, if Fiis an arbitrary family of fuzzy sets from the sourceset, and~A;~B are two fuzzy sets such that~A;~B 2 Fi, the following similarityrelation is used:g(~A;~B) = supx(~A\~B(x)) where ~A\~B(x) = min(~A(x); ~B(x)):For the activation function, a modied version of the classical logistic is used,which is an automorphism of the real interval [0; 1].f(x; p) =( p(x 0:5) a(p)  a(p) if x  0:5 p(x 0:5)+a(p)+ a(p) + 1 otherwisewhere a(p) is an auxiliary function given by a(p) = 0:5+p0:52+4p2and p is areal-valued parameter controlling the curvature, set in the experiments to 0.1.The general training procedure for the HNN is based on genetic algorithms,since the heterogeneity of the variables involved and the non-dierentiability ofthe similarity function prevent the use of gradient-based techniques [1].3 A case study in signal forecasting3.1 Problem descriptionThe problem studied consists of forecasting the output signals of the CentralNervous System (CNS) controllers of the hemodynamical system. This sys-tem, together with the CNS control, form the cardiovascular system. TheCNS generates the regulating signals for the blood vessels and the heart, andit is composed of ve controllers: heart rate, peripheral resistance, myocardialcontractility, venous tone and coronary resistance. All of these controllers aresingle-input/single-output (SISO) systems driven by the same input variable,namely the carotid sinus pressure. Whereas the structure and functioning of thehemodynamical system are well known and a number of quantitative models,mostly based on dierential equations, have been developed, the functioningof the CNS control is of high complexity and still not completely understood.Although some dierential equation models for the CNS have been postulated,these models are not accurate enough, and therefore, the use of other model-ing approaches like neural networks or qualitative methodologies may oer aninteresting alternative for capturing the behaviour of the CNS control [5].3.2 Neural approaches used in the experimentsTwo types of neural network architectures can be used for learning tasks in-volving a dynamic input/output relation, such as prediction and temporal as-sociation: time-delay neural networks (TDNNs) and recurrent neural networks(RNNs) [6]. The HNN model is to be compared to a RNN and two dierentTDNN models described below.3.2.1 Time-delay neural networksIf some xed-length segment of the most recent input values is consideredenough to perform the task successfully, then a temporal sequence can be turnedinto a set of spatial patterns on the input layer of a multi-layer feedforward nettrained with an appropriate algorithm such as backpropagation. These archi-tectures are called TDNNs, since several values from an external signal arepresented simultaneously at the network input using a moving window (shiftregister or tapped delay line) [6]. A main advantage of TDNNs in front ofRNNs is their lower cost of training, which is very important in case of longtraining sequences. TDNNs have been applied extensively in recent years todierent tasks, in particular to prediction and system modeling [7]. In the caseof learning a SISO controller, with an input real-valued variable x(t) and anoutput real-valued variable y(t), the output layer of a TDNN consists of a singleoutput unit that will provide the predicted value for y(t), whereas the inputlayer holds some previous values y(t 1); : : : ; y(t m) and some recent values ofthe input variable x(t); x(t 1); : : : ; x(t p), from which the value y(t) could beestimated (i.e. a total number of m+p+1 input units). Additionally, a hiddenlayer of N units (to be determined) is required. In the present study, two dif-ferent TDNN approaches that dier in the training method have been tested: astandard backpropagation algorithm (TDNN-BP) using sinusoidal units, and ahybrid procedure composed of repeated cycles of simulated annealing coupledwith a conjugate gradient algorithm (TDNN-AC) [8]. For the latter, hyperbolictangent units form the hidden layer whereas the output layer is composed bya linear neuron. It should be noted that the HNN model as used here (TD-HNN) can be viewed as a TDNN that incorporates heterogeneous neurons andis trained by means of genetic algorithms.3.2.2 Recurrent neural networksIn recent years, several RNN architectures including feedback connections, to-gether with their associated training algorithms, have been devised to copenaturally with the learning and computation of tasks involving sequences andtime series [6]. A type of RNN that has been proven useful in grammaticalinference through next-symbol prediction is the rst-order augmented single-layer RNN (or ASLRNN) [9], which is similar to Elman's SRN [10] exceptthat is trained by a true gradient-descent method, using backpropagation forthe feed-forward output layer and Schmidhuber's RTRL algorithm [11] for thefully-connected recurrent hidden layer. Although the use of sigmoidal activa-tion functions has been common in RNNs, a better learning performance canbe achieved using other activation functions such as the sine function [9]. Suchnetworks with sinusoidal units can be seen as generalized discrete Fourier serieswith adjustable frequencies [7]. Hence, the ASLRNN model used here was builtup with sinusoidal units.3.3 Experiment setupThe data used in the training and test phases of the experiments came froma single subject. Five CNS control models, namely, heart rate, peripheral re-sistance, myocardial contractility, venous tone and coronary resistance, wereinferred for this subject by means of the neural approaches aforementioned.The input and output signals of the CNS controllers were recorded with a sam-pling rate of 0.12 seconds from simulations of a purely dierential equationmodel. This model had been tuned to represent a specic patient sueringfrom coronary arterial obstruction, by making the four dierent physiologicalvariables (right auricular pressure, aortic pressure, coronary blood ow, andheart rate) of the simulation model agree with the measurement data takenfrom the patient. The training set was composed of 1,500 data points for eachcontroller, whereas six data sets not used in the training process (600 pointseach) were used as forecasting targets, containing signals that represent specicmorphologies. The HNN and the TDNN architectures were xed to include 1output unit, 8 hidden units, and 7 input units, corresponding to the values x(t),x(t   1), x(t   2), x(t   3), y(t   1), y(t   2) and y(t   3), where x(t) denotesthe current value of the input variable and y(t   1) denotes the value of thecontroller output in the previous time step. All inputs to the HNN were treatedas fuzzy sets and the similarity relation given in Section 2 was used. The rst-order ASLRNN architecture also included 1 output and 8 hidden units, but just2 input units, corresponding to the values x(t) and y(t 1), though in this casethe hidden layer incorporated additional weights for the feed-back connections.In the testing process, the normalized mean square error (in percentage)between the predicted output value, y^(t), and the controller output, y(t), wasused to determine the quality of each of the inferred models. This error is givenby MSE =E[(y(t) y^(t))2]yvar 100% where yvardenotes the variance of y(t).For each CNS controller and neural approach three dierent training trialswere run using a dierent random weight initialization. The HNN was trainedusing a standard genetic algorithm with the following characteristics: binary-coded values, probability of crossover: 0:6, probability of mutation: 0:01, num-ber of individuals: 100, linear scaling with factor: 1:5, selection mechanism:tournament. The algorithm stopped when no improvement was found for thelast 1; 000 generations (typical values were about 5; 000). On the other hand,the TDNN-BP and ASLRNN nets were allotted 3; 000 epochs using a smalllearning rate of  = 0:025 to allow a smooth minimization trajectory. Theseparameters were tuned after some preliminary tests. For each run, the networkyielding the smallest MSE error on the training set during learning was takenas the controller model. The TDNN-AC was trained in only one run and theprocess was stopped when a reasonable error was attained.3.4 ResultsThe nets resulting from the training phase were applied to the training setand to the six test data sets associated with each controller. The normalizedMSE errors for these sets were calculated, together with their averages for thedierent training runs and test sets. The summary of the errors obtained bythe dierent neural approaches is displayed in Table 1.TD-HNN TDNN-BP TDNN-AC ASLRNNTrain. Test Train. Test Train. Test Train. TestHRC 0.11% 0.18% 1.15% 1.52% 0.15% 0.13% 1.63% 1.91%PRC 0.09% 0.12% 0.94% 1.27% 0.26% 0.14% 0.84% 1.10%MCC 0.03% 0.06% 0.81% 1.33% 0.09% 0.08% 0.71% 1.18%VTC 0.03% 0.06% 0.81% 1.33% 0.09% 0.08% 0.71% 1.18%CRC 0.10% 0.11% 0.47% 0.66% 0.03% 0.04% 0.41% 0.53%mean 0.07% 0.11% 0.84% 1.22% 0.12% 0.09% 0.86% 1.18%Table 1: Average normalized MSE errors for the training sets (left) and testsets (right) of the CNS controller models inferred by each neural approach.It is interesting to observe the excellent results yielded by the models in-ferred by both the HNN and the TDNN-AC, especially as compared to theTDNN-BP and ASLRNN, which showed an almost identical prediction per-formance, possibly caused by a short depth of temporal dependencies in themodeled system (i.e. all relevant past information could be included in themoving window that selects the inputs of a TDNN).4 ConclusionsHeterogeneous neural networks have been successfully tested in a signal fore-casting task (learning central nervous system controllers). The learning andgeneralization performance of HNNs are comparable to that of TDNNs trainedwith sophisticated optimization algorithms and better than that of TDNNstrained with backpropagation and RNNs trained with a true gradient-descentalgorithm. However, further experiments addressing this kind of problemsshould be carried out towards a better understanding of their capabilities.References[1] Valdes, J.J., Garca, R., A model for heterogeneous neurons and its use in con-guring neural networks for classication problems, Procs. of IWANN'97, Intl.World Conf. on Articial and Natural Neural Networks. Lecture Notes in Com-puter Science 1240, Springer-Verlag, pp. 237-246.[2] Belanche, Ll., Valdes, J.J., Using Fuzzy Heterogeneous Neural Networks to Learna Model of the Central Nervous System Control. Procs. of EUFIT'98, 6th Euro-pean Congress on Intelligent Techniques and Soft Computing. Aachen, Germany.[3] Gower, J.C., A general coecient of similarity and some of its properties, Bio-metrics, 27, pp. 857-871, 1971.[4] Dubois D., Prade H., Esteva F., Garca P., Godo L., Lopez de Mantaras, R.,Fuzzy set modeling in case-based reasoning. Intl. Journal of Intelligent Systems,1997 (to appear).[5] J. Cueva, R. Alquezar and A. Nebot, "Experimental comparison of fuzzy and neu-ral network techniques in learning models of the central nervous system control",Proc. of EUFIT'97, 5th European Congress on Intell. Tech. and Soft Comput.,Aachen, Germany, pp.1014-1018, September 1997.[6] J. Hertz, A. Krogh and R.G. Palmer, Introduction to the Theory of Neural Com-putation, Addison-Wesley, Redwood City, 1991.[7] A. Lapedes and R. Farber, Nonlinear signal processing using neural networks:prediction and system modeling, Tech. Rep. LA-UR-87-2662, Los Alamos NationalLaboratory, Los Alamos NM, 1987.[8] Ackley, D. A connectionist machine for genetic hillclimbing Kluwer Acad. Press,1987.[9] J.M. Sopena and R. Alquezar, Improvement of learning in recurrent networks bysubstituting the sigmoid activation function. ICANN'94, Proc. Int. Conf. Artif.Neural Networks, Sorrento, Italy, Springer-Verlag, Vol.1, pp.417-420, 1994.[10] J.L. Elman, Finding structure in time. Cogn. Sci. Vol. 14, pp.179-211, 1990.[11] J. Schmidhuber, A xed size storage O(n3) time complexity learning algorithmfor fully recurrent continually running networks, Neural Computation Vol. 4,pp.243-248, 1992.

https://upcommons.upc.edu/bitstream/2117/181370/3/Belanche.pdf

Fuzzy heterogeneous neural networks for signal forecasting

Abstract

Similar works

Full text

Available Versions

UPCommons

UPCommons. Portal del coneixement obert de la UPC