Artificial Neural Networks (ANNs) have the ability of learning and to adapt to new situations by recognizing patterns in previous data. Time Series (TS) (observations ordered in time) often present a high degree of noise which difficults forecasting. Using ANNs for Time Series Forecasting (TSF) may be appealing. However, the main problem with this approach is on the search for the best ANN architecture. Genetic Algorithms (GAs) are suited for problems of combinatorial nature, where other methods seem to fail. Therefore, an integration of ANNs and GAs for TSF, taking the advantages of both methods, may be appealing. ANNs will learn to forecast by back-propagation. Different ANNs architectures will give different forecasts, leading to competition. At the end of the evolutionary process the resulting ANN is expected to return the best possible forecast. It is asserted that the combined strategy exceeded conventional TSF methods on TS of high non-linear degree, particularly for long term forecasts

Cortez, Paulo

Machado, José Manuel

Neves, José

Universidade do Minho: RepositoriUM

An Evolutionary Articial Neural NetworkTime Series Forecasting SystemPaulo Cortez Jose´ Machado Jose´ Nevespaulo@di-ia.uminho.pt jmac@di-ia.uminho.pt jneves@di-ia.uminho.ptDepartamento de Informa´ticaUniversidade do MinhoLargo do Pac¸o, 4709 Braga Codex, PORTUGALVoice: +351(53)604470 Fax: +351(53)604471Keywords: Neural Networks, Genetic Algorithms, TimeSeries.AbstractArtificial Neural Networks (ANNs) have the ability of learn-ing and to adapt to new situations by recognizing patternsin previous data. Time Series (TS) (observations orderedin time) often present a high degree of noise which diffi-cults forecasting. Using ANNs for Time Series Forecasting(TSF) may be appealing. However, the main problem withthis approach is on the search for the best ANN architecture.Genetic Algorithms (GAs) are suited for problems of com-binatorial nature, where other methods seem to fail. There-fore, an integration of ANNs and GAs for TSF, taking theadvantages of both methods, may be appealing. ANNs willlearn to forecast by back-propagation. Different ANNs ar-chitectures will give different forecasts, leading to compe-tition. At the end of the evolutionary process the resultingANN is expected to return the best possible forecast. It isasserted that the combined strategy exceeded conventionalTSF methods on TS of high non-linear degree, particularlyfor long term forecasts.1 IntroductionWith the emerging of open markets competition betweenenterprises will become more and more tough. Winningmeans the need of strategic advantages. However, innovat-ing in an uncertain environment can be disastrous. An or-ganization may be interested in predicting the future basedon previous data, in terms of the variables that rule its prac-tice. One possibility is the use of TSF, based only on previ-ous observations of the same variable, which has the advan-tage of modeling complex systems as black-boxes [8]. Thegoal is to predict the system behavior and not how the sys-tem works. Organizations normally work with two differentkinds of forecasts: short term predictions, few predictionsahead, for current management decisions (eg. dealing withstocks), and long term predictions, for planning (eg. elab-orating budgets) [8]. Conventional TSF methods base theirmodels on TS components such as trend and seasonal ef-fects [11]. Well behaved TS are suited for this kind of ap-proach but fails when TS present strong noise or non-linearcomponents.Optimization algorithms, imitating nature, have be-come popular in both academy and industry. Studies on thenervous system and biological evolution influenced the riseof ANNs and GAs. ANNs have the advantage of learningfrom previous data in order to respond to new situations,specially when one is dealing with incomplete informationor a strong noise component.In principle ANNs can imitate any computable func-tion. This is the main reason why ANNs have been appliedso successfully in many different areas: expert systems,data filtering, computer vision or economy [5]. ANNs seemto be appropriate for classification or approximation prob-lems that are tolerant to some errors, with a lot of availabledata and where well defined rules can not be applied. ANNsseem to be a good strategy for TSF. One drawback of usingANNs is the time required for the search of the best ANN.This is a heavy empirical process, where intuition, heuristicrules or random search are often used.GAs are suited for combinatorial problems, wherethe exhaustion of all possible solutions require enormouscomputational power. GAs can heuristically find solutionswhere other methods seem to fail. Although GAs and ANNsare both search strategies, empirical studies show that they1Table 1: Activation FunctionsName Function Codomainlinear 	sigmoid ﬀﬂﬁ ﬃ! "sigmoid1 #$%ﬁ& '( )* sigmoid2 + )+'( )* tanh ,.-0/1 '2 3! "differ in range [9]. GAs seem to perform a superior globalsearch than ANNs with back-propagation, although back-propagation will reach a better solution [10].This work reports on the integration of ANNs andGAs, a Genetic Algorithm Neural Network (GANN) system[9], leading to TSF. ANNs will learn to forecast by back-propagation and better ANN architectures will be selectedby evolution.2 The Artificial Neural Network Ar-chitectureThere is no known rule that points out to what kind of ANNone must use. The choice is done by looking at the data, tothe problem, and using intuition or experience [5][14].One’s approach for TSF supposes the use of feedfor-ward ANNs. The decision was to use feedforward ANNsfully connected, with bias, without shortcut connections,and with one hidden layer, which makes possible to boundat a certain level the ANN parameters, in order to reducethe vectorial search space. The initial ANN weights wererandomly generated within the range of4#576#5for a nodewith 8 inputs. Standard back-propagation [5] was used forthe ANN training. Table 1 shows the activation functionsused.The ANN topology will be represented by 9;:9;<9;=for an ANN with 9>: input nodes, 9;< hidden nodes and 9;=output nodes [14]. In this work 9?= is always equal to one.Empirical tests indicated that, with the same data, ANNswith one output node gave better forecasts than ANNs with/ output nodes, even for / prediction steps. The ANN re-ceives / previous TS values, for / input nodes, and outputsthe forecasted value. The training cases were built as fol-lows:#!@A@A@A.B C 	B#D)!@A@A@A.BC 	B#@A@A@ C @E@A@3FBﬀ*@A@A@E3FC )Fwhere!@A@A@A.3Frepresent the TS and G the number of theANN’s inputs. The cases are trained in this order due tothe temporal nature of the data. The trained ANN can makeshort or long term predictions: HF(IKJﬂL,ML, )N)FB	!@A@A@A.3F! #OHF(IKJﬂL,ML, )N)FB*@A@A@E3F) .HF3 *DHF(IKJﬂL,ML, )N)FB#*@A@A@E3F) .HF. #OHF0@A@A@ *PHF(IJﬂL,ML, ) *PBHF	*@E@A@A *PQRHF!whereJﬂL,ML, 3*@A@E@A	BScaters for the output function ofthe ANN, and :H Tthe forecast in the U period to V periodsahead of U .One problem that arises with back-propagation train-ing is overfitting: after some training epochs the ANN startsto loose generalization. To overcome this situation, earlystopping was considered [14]. The training data was di-vided into the training set (for the ANN learning) and thevalidation set (to test the ANN generalization capacity). Bydefault the validation set is 10% of the training data, sincethe use of the last (recent) values in the training set im-proves the forecast.The best ANN architecture for short and long termforecasts tends to differ. On long term forecasts ANNs seemto be more affected by overfitting. Thus a special valida-tion test, Specific Mean Square Error (SMSE), that takesinto consideration the number of forecasts ahead, was im-plemented:WYX[ZPH TI]\S^N_ `ﬀ*`a0^ﬁ$bR]\b_ `S!`Nabﬁb.cdcdc '\SeS_ `Oﬀ*`a0e!ﬁ$bPX[WYXfZPgIihkjle`mjlon"leQpgq3reS_ `Bwhere M is the last period of the TS, G the number of valida-tion cases and / the number of forecasting steps. WYX[ZPH Tis the error measure for / forecasting steps from U , and / isan user defined parameter.3 Combining Genetic Algorithmswith Artificial Neural NetworksWhen using feedforward ANNs a number of parametershave to be set before any training: the ANN topology, theconnections between nodes, the activation function, thetraining algorithm and its parameters. The search for thebest ANN goes through a long empirical process, of com-binatorial nature. Some rules are used in order to simplifythe process: the use of only one hidden layer; choosing atraining algorithm that fits to the problem (generally of theback-propagation family); the use of the standard parame-ters for the algorithm and the use of fully-connected ANNs(which avoids the need to define the ANN’s connections).To set the remaining of the parameters one can use randomsearch or GAs. The last ones are known of being more ef-fective and efficient [6].GANNs systems optimize ANNs in two ways: evo-lution by the GA and learning by back-propagation [10][7]. The back-propagation learning guides the evolution-ary search. Chromosomes (encoded ANNs) near optimumwill share bit patterns, permitting the GA to exploit them byhiperplane sampling. GANNs systems use to present seri-ous drawbacks, in particular when long genomes imply theuse of large ANNs [9]. That is not the situation in this work,once TSF may be efficiently modelled by small ANNs [4].When setting the GAs parameters one has to com-promise between the optimum ANN and the computationaltime required. Therefore, some hard decisions were to bemade, namely: the use of a population of 20 individuals;the selection by the classical fitness-based roulette-wheel;one point crossover with a crossover rate of 1 and a muta-tion rate of 0.02. The random seed for the initial populationwas 12345. The fitness function uses the SMSE for the val-idation cases: VA,./ Iq3pgq3r.The chromosome with the lower SMSE will automati-cally survive for the next generation. This purification pro-cess accelerates the search and also warranties that if a op-timum ANN is reached then it will never be lost. There are6 (six) factors that affect a good forecast: the number ofinput nodes, the learning rate, the activation function, thenumber of hidden nodes, the scaling constant and the ran-dom weights initialization seed. Encoding a chromosome:the first three factors, the crticial ones, were put at the leftof the chromosome in a direct encoding using base 2 graycodes.Input Nodes+Function+Rate Hidden Nodes+Figure 1: Neural Network EncodingThe number of input and hidden nodes were set withinthe range	* oallowing a 4 bits binary encoding. Thelearning rate was encoded in 3 bits taking values from theset ﬃ@A )ﬃ	@	  .ﬃ@ .ﬃ@. The 5 (five) activation func-tions were encoded in 3 bits, which allows for 3 do not carevalues, implying that new bits have to be validated uponcreation. The scaling constant is set by the user (which getsthe highest TS value as an indication). The main reason forthis is that forecasts near optimum are archived by scalingconstants that are near the forecasting values. Therefore, ifa series presents a growing trend then the GA would find ascaling value very close to the validation data. That wouldresult in forecasts bellow real values. The last parameter(random weights initialization seed) is not encoded sinceits influence in the forecast is random (by definition).Table 2: GANN’s results for HFforecastingS Size   Topology F Rpgq3r1 100 81 10 (sigmoid1 0.7 138.52 130 106 12#  # (sigmoid2 0.1 16560183 144 119 12 (sigmoid 0.7 30374 369 339 10 (sigmoid 0.8 62.4The whole system was developed using the UNIX op-erating system, the SICStus Prolog and C languages, with aX-Windows interface, and implemented on a TCP/IP basednetwork of workstations [2][13][12]. A X-Windows inter-face was built using the SICStus gmlib library [1].Before building a model the system checks for series’randomness. The test uses the autocorrelation coefficient,which gives a measure of the correlation between a seriesand itself, lagged  periods:BIhPBﬁﬀ ﬀ SﬁﬀB ﬀ hPﬂﬀ ﬀ All autocorrelations values should theoretically be zero fora random series. In practice a series is considered to berandom if all  lags autocorrelation values are within therange:2 3@ﬃ "! #/$B$ 3@ ﬃ%&! #/If a series is random them, instead of running the GA, thesystem returns the average of the TS (which is the only pos-sible forecast in this case).4 The System ResultsThe results for short term forecasting are shown in table 2,where X stands for X    V ' . The parameters are: ( for thenumber of series values in the training set and ) for thenumber of forecasts. Optimum ANN parameters are: * forthe function and + for the learning rate.The system results were compared with the Holt-Winters [11] and ARIMA [3] methods (table 3). For seriesTable 3: Comparing GANN’s MSE results with conven-tional methods for H: forecasting.S GANN ARIMA Holt-Winters1 138.5 163.8/2562 165601813481533 30372704 62.4 67.2 103.6 andthe system outperforms the conventional methods(ARIMA and Holt-Winters). In series	andthe Holt-Winters model performs best results. All the restrictionsTable 4: Comparing the X[WYXfZ(or WYX[Z ) validation testwith the X[WYX[ZPtest for long term forecastsS   Topology F R  	 1 1  sigmoid1 0.7 90 2861 8  sigmoid2 0.6 90 235.94 1  sigmoid 0.8 359 133.74 10  sigmoid 0.8 359 105.5Table 5: The WYX[ZﬀHFfor each forecasting method.S GANN ARIMA Holt-Winters1 235.9 262.4/278.94 105.5377.9imposed on the GANN system are probably the main rea-son for this behavior.Table 4 shows the validity of the SMSE test on longterm forecasts. Note that the best ANNs for long term fore-casts are slightly different than the short term ones. Series2 and 3 don’t appear in this table since the short term sys-tem errors were higher than long term ones obtained withconventional methods. Short terms errors are smaller thanthe long ones, so there was no need to waste computationalpower. For long term forecasts the system results are evenbetter (see table 5).5 Conclusions and Future WorkGANN’s systems can be very attractive on TSF for time se-ries with a high non-linear degree, specially for long termforecasts. However there are some drawbacks: a high com-putational cost. One advantage of this system is that itworks with a minimum of human intervention. For futurework one intends to explore different ANN architecture, toexperiment on other genetic algorithm parameters and op-erators, and develop and apply software agent-based tech-nology to the GANN approach.References[1] J. Almgren, S. Anderson, M. Carlsson, L.Floyd,S. Haridi, C. Frisk, H.Nilsson, and J.Sundberg. SICS-tus Prolog User’s Manual. Swedish Institute of Com-puter Science, Sweden., 1993.[2] O. Belo and J. Neves. A Prolog Implementation of aDistributed Computing Environment for Multi-AgentSystems Based Applications. In Proceedings of TheThird International Conference on Practical Applica-tion of Prolog, PAP’95, Paris, France, 1995.[3] G. Box and G. Jenkins. Time Series Analysis: Fore-casting and Control. Holden Day, San Francisco,USA, 1976.[4] P. Cortez, M. Rocha, J. Machado, and J. Neves. ANeural Network Based Forecasting System. In Pro-ceedings of ICNN’95 - International Conference onNeural Networks, Perth, Western Australia, Novem-ber 1995.[5] S. Gallant. Neural Network Learning and Expert Sys-tems. MIT Press, Cambridge, USA, 1993.[6] D. Goldberg. Genetic Algorithms in Search, Opti-mization and Machine Learning. Addison-WesleyPublishing Company, Inc., NY, USA, 1989.[7] F. Gruau and D. Whitley. Adding learning to thecellular development of neural networks: Evolutionand the baldwin effect. Evolutionary Computation,,3(1):213–233, MIT Pres, 1993. MIT Press.[8] J. Hanke and A. Reitsch. A Business Forecasting.Allyn and Bancon Publishing Company Inc., Mas-sachussetts, USA, 1989.[9] K. Hiroaki. Empirical Studies on the Speed of Con-vergence of Neural Network Training using GeneticAlgorithms. Eighth National Conference on ArtificialInteligence Vol. II, pp 788-795, AAAI, MIT Press.,1990.[10] P. Koenh. Combining Genetic Algorithms and NeuralNetworks. Thesis for master science degree, Univer-sity of Tennessee, Knoxville, USA., 1994.[11] S. Makridakis, S. Weelwright, and V. McGee. Fore-casting: Methods and Applications. John Wiley &Sons, New York, USA, 1978.[12] J. Neves, J. Machado, L. Costa, and P. Cortez. A Soft-ware Agent Distributed System for Dynamic LoadBalancing. In Proceedings of ESM96, European Sim-ulation Multi-Conference, Modelling and Simulation1996, Budapest, Hungary, June 1996.[13] J. Neves, M. Santos, and V. Alves. An Adaptable andDynamic Architecture for Distributed Problem Solv-ing Based on the Blackboard Paradigm. In Proceed-ings of the 7th Australian Joint Conference on Ar-tificial Intelligence, pages 378–385, Armidale, NewSouth Wales, Australia, 1994.[14] L. Prechelt. PROBEN1 - A Set of Neural NetworkBenchmark Problems and Benchmarking Rules. Re-search Report, Fakulta˝t fu˝r Informatik, Universita˝tKarlsruhe Germany, 1994.

An evolutionary artificial neural network time series forecasting system

http://repositorium.sdum.uminho.pt/bitstream/1822/2195/1/havai.pdf

An evolutionary artificial neural network time series forecasting system

Abstract

Similar works

Full text

Available Versions

Universidade do Minho: RepositoriUM