Aswolinskiy W, Reinhart F, Steil JJ. Impact of Regularization on the Model Space for Time Series Classification. In:  New Challenges in Neural Computation (NC2). 2015: 49-56

Aswolinskiy, Witali

Reinhart, Felix

Steil, Jochen J.

Publications at Bielefeld University

Impact of Regularization on the Model Space forTime Series ClassificationWitali Aswolinskiy, Rene´ Felix Reinhart and Jochen SteilResearch Institute for Cognition and Robotics - CoR-LabUniversita¨tsstraße 25, 33615 Bielefeld, Germany{waswolinskiy,freinhart,jsteil}@cor-lab.uni-bielefeld.dehttp://www.springer.com/lncsAbstract. Time series classification is an active research field and ap-plicable in many domains, e.g. speech and gesture recognition. A recentapproach to classify time series is based on modelling each time series byan Echo State Network and then to classify the time series in the readoutweight or model space of these networks. In this paper, we investigatethe effect of Echo State Network regularization on the model space. Theresults show that regularization has a strong impact on the model spacestructure and the separability of the time series in the model space.Keywords: model space, echo state networks, reservoir computing, timeseries classification, time series clustering, regularization1 IntroductionIn time series classification, a label is assigned to a sequence of data points.One main challenge is that the time series can have different lengths. A recentapproach is learning in the model space: For each time series a model is trainedand the model’s parameters are used as features in a consecutive classificationstage [2]. Typically, the models are trained to minimize the one-step-ahead pre-diction error on the time series. The number of model parameters is independentof the time series length, which allows to employ any feature based classifier frommachine learning theory in the classification stage.Echo State Networks (ESNs) [5] are promising candidates for models in thiscontext, because they offer temporal integration of the input, nonlinear compu-tation and quick training. Learning in the model space of ESNs was evaluatedfor time series classification [2] and clustering [3] with promising results. Here,we investigate the influence of regularization during training of ESNs on classi-fication performance and model space structure.Regularization is a technique to reduce model complexity in order to preventoverfitting and comes in the context of ESNs often in form of ridge regression.In ridge regression not only the sum of squared residuals is minimized, butalso the sum of the squared regression coefficients. This shrinks the coefficientsand thereby reduces their variance. Without regularization, and non-orthogonaldata vectors, the coefficients can have a large magnitude and be unstable - minorchanges in the data can lead to major changes of the coefficients [4, 10]. This isespecially important in the context of learning in the model space, where thesecoefficients form the model space. For a better understanding of the model spacewe investigate the effects of different forms of regularization on the model space.We show that L2 but not L1 regularization achieves good classification results,and that the modelling error is not necessarily predictive for the classificationperformance in the model space.2 Learning in the Model Space of Echo State NetworksA classic ESN architecture consists of a reservoir with recurrently connectedneurons and a linear regression readout. The reservoir states x and the readoutsy are updated according tox(k) = (1− λ)x(k−1) + λf(W recx(k−1) +W inu(k)) (1)y(k) = W outx(k), (2)where λ is the leak rate, f is the activation function, e.g. tangens hyperbolicus,Wrec the recurrent weight matrix, Win the weight matrix from the inputs tothe neurons and Wout the weight matrix from the neurons to the readouts y.Only Wout is trained with linear regression - the other weights are initializedrandomly and remain fixed.For classification with ESNs, input time series are fed into the reservoir andthe readout is typically trained on the reservoir states with linear regression topredict the class label for each step of the time series. In contrast, in learning inthe model space of ESNs, the ESNs are an intermediate step to create a time-independent representation of the time series. Let a dataset consist of N discretetime series ui, i = 1...N with varying lengths Ki: ui(0), ...,ui(k), ...,ui(Ki). Foreach time series ui, an ESN is trained to predict from the previous step ui(k)the next step ui(k+1) in the time series. The ESNs are trained independently,but share the same reservoir parameters Win and Wrec in order to create acoherent model space. The prediction errorE(W outi ) =1Ki−1Ki∑k=2(ui(k)−W outi xi(k−1))2 + α∥∥W outi ∥∥L (3)for time series i is minimized, where α is the regularization strength and L is theregularization norm. The resulting readout weights W outi form the model space,where the classification takes place.In the model space, arbitrary classification algorithms can be used. Thisapproach will be denoted here as model space learning (MSL) and is visualizedin Fig. 1. The example shows the transformation of the data - here noisy sum ofsine waves - via regression on the reservoir activations to the readout weights.In this example, the time series and hence the reservoir activations are 100 stepslong. The dimensionality of the model space depends on the dimensionality ofFig. 1. Learning in the Model Space of ESNs. In the upper part of the diagram, noisysine waves are shown, which are fed into the reservoir. For each time series a readoutis trained to predict the next input. In the lower part of the diagram, the time series,the reservoir echos and the readout weights are visualized with matching colors. Thesignal, reservoir activation and readout weight matrices were projected to a plane usingPCA. In the model space, the time series of different classes are easily separable.the signal and the number of reservoir neurons. For signals with d dimensionsand reservoirs with n neurons the model space dimensionality is d ·n or d ·(n+1)if a regression intercept is used. For comparison, in the lower part of Fig. 1, eachtime series is represented as a point in signal space, reservoir space (the spaceof reservoir neuron activations) and model space. For visualization purposes,the data was projected to two dimensions via principal component analysis. Inthis example, time series from different classes can be easily separated in themodel space.3 Effect of Regularization on the Model SpaceIn MSL regularization can occur at two stages. First, while training the ESNmodels and second, while training a classifier in the model space where the ESNreadout weights serve as features. Here, we focus on the effect of regularizingthe ESN readout weights W out. The regularization strength for ESN training isgiven by α in Eq.(3).3.1 L2 and L1 RegularizationIn L2-regularization or ridge regression the sum of squared residuals is mini-mized. In L1-regularization or Lasso regression the magnitude of the weights isminimized [9]. This results in a more sparse weight distribution. We evaluatedthe effect of regularization in the ESN on the model space on two multivari-ate datasets from the the UCI repository [7]. Since a direct analysis of modelspace properties is difficult, we assess the model space structure indirectly byevaluating pattern separability in the classification stage.The Australian Sign Language Signs (AUSLAN) dataset consists of 2565samples (27 repetitions x 95 signs), recorded from a native signer using handposition trackers. The sequences are between 45 and 136 steps long and have 22input dimensions.The Spoken Arabic Digit (SAD) dataset [6] consists of 8800 samples (10digits x 10 repetitions x 88 speakers) with 13 input dimensions - Mel FrequencyCepstral Coefficients (MFCCs). The sequences are between 4 to 93 steps long.Classification error rates were obtained via five times repeated random sub-sampling, also known as Monte Carlo cross validation [8]. In the AUSLANdataset, from the 2565 samples, in each fold, 600 randomly selected sampleswere used for training and the rest for testing. In the SAD dataset the split washalf-and-half.After transforming the time series to model parameters by training the ESNs,in the model space, two classifiers were evaluated: Ridge classifier and supportvector machine (SVM) with radial basis function kernel. The ridge classifier is alinear classifier trained with ridge regression: The K class labels are encoded ina 1-of-K scheme and the regressed scores transformed to estimated class labelswith the winner-takes-all method. Since the random initialization of the reservoirweights Win and Wrec affects the performance, only the results of the best outof five reservoirs were used. In order to mitigate the influence of the classifierparameters on the model space analysis, for each α value and classifier, severalclassifier parameter values were evaluated and from these evaluations the lowesterror rate was chosen. The used ESN and classifier parameters are listed inthe Appendix.L2 Regularization: The classification error rates for the datasets at differentregularization strengths are depicted in Fig. 2 (A,B). The diagram can be dividedin three areas: Overfitting on the left, underfitting on the right and the lowesterror rates in the middle. Interestingly, the SVM classifier does not achieve lowererror rates than the linear ridge classifier. This suggests, that the consideredclassification tasks can be solved linearly in the model space.Fig. 2. Classification error rates in the AUSLAN and SAD datasets with ridge (A,B)and Lasso (C,D) regression for different ESN regularization strengths α and two classi-fiers. The average error rate of 5-fold Monte Carlo cross validation is shown. The verysmall standard deviations are indicated by error bars.L1 Regularization: Fig. 2 (C,D) shows the classification error rates for Lassoregression. Compared to Fig. 2 (A,B) the performance is considerably worse.With ridge regression the error rate in AUSLAN was 3.31±0.4% and in SAD1.5±0.5%. With Lasso regression, the lowest error rate in AUSLAN was 16.6±0.4% and in SAD 5.0±0.6%. These results indicate that sparseness in the modelspace is detrimental for classification performance. A possible explanation is thenature of L1-regularization: The explanatory variables mostly correlated withthe target variable are selected. In MSL, Lasso is executed for each time seriesindependently and thus, for similar but slightly different time series of the sameclass, different reservoir neurons may be selected. This effect emphasizes thedifference between the samples in the model space rather than their similarity.A manual inspection of the model space validated the presence of this effect.3.2 Difficulty of the TaskIn order to investigate, whether there is dependence between regularization andthe difficulty of the classification task, we devised a synthetic dataset. The taskis to differentiate between time seriesu(k) = sin(0.2k) + sin(0.311k) + sin(0.42k)u(k) = sin(0.2k) + (1−∆) sin(0.311k) + (1 +∆) sin(0.42k) .The difficulty of the task is controlled with the parameter ∆: The smaller thevalue of ∆, the more similar are the respective time series and thus the moredifficult the task. The dataset consists of 200 time series of length 100 created bygenerating 10,000 steps of each sequence and dividing them in 100 parts each.Half of the time series were used for training and half for testing.Fig. 3 shows how the model prediction error (A) and the classification error inthe model space (B) depend on the regularization strength α and task simplicity∆. The prediction error is the normalized root mean square error (NRMSE)computed for predicting the next input during training of the ESN models andwas averaged over all models. With ∆=0, the two time series classes are thesame and thus not separable (see first column in Fig. 3B). With increasing ∆the difference between the time series classes increases and the classification errorrate decreases. Weak regularization with α<10−5 leads to overfitting: Low modelprediction error, but high classification error in the model space (see the upperparts of Fig. 3A and 3B). Regularizing with α>10 leads to underfitting duringmodel training, but the classification error rates remain low until α exceeds 107(see the lower parts of Fig. 3A and 3B). Regularizing with 10<α<103 achievesthe best results. Noteworthy is that no correlation between the prediction errorand classification error rate is observable. This means that for pattern separationin the model space, it is less important how exact a model fits the data.3.3 Noise and OverfittingThe main goal of regularization is to prevent the learning of noise which leads tosmaller errors on the training data, but decreases the generalization capabilityand therefore causes larger errors on the test data. As the amount of noise inthe UCI datasets is unknown, we use again a synthetic dataset to study theeffect of regularization on the model space in the presence of noise. The task isto differentiate between time series of the formu(k) = sin(0.2x) + sin(0.311x) + sin(0.42x) + u(k) = sin(0.2x) + 0.7 sin(0.311x) + 1.3 sin(0.42x) +  .This corresponds to the synthetic dataset from the previous section with ∆=0.3and added noise. During the simulations, the standard deviation σ of the Gaus-sian distributed noise  was varied. Fig. 3C and 3D show the the average ofthe prediction error and the classification error rate for different regularizationstrengths α and noise levels σ. Best results are achieved with α around 1. Heretoo, no correlation between the model prediction error and the classificationperformance in the model space can be observed. Low classification rates arepossible even with large model prediction errors, e.g. a NRMSE of 1.0.Fig. 3. Average NRMSE for predicting the next value in a time series with the ESNs(A,C) and classification error rate in the model space (B,D). Dark cells denote highvalues and bright cells low values.4 ConclusionIn this paper we investigated the effect of regularization during training of ESNson the classification performance in the readout weight space (i.e. model space)of these ESNs. Regularization improved the performance for more difficult tasksas well as in the presence of noise. L2 regularization performed considerablybetter then L1 regularization. In conclusion, L2 regularization is more suitablefor MSL, and the model prediction error can not be used to estimate the expectedclassification performance in the model space.5 AppendixPreprocessing: All datasets were scaled to the range [−1/d, 1/d].Reservoir Parameters: n: Number of reservoir neurons, Input Scaling: Scal-ing of the weights from the input to the reservoir; Rec. Connectivity: Densityof the recurrent weight matrix Wrec, Rec. Scaling: Spectral radius of the re-current weights W rec, λ: Leak rate of the neurons, Bias Scaling: Scaling of theneuron biases.– Synthetic: n: 50, Input Scaling: 1, Rec. Connectivity: 10%, Rec. Scaling: 0.9,λ: 1, Bias Scaling: 0.1– AUSLAN: n: 100, Input Scaling: 1, Rec. Connectivity: 10%, Rec. Scaling:0.9, λ: 1, Bias Scaling: 0.1– SAD: n: 50, Input Scaling: 2, Rec. Connectivity: 10%, Rec. Scaling: 0.9, λ:0.7, Bias Scaling: 0.1Classifier parameters: For both synthetic datasets, a linear regression classi-fier without regularization was used. In AUSLAN and SAD:– Ridge Classifier: Ridge from 1e-5 to 1e+5 in logarithmic scale.– SVM Classifier [1]: Penalty C from 0.1 to 10000 in logarithmic scale, Kernelcoefficient gamma from 0.01 to 100 in logarithmic scale.Acknowledgments. This project is funded by the German Federal Ministry ofEducation and Research (BMBF) within the Leading-Edge Cluster Competition“it’s OWL” (intelligent technical systems OstWestfalenLippe) and managed bythe Project Management Agency Karlsruhe (PTKA). The authors are responsi-ble for the contents of this publication.References1. Chang, C.C., Lin, C.J.: LIBSVM: A library for support vector machines. ACMTransactions on Intelligent Systems and Technology 2, 27:1–27:27 (2011), softwareavailable at http://www.csie.ntu.edu.tw/~cjlin/libsvm2. Chen, H., Tang, F., Tino, P., Yao, X.: Model-based kernel for efficient time seriesanalysis. In: ACM SIGKDD International Conference on Knowledge Discovery andData Mining. pp. 392–400 (2013)3. Chen, H., Tino, P., Rodan, A., Yao, X.: Learning in the model space for cognitivefault diagnosis. IEEE Transactions on Neural Networks and Learning Systems25(1), 124–136 (2014)4. Hoerl, A.E., Kennard, R.W.: Ridge regression: applications to nonorthogonal prob-lems. Technometrics 12(1), 69–82 (1970)5. Jaeger, H.: The “echo state” approach to analysing and training recurrent neuralnetworks-with an erratum note. GMD Technical Report 148, 34 (2001)6. Kadous, M.W.: Temporal classification: Extending the classification paradigm tomultivariate time series. Ph.D. thesis, The University of New South Wales (2002)7. Lichman, M.: UCI machine learning repository (2013), http://archive.ics.uci.edu/ml8. Picard, R.R., Cook, R.D.: Cross-validation of regression models. Journal of theAmerican Statistical Association 79(387), 575–583 (1984)9. Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of theRoyal Statistical Society. Series B (Methodological) pp. 267–288 (1996)10. Vinod, H.D.: A survey of ridge regression and related techniques for improvementsover ordinary least squares. Review of Economics and Statistics pp. 121–131 (1978)

Impact of Regularization on the Model Space for Time Series Classification

https://pub.uni-bielefeld.de/download/2900720/2900969

Impact of Regularization on the Model Space for Time Series Classification

Abstract

Similar works

Full text

Available Versions

Publications at Bielefeld University