Function Approximation With Multilayered Perceptrons Using L1 Criterion

Kaedah ralat kuasa dua terkecil atau kaedah kriteria L2 biasanya digunakan bagi persoalan penghampiran fungsian dan pengitlakan di dalam algoritma perambatan balik ralat. Tujuan kajian ini adalah untuk mempersembahkan suatu kriteria ralat mutlak terkecil bagi perambatan balik sigmoid selain daripada kriteria ralat kuasa dua terkecil yang biasa digunakan. Kami membentangkan struktur fungsi ralat untuk diminimumkan serta hasil pembezaan terhadap pemberat yang akan dikemaskinikan. Tumpuan ·kajian ini ialah terhadap model perseptron multilapisan yang mempunyai satu lapisan tersembunyi tetapi perlaksanaannya boleh dilanjutkan kepada model yang mempunyai dua atau lebih lapisan tersembunyi. The least squares error or L2 criterion approach has been commonly used in functional approximation and generalization in the error backpropagation algorithm. The purpose of this study is to present an absolute error criterion for the sigmoidal backpropagatioll I rather than the usual least squares error criterion. We present the structure of the error function to be minimized and its derivatives with respect to the weights to be updated. The focus in the study is on the single hidden layer multilayer perceptron (MLP) but the implementation may be extended to include two or more hidden layers

Model combination in neural-based forecasting

This paper discusses different ways of combining neural predictive models or neural-based forecasts. The proposed approaches consider Gaussian radial basis function networks, which can be efficiently identified and estimated through recursive/adaptive methods. The usual framework for linearly combining estimates from different models is extended, to cope with the case where the forecasting errors from those models are correlated. A prefiltering methodology is pro posed, addressing the problems raised by heavily nonstationary time series. Moreover, the paper discusses two approaches for decision-making from forecasting models: either inferring decisions from combined predictive estimates, or combining prescriptive solutions derived from different forecasting models.info:eu-repo/semantics/publishedVersio

Implementation of gaussian process models for non-linear system identification

This thesis is concerned with investigating the use of Gaussian Process (GP) models for the identification of nonlinear dynamic systems. The Gaussian Process model is a non-parametric approach to system identification where the model of the underlying system is to be identified through the application of Bayesian analysis to empirical data. The GP modelling approach has been proposed as an alternative to more conventional methods of system identification due to a number of attractive features. In particular, the Bayesian probabilistic framework employed by the GP model has been shown to have potential in tackling the problems found in the optimisation of complex nonlinear models such as those based on multiple model or neural network structures. Furthermore, due to this probabilistic framework, the predictions made by the GP model are probability distributions composed of mean and variance components. This is in contrast to more conventional methods where a predictive point estimate is typically the output of the model. This additional variance component of the model output has been shown to be of potential use in model-predictive or adaptive control implementations. A further property that is of potential interest to those working on system identification problems is that the GP model has been shown to be particularly effective in identifying models from sparse datasets. Therefore, the GP model has been proposed for the identification of models in off-equilibrium regions of operating space, where more established methods might struggle due to a lack of data. The majority of the existing research into modelling with GPs has concentrated on detailing the mathematical methodology and theoretical possibilities of the approach. Furthermore, much of this research has focused on the application of the method toward statistics and machine learning problems. This thesis investigates the use of the GP model for identifying nonlinear dynamic systems from an engineering perspective. In particular, it is the implementation aspects of the GP model that are the main focus of this work. Due to its non-parametric nature, the GP model may also be considered a ‘black-box’ method as the identification process relies almost exclusively on empirical data, and not on prior knowledge of the system. As a result, the methods used to collect and process this data are of great importance, and the experimental design and data pre-processing aspects of the system identification procedure are investigated in detail. Therefore, in the research presented here the inclusion of prior system knowledge into the overall modelling procedure is shown to be an invaluable asset in improving the overall performance of the GP model. In previous research, the computational implementation of the GP modelling approach has been shown to become problematic for applications where the size of training dataset is large (i.e. one thousand or more points). This is due to the requirement in the GP modelling approach for repeated inversion of a covariance matrix whose size is dictated by the number of points included in the training dataset. Therefore, in order to maintain the computational viability of the approach, a number of different strategies have been proposed to lessen the computational burden. Many of these methods seek to make the covariance matrix sparse through the selection of a subset of existing training data. However, instead of operating on an existing training dataset, in this thesis an alternative approach is proposed where the training dataset is specifically designed to be as small as possible whilst still containing as much information. In order to achieve this goal of improving the ‘efficiency’ of the training dataset, the basis of the experimental design involves adopting a more deterministic approach to exciting the system, rather than the more common random excitation approach used for the identification of black-box models. This strategy is made possible through the active use of prior knowledge of the system. The implementation of the GP modelling approach has been demonstrated on a range of simulated and real-world examples. The simulated examples investigated include both static and dynamic systems. The GP model is then applied to two laboratory-scale nonlinear systems: a Coupled Tanks system where the volume of liquid in the second tank must be predicted, and a Heat Transfer system where the temperature of the airflow along a tube must be predicted. Further extensions to the GP model are also investigated including the propagation of uncertainty from one prediction to the next, the application of sparse matrix methods, and also the use of derivative observations. A feature of the application of GP modelling approach to nonlinear system identification problems is the reliance on the squared exponential covariance function. In this thesis the benefits and limitations of this particular covariance function are made clear, and the use of alternative covariance functions and ‘mixed-model’ implementations is also discussed

Evolutionary model type selection for global surrogate modeling

Due to the scale and computational complexity of currently used simulation codes, global surrogate (metamodels) models have become indispensable tools for exploring and understanding the design space. Due to their compact formulation they are cheap to evaluate and thus readily facilitate visualization, design space exploration, rapid prototyping, and sensitivity analysis. They can also be used as accurate building blocks in design packages or larger simulation environments. Consequently, there is great interest in techniques that facilitate the construction of such approximation models while minimizing the computational cost and maximizing model accuracy. Many surrogate model types exist ( Support Vector Machines, Kriging, Neural Networks, etc.) but no type is optimal in all circumstances. Nor is there any hard theory available that can help make this choice. In this paper we present an automatic approach to the model type selection problem. We describe an adaptive global surrogate modeling environment with adaptive sampling, driven by speciated evolution. Different model types are evolved cooperatively using a Genetic Algorithm ( heterogeneous evolution) and compete to approximate the iteratively selected data. In this way the optimal model type and complexity for a given data set or simulation code can be dynamically determined. Its utility and performance is demonstrated on a number of problems where it outperforms traditional sequential execution of each model type

U-model based adaptive internal model control for tracking of nonlinear dynamic plants

We present a technique to infer lower bounds on the worst-case runtime complexity of integer programs, where in contrast to earlier work, our approach is not restricted to tail-recursion. Our technique constructs symbolic representations of program executions using a framework for iterative, under-approximating program simplification. The core of this simplification is a method for (under-approximating) program acceleration based on recurrence solving and a variation of ranking functions. Afterwards, we deduce asymptotic lower bounds from the resulting simplified programs using a special-purpose calculus and an SMT encoding. We implemented our technique in our tool LoAT and show that it infers non-trivial lower bounds for a large class of examples

Dynamic non-linear system modelling using wavelet-based soft computing techniques

The enormous number of complex systems results in the necessity of high-level and cost-efficient modelling structures for the operators and system designers. Model-based approaches offer a very challenging way to integrate a priori knowledge into the procedure. Soft computing based models in particular, can successfully be applied in cases of highly nonlinear problems. A further reason for dealing with so called soft computational model based techniques is that in real-world cases, many times only partial, uncertain and/or inaccurate data is available. Wavelet-Based soft computing techniques are considered, as one of the latest trends in system identification/modelling. This thesis provides a comprehensive synopsis of the main wavelet-based approaches to model the non-linear dynamical systems in real world problems in conjunction with possible twists and novelties aiming for more accurate and less complex modelling structure. Initially, an on-line structure and parameter design has been considered in an adaptive Neuro- Fuzzy (NF) scheme. The problem of redundant membership functions and consequently fuzzy rules is circumvented by applying an adaptive structure. The growth of a special type of Fungus (Monascus ruber van Tieghem) is examined against several other approaches for further justification of the proposed methodology. By extending the line of research, two Morlet Wavelet Neural Network (WNN) structures have been introduced. Increasing the accuracy and decreasing the computational cost are both the primary targets of proposed novelties. Modifying the synoptic weights by replacing them with Linear Combination Weights (LCW) and also imposing a Hybrid Learning Algorithm (HLA) comprising of Gradient Descent (GD) and Recursive Least Square (RLS), are the tools utilised for the above challenges. These two models differ from the point of view of structure while they share the same HLA scheme. The second approach contains an additional Multiplication layer, plus its hidden layer contains several sub-WNNs for each input dimension. The practical superiority of these extensions is demonstrated by simulation and experimental results on real non-linear dynamic system; Listeria Monocytogenes survival curves in Ultra-High Temperature (UHT) whole milk, and consolidated with comprehensive comparison with other suggested schemes. At the next stage, the extended clustering-based fuzzy version of the proposed WNN schemes, is presented as the ultimate structure in this thesis. The proposed Fuzzy Wavelet Neural network (FWNN) benefitted from Gaussian Mixture Models (GMMs) clustering feature, updated by a modified Expectation-Maximization (EM) algorithm. One of the main aims of this thesis is to illustrate how the GMM-EM scheme could be used not only for detecting useful knowledge from the data by building accurate regression, but also for the identification of complex systems. The structure of FWNN is based on the basis of fuzzy rules including wavelet functions in the consequent parts of rules. In order to improve the function approximation accuracy and general capability of the FWNN system, an efficient hybrid learning approach is used to adjust the parameters of dilation, translation, weights, and membership. Extended Kalman Filter (EKF) is employed for wavelet parameters adjustment together with Weighted Least Square (WLS) which is dedicated for the Linear Combination Weights fine-tuning. The results of a real-world application of Short Time Load Forecasting (STLF) further re-enforced the plausibility of the above technique

Regional Forecasting with Support Vector Regressions: The Case of Spain

This study attempts to assess the forecasting accuracy of Support Vector Regression (SVR) with regard to other Artificial Intelligence techniques based on statistical learning. We use two different neural networks and three SVR models that differ by the type of kernel used. We focus on international tourism demand to all seventeen regions of Spain. The SVR with a Gaussian kernel shows the best forecasting performance. The best predictions are obtained for longer forecast horizons, which suggest the suitability of machine learning techniques for medium and long term forecasting

Quantitative Analyses on Non-Linearities in Financial Markets

"The brief market plunge was just a small indicator of how complex and chaotic, in the formal sense, these systems have become. Our nancial system is so complicated and so interactive [...]. What happened in the stock market is just a little example of how things can cascade or how technology can interact with market panic" (Ben Bernanke, IHT, May 17, 2010) One of the most important issues in economics is modeling and fore- casting the uctuations that characterize both nancial and real mar- kets, such as interest rates, commodities and stock prices, output growth, unemployment, or exchange rate. There are mainly two op- posite views concerning these economic uctuations. According to the rst one, which was the predominant thought in the 1930s, the economic system is mainly linear and stable, only randomly hit by exogenous shocks. Ragnar Frisch, Eugen Slutsky and Jan Tinbergen, to cite a few, are important exponents of this view, and they demon- strated that the uctuations observed in the real business cycle may be produced in a stable linear system subject to an external sequence of random shocks. This view has been criticized starting from the 1940s and the 1950s, since it was not able to provide a strong eco- nomic explanation of observed uctuations. Richard Goodwin,John Hicks and Nicholas Kaldor introduced a nonlinear view of the econ- omy, showing that even in absence of external shocks, uctuations might arise. The economists then suggested an alternative within the exogenous approach, at rst by using the stochastic real busi- ness cycle models (Finn E. Kidland and Edward C. Prescott, 1982) and, more recently, by the adoption of the New Keynesian Dynamic Stochastic General Equilibrium (DSGE) models, very adopted from the most important institutions and central banks. These models, however, have also been criticized for the assumption of the rational- ity of agents' behaviour, since rational expectations have been found to be systematically wrong in the business cycle. Expectations are of fundamental importance in economics and nance, since the agents' decisions about the future depends upon their expectations and their beliefs. It is in fact very unlikely that agents are perfect foresighters with rational expectations in a complex world, characterized by an irregular pattern of prices and quantities dealt in nancial markets, in which sophisticated nancial instruments are widespread. In the rst chapter of this dissertation, I will face the machine learn- ing technique, which is a nonlinear tool used for a better tting, fore- casting and clustering of dierent nancial time series and existing information in nancial markets. In particular, I will present a collec- tion of three dierent applications of these techniques, adapted from three dierent joint works: "Yield curve estimation under extreme conditions: do RBF net- works perform better?, joint with Pier Giuseppe Giribone, Marco Neelli, Marina Resta, published Anna Esposito, Marcos Faundez- Zanuy, Carlo Francesco Morabito, Eros Pasero Edrs, Multidisci- plinary Approaches to Neural Computing/Vol. 69/ WIRN 2017 and Chapter 22 in book "Neural Advances in Processing Non- linear Dynamic Signals", Springer; Interest rates term structure models and their impact on actuarial forecasting, joint with Pier Giuseppe Giribone and Marina Resta, presented at XVIII Quantitative Finance Workshop, University of Roma 3, January 2018; Applications of Kohonen Maps in financial markets: design of an automatic system for the detection of pricing anomalies, joint with Pier Giuseppe Giribone and published on Risk Management Magazine, 3-2017. In the second chapter, I will present the study A nancial market model with conrmation bias, in which nonlinearity is present as a result of the formation of heterogeneous expectations. This work is joint with Fabio Tramontana and it has been presented during the X MDEF (Dynamic Models in Economics and Finance) Workshop at University of Urbino Carlo Bo. Finally, the third chapter is a rielaboration of another joint paper, "The eects of negative nominal risk rates on the pricing of American Calls: some theoretical and numerical insights", with Pier Giuseppe Giribone and Marina Resta, published on Modern Economy 8(7), July 2017, pp 878-887. The problem of quantifying the value of early ex- ercise in an option written on equity is a complex mathematical issue that deals with continuous optimal control. In order to solve the con- tinuous dynamic optimization problem that involves high non linearity in the state variables, we have adopted a discretization scheme based on a stochastic trinomial tree. This methodology reveals a higher reliability and exibility than the traditional approaches based on approximated quasi-closed formulas in a context where financial markets are characterized by strong anomalies such as negative interest rates