Architectural designs of Echo State Network

It investigates systematically the reservoir construction of Echo State Network (ESN). This thesis proposes two very simple deterministic ESN organisation (Simple Cycle reservoir (SCR) and Cycle Reservoir with Jumps (CRJ). Simple Cycle reservoir (SCR) is sufficient to obtain performances comparable to those of the classical ESN. While Cycle Reservoir with Jumps (CRJ) significantly outperform the those of the classical ESN. This thesis also studies and discusses three reservoir characterisations - short-term memory capacity (MC), eigen-spectrum of the reservoir weight matrix and Lyapunov Exponent with their relation to the ESN performance. It also designs and utilises an ensemble of ESNs with diverse reservoirs whose collective readout is obtained through Negative Correlation Learning (NCL) of ensemble of Multi-Layer Perceptrons (MLP), where each individual MPL realises the readout from a single ESN. Finally, this thesis investigates the relation between two quantitative measures characterising short term memory in input driven dynamical systems, namely the short term memory capacity (MC), and the Fisher memory curve (FMC)

Hyperparameters optimization on neural networks for bond trading

Project Work presented as the partial requirement for obtaining a Master's degree in Statistics and Information Management, specialization in Risk Analysis and ManagementArtificial Neural Networks have been recently spotlighted as de facto tools used for classification. Their ability to deal with complex decision boundaries makes them potentially suitable to work on trading within financial markets, namely on Bonds. Such classifier faces high flexibility on its parameters in parallel with great modularity of its techniques, arising thus the need to efficiently optimize its hyperparameters. To determine the most effcient search method to optimize almost the majority of the Neural Networks hyperparameters, we have compared the results obtained by the manual, evolutionary (genetic algorithm) and random search methods. The search methods compete on several metrics from which we aim to estimate the generalization capability, i.e. the capacity to correctly predict on unseen data. We have found the manual method to present better generalization results than the remaining automatic methods. Also, no benefit was found on the direction provided by the genetic search method when compared to the purely random. Such results demonstrate the importance of human oversight during the hyperparameters optimization and weight training phases, capable of analyzing in parallel multiple metrics and data visualization techniques, a process critical to avoid suboptimal solutions when navigating complex hyperspaces

Neural text line extraction in historical documents: a two-stage clustering approach

Accessibility of the valuable cultural heritage which is hidden in countless scanned historical documents is the motivation for the presented dissertation. The developed (fully automatic) text line extraction methodology combines state-of-the-art machine learning techniques and modern image processing methods. It demonstrates its quality by outperforming several other approaches on a couple of benchmarking datasets. The method is already being used by a wide audience of researchers from different disciplines and thus contributes its (small) part to the aforementioned goal.Das Erschließen des unermesslichen Wissens, welches in unzähligen gescannten historischen Dokumenten verborgen liegt, bildet die Motivation für die vorgelegte Dissertation. Durch das Verknüpfen moderner Verfahren des maschinellen Lernens und der klassischen Bildverarbeitung wird in dieser Arbeit ein vollautomatisches Verfahren zur Extraktion von Textzeilen aus historischen Dokumenten entwickelt. Die Qualität wird auf verschiedensten Datensätzen im Vergleich zu anderen Ansätzen nachgewiesen. Das Verfahren wird bereits durch eine Vielzahl von Forschern verschiedenster Disziplinen genutzt

Essays in high-dimensional nonlinear time series analysis

In this thesis, I study high-dimensional nonlinear time series analysis, and its applications in financial forecasting and identifying risk in highly interconnected financial networks. The first chapter is devoted to the testing for nonlinearity in financial time series. I present a tentative classification of the various linearity tests that have been proposed in the literature. Then I investigate nonlinear features of real financial series to determine if the data justify the use of nonlinear techniques, such as those inspired by machine learning theories. In Chapter 3 & 5, I develop forecasting strategies with a high-dimensional panel of predictors while considering nonlinear dynamics. Combining these two elements is a developing area of research. In the third chapter, I propose a nonlinear generalization of the statistical factor models. As a first step, factor estimation, I employ an auto-associative neural network to estimate nonlinear factors from predictors. In the second step, forecasting equation, I apply a nonlinear function -feedforward neural networkon estimated factors for prediction. I show that these features can go beyond covariance analysis and enhance forecast accuracy. I apply this approach to forecast equity returns, and show that capturing nonlinear dynamics between equities significantly improves the quality of forecasts over current univariate and multivariate factor models. In Chapter 5, I propose a high-dimensional learning based on a shrinkage estimation of a backpropagation algorithm for skip-layer neural networks. This thesis emphasizes that linear models can be represented as special cases of these two aforementioned models, which basically means that if there is no nonlinearity between series, the proposed models will reduce to a linear model. This thesis also includes a chapter (chapter 4, with Negar Kiyavash and Seyedjalal Etesami), which in this chapter, we propose a new approach for identifying and measuring systemic risk in financial networks by introducing a nonlinearly modified Granger-causality network based on directed information graphs. The suggested method allows for nonlinearity and has predictive power over future economic activity through a time-varying network of interconnections. We apply the method to the daily returns of U.S. financial Institutions including banks, brokers and insurance companiesto identifythe level of systemic risk inthe financial sector and the contribution of each financial institution

River stage prediction based on a distributed support vector regression

Author name used in this publication: K. W. Chau2008-2009 > Academic research: refereed > Publication in refereed journalAccepted ManuscriptPublishe

A Deep Learning Parameterization for Ozone Dry Deposition Velocities

The loss of ozone to terrestrial and aquatic systems, known as dry deposition, is a highly uncertain process governed by turbulent transport, interfacial chemistry, and plant physiology. We demonstrate the value of using Deep Neural Networks (DNN) in predicting ozone dry deposition velocities. We find that a feedforward DNN trained on observations from a coniferous forest site (Hyytiala, Finland) can predict hourly ozone dry deposition velocities at a mixed forest site (Harvard Forest, Massachusetts) more accurately than modern theoretical models, with a reduction in the normalized mean bias (0.05 versus similar to 0.1). The same DNN model, when driven by assimilated meteorology at 2 degrees x 2.5 degrees spatial resolution, outperforms the Wesely scheme as implemented in the GEOS-Chem model. With more available training data from other climate and ecological zones, this methodology could yield a generalizable DNN suitable for global models. Plain Language Summary Ozone in the lower atmosphere is a toxic pollutant and greenhouse gas. In this work, we use a machine learning technique known as deep learning, to simulate the loss of ozone to Earth's surface. We show that our deep learning simulation of this loss process outperforms existing traditional models and demonstrate the opportunity for using machine learning to improve our understanding of the chemical composition of the atmosphere.Peer reviewe