An analysis of training and generalization errors in shallow and deep networks

This paper is motivated by an open problem around deep networks, namely, the apparent absence of over-fitting despite large over-parametrization which allows perfect fitting of the training data. In this paper, we analyze this phenomenon in the case of regression problems when each unit evaluates a periodic activation function. We argue that the minimal expected value of the square loss is inappropriate to measure the generalization error in approximation of compositional functions in order to take full advantage of the compositional structure. Instead, we measure the generalization error in the sense of maximum loss, and sometimes, as a pointwise error. We give estimates on exactly how many parameters ensure both zero training error as well as a good generalization error. We prove that a solution of a regularization problem is guaranteed to yield a good training error as well as a good generalization error and estimate how much error to expect at which test data.Comment: 21 pages; Accepted for publication in Neural Network

Application of Higher-Order Neural Networks to Financial Time-Series Prediction

Financial time series data is characterized by non-linearities, discontinuities and high frequency, multi-polynomial components. Not surprisingly, conventional Artificial Neural Networks (ANNs) have difficulty in modelling such complex data. A more appropriate approach is to apply Higher-Order ANNs, which are capable of extracting higher order polynomial coefficients in the data. Moreover, since there is a one-to-one correspondence between network weights and polynomial coefficients, HONNs (unlike ANNs generally) can be considered open-, rather than 'closed box' solutions, and thus hold more appeal to the financial community. After developing Polynomial and Trigonometric HONNs, we introduce the concept of HONN groups. The latter incorporate piecewise continuous activation functions and thresholds, and as a result are capable of modelling discontinuous (piecewise continuous) data, and what's more to any degree of accuracy. Several other PHONN variants are also described. The performance of P(T)HONNs and HONN groups on representative financial time series is described (credit ratings and exchange rates). In short, HONNs offer roughly twice the performance of MLP/BP on financial time series prediction, and HONN groups around 10% further improvement

A Comprehensive Survey on Functional Approximation

The theory of functional approximation has numerous applications in sciences and industry. This thesis focuses on the possible approaches to approximate a continuous function on a compact subset of R2 using a variety of constructions. The results are presented from the following four general topics: polynomials, Fourier series, wavelets, and neural networks. Approximation with polynomials on subsets of R leads to the discussion of the Stone-Weierstrass theorem. Convergence of Fourier series is characterized on the unit circle. Wavelets are introduced following the Fourier transform, and their construction as well as ability to approximate functions in L2(R) is discussed. At the end, the universal approximation theorem for artificial neural networks is presented, and the function representation and approximation with single- and multilayer neural networks on R2 is constructed