Nonparametric Weight Initialization of Neural Networks via Integral Representation

A new initialization method for hidden parameters in a neural network is proposed. Derived from the integral representation of the neural network, a nonparametric probability distribution of hidden parameters is introduced. In this proposal, hidden parameters are initialized by samples drawn from this distribution, and output parameters are fitted by ordinary linear regression. Numerical experiments show that backpropagation with proposed initialization converges faster than uniformly random initialization. Also it is shown that the proposed method achieves enough accuracy by itself without backpropagation in some cases.Comment: For ICLR2014, revised into 9 pages; revised into 12 pages (with supplements

Kolmogorov superposition theorem and its applications

Hilbert’s 13th problem asked whether every continuous multivariate function can be written as superposition of continuous functions of 2 variables. Kolmogorov and Arnold show that every continuous multivariate function can be represented as superposition of continuous univariate functions and addition in a universal form and thus solved the problem positively. In Kolmogorov’s representation, only one univariate function (the outer function) depends on and all the other univariate functions (inner functions) are independent of the multivariate function to be represented. This greatly inspired research on representation and superposition of functions using Kolmogorov’s superposition theorem (KST). However, the numeric applications and theoretic development of KST is considerably limited due to the lack of smoothness of the univariate functions in the representation. Therefore, we investigate the properties of the outer and inner functions in detail. We show that the outer function for a given multivariate function is not unique, does not preserve the positivity of the multivariate function and has a largely degraded modulus of continuity. The structure of the set of inner functions only depends on the number of variables of the multivariate function. We show that inner functions constructed in Kolmogorov’s representation for continuous functions of a fixed number of variables can be reused by extension or projection to represent continuous functions of a different number of variables. After an investigation of the functions in KST, we combine KST with Fourier transform and write a formula regarding the change of the outer functions under different inner functions for a given multivariate function. KST is also applied to estimate the optimal cost between measures in high dimension by the optimal cost between measures in low dimension. Furthermore, we apply KST to image encryption and show that the maximal error can be obtained in the encryption schemes we suggested.Open Acces

Time Series Analysis

We provide a concise overview of time series analysis in the time and frequency domains, with lots of references for further reading.time series analysis, time domain, frequency domain

Time Series Analysis

We provide a concise overview of time series analysis in the time and frequency domains, with lots of references for further reading.time series analysis, time domain, frequency domain, Research Methods/ Statistical Methods,

Using Poisson processes to model lattice cellular networks

An almost ubiquitous assumption made in the stochastic-analytic study of the quality of service in cellular networks is Poisson distribution of base stations. It is usually justified by various irregularities in the real placement of base stations, which ideally should form the hexagonal pattern. We provide a different and rigorous argument justifying the Poisson assumption under sufficiently strong log-normal shadowing observed in the network, in the evaluation of a natural class of the typical-user service-characteristics including its SINR. Namely, we present a Poisson-convergence result for a broad range of stationary (including lattice) networks subject to log-normal shadowing of increasing variance. We show also for the Poisson model that the distribution of all these characteristics does not depend on the particular form of the additional fading distribution. Our approach involves a mapping of 2D network model to 1D image of it "perceived" by the typical user. For this image we prove our convergence result and the invariance of the Poisson limit with respect to the distribution of the additional shadowing or fading. Moreover, we present some new results for Poisson model allowing one to calculate the distribution function of the SINR in its whole domain. We use them to study and optimize the mean energy efficiency in cellular networks

A unified wavelet-based modelling framework for non-linear system identification: the WANARX model structure

A new unified modelling framework based on the superposition of additive submodels, functional components, and wavelet decompositions is proposed for non-linear system identification. A non-linear model, which is often represented using a multivariate non-linear function, is initially decomposed into a number of functional components via the wellknown analysis of variance (ANOVA) expression, which can be viewed as a special form of the NARX (non-linear autoregressive with exogenous inputs) model for representing dynamic input–output systems. By expanding each functional component using wavelet decompositions including the regular lattice frame decomposition, wavelet series and multiresolution wavelet decompositions, the multivariate non-linear model can then be converted into a linear-in-theparameters problem, which can be solved using least-squares type methods. An efficient model structure determination approach based upon a forward orthogonal least squares (OLS) algorithm, which involves a stepwise orthogonalization of the regressors and a forward selection of the relevant model terms based on the error reduction ratio (ERR), is employed to solve the linear-in-the-parameters problem in the present study. The new modelling structure is referred to as a wavelet-based ANOVA decomposition of the NARX model or simply WANARX model, and can be applied to represent high-order and high dimensional non-linear systems