Correlated Initialization for Correlated Data

Spatial data exhibits the property that nearby points are correlated. This holds also for learnt representations across layers, but not for commonly used weight initialization methods. Our theoretical analysis reveals for uncorrelated initialization that (i) flow through layers suffers from much more rapid decrease and (ii) training of individual parameters is subject to more ``zig-zagging''. We propose multiple methods for correlated initialization. For CNNs, they yield accuracy gains of several per cent in the absence of regularization. Even for properly tuned L2-regularization gains are often possible

Super-paramagnetic clustering of yeast gene expression profiles

High-density DNA arrays, used to monitor gene expression at a genomic scale, have produced vast amounts of information which require the development of efficient computational methods to analyze them. The important first step is to extract the fundamental patterns of gene expression inherent in the data. This paper describes the application of a novel clustering algorithm, Super-Paramagnetic Clustering (SPC) to analysis of gene expression profiles that were generated recently during a study of the yeast cell cycle. SPC was used to organize genes into biologically relevant clusters that are suggestive for their co-regulation. Some of the advantages of SPC are its robustness against noise and initialization, a clear signature of cluster formation and splitting, and an unsupervised self-organized determination of the number of clusters at each resolution. Our analysis revealed interesting correlated behavior of several groups of genes which has not been previously identified

Calibrating spectral estimation for the LISA Technology Package with multichannel synthetic noise generation

The scientific objectives of the Lisa Technology Package (LTP) experiment, on board of the LISA Pathfinder mission, demand for an accurate calibration and validation of the data analysis tools in advance of the mission launch. The levels of confidence required on the mission outcomes can be reached only with an intense activity on synthetically generated data. A flexible procedure allowing the generation of cross-correlated stationary noise time series was set-up. Multi-channel time series with the desired cross correlation behavior can be generated once a model for a multichannel cross-spectral matrix is provided. The core of the procedure is the synthesis of a noise coloring multichannel filter through a frequency-by-frequency eigendecomposition of the model cross-spectral matrix and a Z-domain fit. The common problem of initial transients in noise time series is solved with a proper initialization of the filter recursive equations. The noise generator performances were tested in a two dimensional case study of the LTP dynamics along the two principal channels of the sensing interferometer.Comment: Accepted for publication in Physical Review D (http://prd.aps.org/

Early Neuron Alignment in Two-layer ReLU Networks with Small Initialization

This paper studies the problem of training a two-layer ReLU network for binary classification using gradient flow with small initialization. We consider a training dataset with well-separated input vectors: Any pair of input data with the same label are positively correlated, and any pair with different labels are negatively correlated. Our analysis shows that, during the early phase of training, neurons in the first layer try to align with either the positive data or the negative data, depending on its corresponding weight on the second layer. A careful analysis of the neurons' directional dynamics allows us to provide an $\mathcal{O}(\frac{\log n}{\sqrt{\mu}})$ upper bound on the time it takes for all neurons to achieve good alignment with the input data, where $n$ is the number of data points and $\mu$ measures how well the data are separated. After the early alignment phase, the loss converges to zero at a $\mathcal{O}(\frac{1}{t})$ rate, and the weight matrix on the first layer is approximately low-rank. Numerical experiments on the MNIST dataset illustrate our theoretical findings

Source bearing and steering-vector estimation using partially calibrated arrays

The problem of source direction-of-arrival (DOA) estimation using a sensor array is addressed, where some of the sensors are perfectly calibrated, while others are uncalibrated. An algorithm is proposed for estimating the source directions in addition to the estimation of unknown array parameters such as sensor gains and phases, as a way of performing array self-calibration. The cost function is an extension of the maximum likelihood (ML) criteria that were originally developed for DOA estimation with a perfectly calibrated array. A particle swarm optimization (PSO) algorithm is used to explore the high-dimensional problem space and find the global minimum of the cost function. The design of the PSO is a combination of the problem-independent kernel and some newly introduced problem-specific features such as search space mapping, particle velocity control, and particle position clipping. This architecture plus properly selected parameters make the PSO highly flexible and reusable, while being sufficiently specific and effective in the current application. Simulation results demonstrate that the proposed technique may produce more accurate estimates of the source bearings and unknown array parameters in a cheaper way as compared with other popular methods, with the root-mean-squared error (RMSE) approaching and asymptotically attaining the Cramer Rao bound (CRB) even in unfavorable conditions