Statistical physics of independent component analysis

Statistical physics is used to investigate independent component analysis with polynomial contrast functions. While the replica method fails, an adapted cavity approach yields valid results. The learning curves, obtained in a suitable thermodynamic limit, display a first order phase transition from poor to perfect generalization.Comment: 7 pages, 1 figure, to appear in Europhys. Lett

Accurate and robust image superresolution by neural processing of local image representations

Image superresolution involves the processing of an image sequence to generate a still image with higher resolution. Classical approaches, such as bayesian MAP methods, require iterative minimization procedures, with high computational costs. Recently, the authors proposed a method to tackle this problem, based on the use of a hybrid MLP-PNN architecture. In this paper, we present a novel superresolution method, based on an evolution of this concept, to incorporate the use of local image models. A neural processing stage receives as input the value of model coefficients on local windows. The data dimension-ality is firstly reduced by application of PCA. An MLP, trained on synthetic se-quences with various amounts of noise, estimates the high-resolution image data. The effect of varying the dimension of the network input space is exam-ined, showing a complex, structured behavior. Quantitative results are presented showing the accuracy and robustness of the proposed method

Learned-Norm Pooling for Deep Feedforward and Recurrent Neural Networks

In this paper we propose and investigate a novel nonlinear unit, called $L_p$ unit, for deep neural networks. The proposed $L_p$ unit receives signals from several projections of a subset of units in the layer below and computes a normalized $L_p$ norm. We notice two interesting interpretations of the $L_p$ unit. First, the proposed unit can be understood as a generalization of a number of conventional pooling operators such as average, root-mean-square and max pooling widely used in, for instance, convolutional neural networks (CNN), HMAX models and neocognitrons. Furthermore, the $L_p$ unit is, to a certain degree, similar to the recently proposed maxout unit (Goodfellow et al., 2013) which achieved the state-of-the-art object recognition results on a number of benchmark datasets. Secondly, we provide a geometrical interpretation of the activation function based on which we argue that the $L_p$ unit is more efficient at representing complex, nonlinear separating boundaries. Each $L_p$ unit defines a superelliptic boundary, with its exact shape defined by the order $p$. We claim that this makes it possible to model arbitrarily shaped, curved boundaries more efficiently by combining a few $L_p$ units of different orders. This insight justifies the need for learning different orders for each unit in the model. We empirically evaluate the proposed $L_p$ units on a number of datasets and show that multilayer perceptrons (MLP) consisting of the $L_p$ units achieve the state-of-the-art results on a number of benchmark datasets. Furthermore, we evaluate the proposed $L_p$ unit on the recently proposed deep recurrent neural networks (RNN).Comment: ECML/PKDD 201

"Johdatus operaatiotutkimukseen"

Artikkeli perustuu teokseen: Jelena Ventsel. Vedenije v issledovanije operatsij, Moskova 1964 ja sen saksankieliseen käännökseen. Artikkelin tarkoituksena on luoda yleisluonteinen katsaus lähteenä olevaan teokseen. Se on rajoitettu yleisluonteiseksi niin, että "sivuutetaan operaatiotutkimuksen työvälineiden, tilastomatematiikan ja todennäköisyyslaskennan käyttö". Kaikkia alkuperäisteoksen osioita ei esitellä ja painopiste on asetettu ensimmäiseen kappaleeseen sen sisällön yleisestä kiinnostavuudesta johtuen. "Sotilaallisella alueella taisteluvälineiden taistelutehoa koskevaa teoriaa voidaan pitää operaatiotutkimuksen ensimmäisenä vaiheena." Viime vuosikymmeninä operaatioanalyysin roolin todetaan sotalaitosten kaikilla alueilla jatkuvasti kasvaneen. Artikkelissa esitellään tarvittaessa sieltä poimittujen esimerkkien avulla teoksen käsitteistöä ja keskeistä sisältöä. Käsittelemättä jätetään kappaleet II- IX , koska niissä esitetyt menetelmät ovat "laajimmin tunnettuja"

New insights into foreground analysis of the WMAP five-year data using FASTICA

In this paper, we present a foreground analysis of the WMAP 5-year data using the FASTICA algorithm, improving on the treatment of the WMAP 3-year data in Bottino et al 2008. We revisit the nature of the free-free spectrum with the emphasis on attempting to confirm or otherwise the spectral feature claimed in Dobbler et al 2008b and explained in terms of spinning dust emission in the warm ionised medium. With the application of different Galactic cuts, the index is always flatter than the canonical value of 2.14 except for the Kp0 mask which is steeper. Irrespective of this, we can not confirm the presence of any feature in the free-free spectrum. We experiment with a more extensive approach to the cleaning of the data, introduced in connection with the iterative application of FASTICA. We confirm the presence of a residual foreground whose spatial distribution is concentrated along the Galactic plane, with pronounced emission near the Galactic center. This is consistent with the WMAP haze detected in Finkbeiner 2004. Finally, we attempted to perform the same analysis on full-sky maps. The code returns good results even for those regions where the cross-talk among the components is high. However, slightly better results in terms of the possibility of reconstructing a full-sky CMB map, are achieved with a simultaneous analysis of both the five WMAP maps and foreground templates. Nonetheless, some residuals are still present and detected in terms of an excess in the CMB power spectrum, on small angular scales. Therefore, a minimal mask for the brightest regions of the plane is necessary, and has been defined.Comment: Accepted for publication in MNRAS, 25 pages, 17 figures, 4 tables. Version with full resolution figures available at: http://www.mpa-garching.mpg.de/~bottino/downloads/bottino_etal.pd

ICA as a preprocessing technique for classification

In this paper we propose the use of the independent component analysis (ICA) [1] technique for improving the classification rate of decision trees and multilayer perceptrons [2], [3]. The use of an ICA for the preprocessing stage, makes the structure of both classifiers simpler, and therefore improves the generalization properties. The hypothesis behind the proposed preprocessing is that an ICA analysis will transform the feature space into a space where the components are independent, and aligned to the axes and therefore will be more adapted to the way that a decision tree is constructed. Also the inference of the weights of a multilayer perceptron will be much easier because the gradient search in the weight space will follow independent trajectories. The result is that classifiers are less complex and on some databases the error rate is lower. This idea is also applicable to regressio

Fourier PCA and Robust Tensor Decomposition

Fourier PCA is Principal Component Analysis of a matrix obtained from higher order derivatives of the logarithm of the Fourier transform of a distribution.We make this method algorithmic by developing a tensor decomposition method for a pair of tensors sharing the same vectors in rank-$1$ decompositions. Our main application is the first provably polynomial-time algorithm for underdetermined ICA, i.e., learning an $n \times m$ matrix $A$ from observations $y=Ax$ where $x$ is drawn from an unknown product distribution with arbitrary non-Gaussian components. The number of component distributions $m$ can be arbitrarily higher than the dimension $n$ and the columns of $A$ only need to satisfy a natural and efficiently verifiable nondegeneracy condition. As a second application, we give an alternative algorithm for learning mixtures of spherical Gaussians with linearly independent means. These results also hold in the presence of Gaussian noise.Comment: Extensively revised; details added; minor errors corrected; exposition improve

Non-Redundant Spectral Dimensionality Reduction

Spectral dimensionality reduction algorithms are widely used in numerous domains, including for recognition, segmentation, tracking and visualization. However, despite their popularity, these algorithms suffer from a major limitation known as the "repeated Eigen-directions" phenomenon. That is, many of the embedding coordinates they produce typically capture the same direction along the data manifold. This leads to redundant and inefficient representations that do not reveal the true intrinsic dimensionality of the data. In this paper, we propose a general method for avoiding redundancy in spectral algorithms. Our approach relies on replacing the orthogonality constraints underlying those methods by unpredictability constraints. Specifically, we require that each embedding coordinate be unpredictable (in the statistical sense) from all previous ones. We prove that these constraints necessarily prevent redundancy, and provide a simple technique to incorporate them into existing methods. As we illustrate on challenging high-dimensional scenarios, our approach produces significantly more informative and compact representations, which improve visualization and classification tasks

Does environment affect the star formation histories of early-type galaxies?

Differences in the stellar populations of galaxies can be used to quantify the effect of environment on the star formation history. We target a sample of early-type galaxies from the Sloan Digital Sky Survey in two different environmental regimes: close pairs and a general sample where environment is measured by the mass of their host dark matter halo. We apply a blind source separation technique based on principal component analysis, from which we define two parameters that correlate, respectively, with the average stellar age (eta) and with the presence of recent star formation (zeta) from the spectral energy distribution of the galaxy. We find that environment leaves a second order imprint on the spectra, whereas local properties - such as internal velocity dispersion - obey a much stronger correlation with the stellar age distribution.Comment: 5 pages, 2 figures. Proceedings of JENAM 2010, Symposium 2: "Environment and the formation of galaxies: 30 years later