822 research outputs found
A Generalized EigenGame with Extensions to Multiview Representation Learning
Generalized Eigenvalue Problems (GEPs) encompass a range of interesting
dimensionality reduction methods. Development of efficient stochastic
approaches to these problems would allow them to scale to larger datasets.
Canonical Correlation Analysis (CCA) is one example of a GEP for dimensionality
reduction which has found extensive use in problems with two or more views of
the data. Deep learning extensions of CCA require large mini-batch sizes, and
therefore large memory consumption, in the stochastic setting to achieve good
performance and this has limited its application in practice. Inspired by the
Generalized Hebbian Algorithm, we develop an approach to solving stochastic
GEPs in which all constraints are softly enforced by Lagrange multipliers. Then
by considering the integral of this Lagrangian function, its pseudo-utility,
and inspired by recent formulations of Principal Components Analysis and GEPs
as games with differentiable utilities, we develop a game-theory inspired
approach to solving GEPs. We show that our approaches share much of the
theoretical grounding of the previous Hebbian and game theoretic approaches for
the linear case but our method permits extension to general function
approximators like neural networks for certain GEPs for dimensionality
reduction including CCA which means our method can be used for deep multiview
representation learning. We demonstrate the effectiveness of our method for
solving GEPs in the stochastic setting using canonical multiview datasets and
demonstrate state-of-the-art performance for optimizing Deep CCA
Analysis and Segmentation of Face Images using Point Annotations and Linear Subspace Techniques
This report provides an analysis of 37 annotated frontal face images. All results presented have been obtained using our freely available Active Appearance Model (AAM) implementation. To ensure the reproducibility of the presented experiments, the data set has also been made available. As such, the data and this report may serve as a point of reference to compare other AAM implementations against. In addition, we address the problem of AAM model truncation using parallel analysis along with a comparable study of the two prevalent AAM learning methods; principal component regression and estimation of fixed Jacobian matrices. To assess applicability and efficiency, timings for model building, warping and optimisation are given together with a description of ho
Active Learning of Spin Network Models
The inverse statistical problem of finding direct interactions in complex networks is difficult. In the context of the experimental sciences, well-controlled perturbations can be applied to a system, probing the internal structure of the network. Therefore, we propose a general mathematical framework to study inference with iteratively applied perturbations to a network. Formulating active learning in the language of information geometry, our framework quantifies the difficulty of inference as well as the information gain due to perturbations through the curvature of the underlying parameter manifold as measured though the empirical Fisher information. Perturbations are then chosen that reduce most the variance of the Bayesian posterior. We apply this framework to a specific probabilistic graphical model where the nodes in the network are modeled as binary variables, "spins" with Ising-form pairwise interactions. Based on this strategy, we significantly improve the accuracy and efficiency of inference from a reasonable number of experimental queries for medium sized networks. Our active learning framework could be powerful in the analysis of complex networks as well as in the rational design of experiments
Sparse CCA: Adaptive Estimation and Computational Barriers
Canonical correlation analysis is a classical technique for exploring the
relationship between two sets of variables. It has important applications in
analyzing high dimensional datasets originated from genomics, imaging and other
fields. This paper considers adaptive minimax and computationally tractable
estimation of leading sparse canonical coefficient vectors in high dimensions.
First, we establish separate minimax estimation rates for canonical coefficient
vectors of each set of random variables under no structural assumption on
marginal covariance matrices. Second, we propose a computationally feasible
estimator to attain the optimal rates adaptively under an additional sample
size condition. Finally, we show that a sample size condition of this kind is
needed for any randomized polynomial-time estimator to be consistent, assuming
hardness of certain instances of the Planted Clique detection problem. The
result is faithful to the Gaussian models used in the paper. As a byproduct, we
obtain the first computational lower bounds for sparse PCA under the Gaussian
single spiked covariance model
- …