Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations

We would like to learn a representation of the data which decomposes an observation into factors of variation which we can independently control. Specifically, we want to use minimal supervision to learn a latent representation that reflects the semantics behind a specific grouping of the data, where within a group the samples share a common factor of variation. For example, consider a collection of face images grouped by identity. We wish to anchor the semantics of the grouping into a relevant and disentangled representation that we can easily exploit. However, existing deep probabilistic models often assume that the observations are independent and identically distributed. We present the Multi-Level Variational Autoencoder (ML-VAE), a new deep probabilistic model for learning a disentangled representation of a set of grouped observations. The ML-VAE separates the latent representation into semantically meaningful parts by working both at the group level and the observation level, while retaining efficient test-time inference. Quantitative and qualitative evaluations show that the ML-VAE model (i) learns a semantically meaningful disentanglement of grouped data, (ii) enables manipulation of the latent representation, and (iii) generalises to unseen groups

Le Cam meets LeCun: Deficiency and Generic Feature Learning

"Deep Learning" methods attempt to learn generic features in an unsupervised fashion from a large unlabelled data set. These generic features should perform as well as the best hand crafted features for any learning problem that makes use of this data. We provide a definition of generic features, characterize when it is possible to learn them and provide methods closely related to the autoencoder and deep belief network of deep learning. In order to do so we use the notion of deficiency and illustrate its value in studying certain general learning problems.Comment: 25 pages, 2 figure

Generalized Statistics Variational Perturbation Approximation using q-Deformed Calculus

A principled framework to generalize variational perturbation approximations (VPA's) formulated within the ambit of the nonadditive statistics of Tsallis statistics, is introduced. This is accomplished by operating on the terms constituting the perturbation expansion of the generalized free energy (GFE) with a variational procedure formulated using \emph{q-deformed calculus}. A candidate \textit{q-deformed} generalized VPA (GVPA) is derived with the aid of the Hellmann-Feynman theorem. The generalized Bogoliubov inequality for the approximate GFE are derived for the case of canonical probability densities that maximize the Tsallis entropy. Numerical examples demonstrating the application of the \textit{q-deformed} GVPA are presented. The qualitative distinctions between the \textit{q-deformed} GVPA model \textit{vis-\'{a}-vis} prior GVPA models are highlighted.Comment: 26 pages, 4 figure

Toward Minimal-Sufficiency in Regression Tasks: An Approach Based on a Variational Estimation Bottleneck

We propose a new variational estimation bottleneck based on a mean-squared error metric to aid regression tasks. In particular, this bottleneck - which draws inspiration from a variational information bottleneck for classification counterparts - consists of two components: (1) one captures the notion of Vr -sufficiency that quantifies the ability for an estimator in some class of estimators Vr to infer the quantity of interest; (2) the other component appears to capture a notion of Vr - minimality that quantifies the ability of the estimator to generalize to new data. We demonstrate how to train this bottleneck for regression problems. We also conduct various experiments in image denoising and deraining applications showcasing that our proposed approach can lead to neural network regressors offering better performance without suffering from overfitting