Nonlinear Information Bottleneck

Information bottleneck (IB) is a technique for extracting information in one random variable $X$ that is relevant for predicting another random variable $Y$. IB works by encoding $X$ in a compressed "bottleneck" random variable $M$ from which $Y$ can be accurately decoded. However, finding the optimal bottleneck variable involves a difficult optimization problem, which until recently has been considered for only two limited cases: discrete $X$ and $Y$ with small state spaces, and continuous $X$ and $Y$ with a Gaussian joint distribution (in which case optimal encoding and decoding maps are linear). We propose a method for performing IB on arbitrarily-distributed discrete and/or continuous $X$ and $Y$, while allowing for nonlinear encoding and decoding maps. Our approach relies on a novel non-parametric upper bound for mutual information. We describe how to implement our method using neural networks. We then show that it achieves better performance than the recently-proposed "variational IB" method on several real-world datasets

High-occupancy effects and stimulation phenomena in semiconductor microcavities

This paper describes recent work on high-occupancy effects in semiconductor microcavities, with emphasis on the variety of new physics and the potential for applications that has been demonstrated recently. It is shown that the ability to manipulate both exciton and photon properties, and how they interact together to form strongly coupled exciton-photon coupled modes, exciton polaritons, leads to a number of very interesting phenomena, which are either difficult or impossible to achieve in bulk semiconductors or quantum wells. The very low polariton density of states enables state occupancies greater than one to be easily achieved, and hence stimulation phenomena to be realized under conditions of resonant excitation. The particular form of the lower polariton dispersion curve in microcavities allows energy and momentum conserving polariton-polariton scattering under resonant excitation. Stimulated scattering of the bosonic quasi-particles occurs to the emitting state at the center of the Brillouin zone, and to a companion state at high wave vector. The stimulation phenomena lead to the formation of highly occupied states with macroscopic coherence in two specific regions of k space. The results are contrasted with phenomena that occur under conditions of nonresonant excitation. Prospects to achieve "polariton lasing" under nonresonant excitation, and high-gain, room-temperature ultrafast amplifiers and low-threshold optical parametric oscillator under resonant excitation conditions are discussed

Caveats for information bottleneck in deterministic scenarios

Information bottleneck (IB) is a method for extracting information from one random variable $X$ that is relevant for predicting another random variable $Y$. To do so, IB identifies an intermediate "bottleneck" variable $T$ that has low mutual information $I(X;T)$ and high mutual information $I(Y;T)$. The "IB curve" characterizes the set of bottleneck variables that achieve maximal $I(Y;T)$ for a given $I(X;T)$, and is typically explored by maximizing the "IB Lagrangian", $I(Y;T) - \beta I(X;T)$. In some cases, $Y$ is a deterministic function of $X$, including many classification problems in supervised learning where the output class $Y$ is a deterministic function of the input $X$. We demonstrate three caveats when using IB in any situation where $Y$ is a deterministic function of $X$: (1) the IB curve cannot be recovered by maximizing the IB Lagrangian for different values of $\beta$; (2) there are "uninteresting" trivial solutions at all points of the IB curve; and (3) for multi-layer classifiers that achieve low prediction error, different layers cannot exhibit a strict trade-off between compression and prediction, contrary to a recent proposal. We also show that when $Y$ is a small perturbation away from being a deterministic function of $X$, these three caveats arise in an approximate way. To address problem (1), we propose a functional that, unlike the IB Lagrangian, can recover the IB curve in all cases. We demonstrate the three caveats on the MNIST dataset