Unsupervised Learning via Total Correlation Explanation

Learning by children and animals occurs effortlessly and largely without obvious supervision. Successes in automating supervised learning have not translated to the more ambiguous realm of unsupervised learning where goals and labels are not provided. Barlow (1961) suggested that the signal that brains leverage for unsupervised learning is dependence, or redundancy, in the sensory environment. Dependence can be characterized using the information-theoretic multivariate mutual information measure called total correlation. The principle of Total Cor-relation Ex-planation (CorEx) is to learn representations of data that "explain" as much dependence in the data as possible. We review some manifestations of this principle along with successes in unsupervised learning problems across diverse domains including human behavior, biology, and language.Comment: Invited contribution for IJCAI 2017 Early Career Spotlight. 5 pages, 1 figur

Relaxed uncertainty relations and information processing

We consider a range of "theories" that violate the uncertainty relation for anti-commuting observables derived in [JMP, 49, 062105 (2008)]. We first show that Tsirelson's bound for the CHSH inequality can be derived from this uncertainty relation, and that relaxing this relation allows for non-local correlations that are stronger than what can be obtained in quantum mechanics. We continue to construct a hierarchy of related non-signaling theories, and show that on one hand they admit superstrong random access encodings and exponential savings for a particular communication problem, while on the other hand it becomes much harder in these theories to learn a state. We show that the existence of these effects stems from the absence of certain constraints on the expectation values of commuting measurements from our non-signaling theories that are present in quantum theory.Comment: 33 pages, 1 figure. v2: improved notation, to appear in QI