Efficient Information Theoretic Clustering on Discrete Lattices

We consider the problem of clustering data that reside on discrete, low dimensional lattices. Canonical examples for this setting are found in image segmentation and key point extraction. Our solution is based on a recent approach to information theoretic clustering where clusters result from an iterative procedure that minimizes a divergence measure. We replace costly processing steps in the original algorithm by means of convolutions. These allow for highly efficient implementations and thus significantly reduce runtime. This paper therefore bridges a gap between machine learning and signal processing.Comment: This paper has been presented at the workshop LWA 201

Quality-based Multimodal Classification Using Tree-Structured Sparsity

Recent studies have demonstrated advantages of information fusion based on sparsity models for multimodal classification. Among several sparsity models, tree-structured sparsity provides a flexible framework for extraction of cross-correlated information from different sources and for enforcing group sparsity at multiple granularities. However, the existing algorithm only solves an approximated version of the cost functional and the resulting solution is not necessarily sparse at group levels. This paper reformulates the tree-structured sparse model for multimodal classification task. An accelerated proximal algorithm is proposed to solve the optimization problem, which is an efficient tool for feature-level fusion among either homogeneous or heterogeneous sources of information. In addition, a (fuzzy-set-theoretic) possibilistic scheme is proposed to weight the available modalities, based on their respective reliability, in a joint optimization problem for finding the sparsity codes. This approach provides a general framework for quality-based fusion that offers added robustness to several sparsity-based multimodal classification algorithms. To demonstrate their efficacy, the proposed methods are evaluated on three different applications - multiview face recognition, multimodal face recognition, and target classification.Comment: To Appear in 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2014

Efficient Feature Selection in the Presence of Multiple Feature Classes

We present an information theoretic approach to feature selection when the data possesses feature classes. Feature classes are pervasive in real data. For example, in gene expression data, the genes which serve as features may be divided into classes based on their membership in gene families or pathways. When doing word sense disambiguation or named entity extraction, features fall into classes including adjacent words, their parts of speech, and the topic and venue of the document the word is in. When predictive features occur predominantly in a small number of feature classes, our information theoretic approach significantly improves feature selection. Experiments on real and synthetic data demonstrate substantial improvement in predictive accuracy over the standard L0 penalty-based stepwise and stream wise feature selection methods as well as over Lasso and Elastic Nets, all of which are oblivious to the existence of feature classes

Efficiently Extracting Randomness from Imperfect Stochastic Processes

We study the problem of extracting a prescribed number of random bits by reading the smallest possible number of symbols from non-ideal stochastic processes. The related interval algorithm proposed by Han and Hoshi has asymptotically optimal performance; however, it assumes that the distribution of the input stochastic process is known. The motivation for our work is the fact that, in practice, sources of randomness have inherent correlations and are affected by measurement's noise. Namely, it is hard to obtain an accurate estimation of the distribution. This challenge was addressed by the concepts of seeded and seedless extractors that can handle general random sources with unknown distributions. However, known seeded and seedless extractors provide extraction efficiencies that are substantially smaller than Shannon's entropy limit. Our main contribution is the design of extractors that have a variable input-length and a fixed output length, are efficient in the consumption of symbols from the source, are capable of generating random bits from general stochastic processes and approach the information theoretic upper bound on efficiency.Comment: 2 columns, 16 page

Quantum Side Information: Uncertainty Relations, Extractors, Channel Simulations

In the first part of this thesis, we discuss the algebraic approach to classical and quantum physics and develop information theoretic concepts within this setup. In the second part, we discuss the uncertainty principle in quantum mechanics. The principle states that even if we have full classical information about the state of a quantum system, it is impossible to deterministically predict the outcomes of all possible measurements. In comparison, the perspective of a quantum observer allows to have quantum information about the state of a quantum system. This then leads to an interplay between uncertainty and quantum correlations. We provide an information theoretic analysis by discussing entropic uncertainty relations with quantum side information. In the third part, we discuss the concept of randomness extractors. Classical and quantum randomness are an essential resource in information theory, cryptography, and computation. However, most sources of randomness exhibit only weak forms of unpredictability, and the goal of randomness extraction is to convert such weak randomness into (almost) perfect randomness. We discuss various constructions for classical and quantum randomness extractors, and we examine especially the performance of these constructions relative to an observer with quantum side information. In the fourth part, we discuss channel simulations. Shannon's noisy channel theorem can be understood as the use of a noisy channel to simulate a noiseless one. Channel simulations as we want to consider them here are about the reverse problem: simulating noisy channels from noiseless ones. Starting from the purely classical case (the classical reverse Shannon theorem), we develop various kinds of quantum channel simulation results. We achieve this by using classical and quantum randomness extractors that also work with respect to quantum side information.Comment: PhD thesis, ETH Zurich. 214 pages, 13 figures, 1 table. Chapter 2 is based on arXiv:1107.5460 and arXiv:1308.4527 . Section 3.1 is based on arXiv:1302.5902 and Section 3.2 is a preliminary version of arXiv:1308.4527 (you better read arXiv:1308.4527). Chapter 4 is (partly) based on arXiv:1012.6044 and arXiv:1111.2026 . Chapter 5 is based on arXiv:0912.3805, arXiv:1108.5357 and arXiv:1301.159