Large deviations and applications : the finite dimensional case

Includes bibliographical references (p. 96-98).Research supported by the Air Force Office of Scientific Research. AFOSR-89-0276B Research supported by the Army Research Office. DAAL03-86-K-171A. Dembo, O. Zeitouni

Large Deviations Techniques for Error Exponents to Multiple Hypothesis LAO Testing

In this article the problem of multiple hypotheses testing using a theory of large deviations is studied. The reliability matrix of Logarithmically Asymptotically Optimal (LAO) tests is introduced and described, and the conditions for the positive of all its elements are indicated

Local to global geometric methods in information theory

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2008.Includes bibliographical references (p. 201-203).This thesis treats several information theoretic problems with a unified geometric approach. The development of this approach was motivated by the challenges encountered while working on these problems, and in turn, the testing of the initial tools to these problems suggested numerous refinements and improvements on the geometric methods. In ergodic probabilistic settings, Sanov's theorem gives asymptotic estimates on the probabilities of very rare events. The theorem also characterizes the exponential decay of the probabilities, as the sample size grows, and the exponential rate is given by the minimization of a certain divergence expression. In his seminal paper, A Mathematical Theory of Communication, Shannon introduced two influential ideas to simplify the complex task of evaluating the performance of a coding scheme: the asymptotic perspective (in the number of channel uses) and the random coding argument. In this setting, Sanov's theorem can be used to analyze ergodic information theoretic problems, and the performance of a coding scheme can be estimated by expressions involving the divergence. One would then like to use a geometric intuition to solve these problems, but the divergence is not a distance and our naive geometric intuition may lead to incorrect conclusions. In information geometry, a specific differential geometric structure is introduced by means of "dual affine connections". The approach we take in this thesis is slightly different and is based on introducing additional asymptotic regimes to analyze the divergence expressions. The following two properties play an important role. The divergence may not be a distance, but locally (i.e., when its arguments are "close to each other"), the divergence behaves like a squared distance.(cont.) Moreover, globally (i.e., when its arguments have no local restriction), it also preserves certain properties satisfied by squared distances. Therefore, we develop the Very Noisy and Hermite transformations, as techniques to map our global information theoretic problems in local ones. Through this localization, our global divergence expressions reduce in the limit to expressions defined in an inner product space. This provides us with a valuable geometric insight to the global problems, as well as a strong tool to find counter-examples. Finally, in certain cases, we have been able to "lift" results proven locally to results proven globally.(cont.) Therefore, we develop the Very Noisy and Hermite transformations, as techniques to map our global information theoretic problems in local ones. Through this localization, our global divergence expressions reduce in the limit to expressions defined in an inner product space. This provides us with a valuable geometric insight to the global problems, as well as a strong tool to find counter-examples. Finally, in certain cases, we have been able to "lift" results proven locally to results proven globally. We consider the following three problems. First, we address the problem of finding good linear decoders (maximizing additive metrics) for compound discrete memoryless channels. Known universal decoders are not linear and most of them heavily depend on the finite alphabet assumption. We show that by using a finite number of additive metrics, we can construct decoders that are universal (capacity achieving) on most compound sets. We then consider additive Gaussian noise channels. For a given perturbation of a Gaussian input distribution, we define an operator that measures how much variation is induced in the output entropy. We found that the singular functions of this operator are the Hermite polynomials, and the singular values are the powers of a signal to noise ratio. We show, in particular, how to use this structure on a Gaussian interference channel to characterize a regime where interference should not be treated as noise. Finally, we consider multi-input multi-output channels and discuss the properties of the optimal input distributions, for various random fading matrix ensembles. In particular, we prove Telatar's conjecture on the covariance structure minimizing the outage probability for output dimension one and input dimensions less than one hundred.by Emmanuel Auguste Abbe.Ph.D

When all information is not created equal

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2008.Includes bibliographical references (p. 191-196).Following Shannon's landmark paper, the classical theoretical framework for communication is based on a simplifying assumption that all information is equally important, thus aiming to provide a uniform protection to all information. However, this homogeneous view of information is not suitable for a variety of modern-day communication scenarios such as wireless and sensor networks, video transmission, interactive systems, and control applications. For example, an emergency alarm from a sensor network needs more protection than other transmitted information. Similarly, the coarse resolution of an image needs better protection than its finer details. For such heterogeneous information, if providing a uniformly high protection level to all parts of the information is infeasible, it is desirable to provide different protection levels based on the importance of those parts. The main objective of this thesis is to extend classical information theory to address this heterogeneous nature of information. Many theoretical tools needed for this are fundamentally different from the conventional homogeneous setting. One key issue is that bits are no more a sufficient measure of information. We develop a general framework for understanding the fundamental limits of transmitting such information, calculate such fundamental limits, and provide optimal architectures for achieving these limits. Our analysis shows that even without sacrificing the data-rate from channel capacity, some crucial parts of information can be protected with exponential reliability. This research would challenge the notion that a set of homogenous bits should necessarily be viewed as a universal interface to the physical layer; this potentially impacts the design of network architectures. This thesis also develops two novel approaches for simplifying such difficult problems in information theory. Our formulations are based on ideas from graphical models and Euclidean geometry and provide canonical examples for network information theory. They provide fresh insights into previously intractable problems as well as generalize previous related results.by Shashibhushan Prataprao Borade.Ph.D

Learning with mistures of trees

Thesis (Ph.D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 1999.Includes bibliographical references (p. 125-129).by Marina Meilă-Predoviciu.Ph.D

Sparse graph codes for compression, sensing, and secrecy

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2010.Cataloged from student PDF version of thesis.Includes bibliographical references (p. 201-212).Sparse graph codes were first introduced by Gallager over 40 years ago. Over the last two decades, such codes have been the subject of intense research, and capacity approaching sparse graph codes with low complexity encoding and decoding algorithms have been designed for many channels. Motivated by the success of sparse graph codes for channel coding, we explore the use of sparse graph codes for four other problems related to compression, sensing, and security. First, we construct locally encodable and decodable source codes for a simple class of sources. Local encodability refers to the property that when the original source data changes slightly, the compression produced by the source code can be updated easily. Local decodability refers to the property that a single source symbol can be recovered without having to decode the entire source block. Second, we analyze a simple message-passing algorithm for compressed sensing recovery, and show that our algorithm provides a nontrivial f1/f1 guarantee. We also show that very sparse matrices and matrices whose entries must be either 0 or 1 have poor performance with respect to the restricted isometry property for the f2 norm. Third, we analyze the performance of a special class of sparse graph codes, LDPC codes, for the problem of quantizing a uniformly random bit string under Hamming distortion. We show that LDPC codes can come arbitrarily close to the rate-distortion bound using an optimal quantizer. This is a special case of a general result showing a duality between lossy source coding and channel coding-if we ignore computational complexity, then good channel codes are automatically good lossy source codes. We also prove a lower bound on the average degree of vertices in an LDPC code as a function of the gap to the rate-distortion bound. Finally, we construct efficient, capacity-achieving codes for the wiretap channel, a model of communication that allows one to provide information-theoretic, rather than computational, security guarantees. Our main results include the introduction of a new security critertion which is an information-theoretic analog of semantic security, the construction of capacity-achieving codes possessing strong security with nearly linear time encoding and decoding algorithms for any degraded wiretap channel, and the construction of capacity-achieving codes possessing semantic security with linear time encoding and decoding algorithms for erasure wiretap channels. Our analysis relies on a relatively small set of tools. One tool is density evolution, a powerful method for analyzing the behavior of message-passing algorithms on long, random sparse graph codes. Another concept we use extensively is the notion of an expander graph. Expander graphs have powerful properties that allow us to prove adversarial, rather than probabilistic, guarantees for message-passing algorithms. Expander graphs are also useful in the context of the wiretap channel because they provide a method for constructing randomness extractors. Finally, we use several well-known isoperimetric inequalities (Harper's inequality, Azuma's inequality, and the Gaussian Isoperimetric inequality) in our analysis of the duality between lossy source coding and channel coding.by Venkat Bala Chandar.Ph.D

Vibrational and Anion Photoelectron Spectroscopy of Transition Metal Clusters

The understanding of chemical bonding and of the resulting atomic arrangements is a central topic in molecular physics. The bonding mechanisms of transition metal atoms still constitute a challenge in their theoretical description due to the massive number of valence electrons. Moreover, small transition metal clusters and their complexes may serve as models for catalytic systems of interest for science and technology. The goal of this thesis is the characterization of the geometric and electronic structures of isolated transition metal clusters in the gas phase and, consequently, a better understanding of their bonding nature. The first part of the thesis encompasses experimental results and conclusions for the anionic platinum trimer (Pt3-) and the tantalum nitride anion (TaN-). The data is obtained through anion photoelectron spectroscopy via velocity map imaging (VMI), which permits the simultaneous measurement of photoelectron spectra and photoelectron angular distributions (PADs). The study on TaN- reports the first photoelectron spectra of this diatomic molecule. The spectroscopic assignments, carried out with the support of previous theoretical and experimental works, provide measurements of the adiabatic electron affinity (EAad) and of the vibrational frequencies of the anion and the neutral molecule. In addition, the analysis of the PADs reveals the existence of two core excited shape resonances and disentangles the hybridization of a key molecular orbital. In the study of Pt3-, the experiment is performed in the slow electron velocity map imaging (SEVI) mode to resolve the low-frequency vibrational structure, characteristic of metal clusters. A plethora of information is obtained with the support of density functional theory (DFT) calculations, which includes the presence of two isomeric forms (triangular and linear) and hints at pseudo Jahn-Teller and Jahn-Teller effects. Some of the PADs reveal an oscillatory energetic dependence that, according to the quantum analogy established by Fano with the Young’s double slit experiment, is interpreted as interferometric efects in the linear isomer. In the second part of the thesis, the characterizations of CoArn+ (n=3 to 6), Con-mMnmArx+ (n=3 to 14; m=0 to 2) and saturated cationic Ru cluster carbonyls Run(CO)m+ (n=1 to 8) via (far-)infrared multiple photon dissociation (IRMPD) spectroscopy, more suitable to obtain structural information on the studied complexes, are reported. The investigation on CoArn+ readdresses the question on the nature of the interaction between the metal cation and the rare gas atom by proposing some amount of covalency in the bonding. Motivated by the role of ruthenium as catalyst for CO methanation in the Fischer-Tropsch process, the composition and structures of saturated cationic ruthenium carbonyls are studied. Their IR spectra are obtained in the range of the C-O stretches, Ru-CO stretches and deformation modes. Structural assignment is achieved with the aim of DFT calculations and the results corroborate former predictions. Finally, IR spectra of Con-mMnmArx+ are presented and discussed in comparison to those of pure cobalt clusters with the purpose of motivating future theoretical studies that may solve the puzzling structures of these binary metal clusters.Das Verständnis der chemischen Bindung und der daraus folgenden räumlichen Anordnungen von Atomen ist ein zentrales Thema der Molekülphysik. Die große Zahl der Valenzelektronen von Übergangsmetall-Atomen stellt auch heute noch eine Herausforderung für die theoretische Beschreibung ihrer Bindungsmechanismen dar. Gleichwohl können kleine Übergangsmetallcluster und ihre Komplexe als Modelle für katalytische Systeme von Relevanz in Wissenschaft und Technik dienen. Das Ziel dieser Arbeit ist die Charakterisierung der geometrischen und elektronischen Strukturen von isolierten Übergangsmetallclustern in der Gasphase und damit ein besseres Verständnis für das Wesen ihrer chemischen Bindung. Der erste Teil dieser Arbeit beinhaltet experimentelle Ergebnisse und darauf beruhende Schlussfolgerungen für das Anion des Platintrimers (Pt3-)und das Tantalnitrid-Anion (TaN-). Die Daten basieren auf Anionen-Photoelek- tronenspektroskopie mittels velocity map imaging (VMI), einer Technik, die die gleichzeitige Messung der Photoelektronenspektren und der Winkelverteil- ung der Photoelektronen ermöglicht. Für das zweiatomige Molekül TaN- wurde das Photoelektronenspektrum erstmalig bestimmt. Die spektroskopische Zuordnung gelang unter Zuhilfenahme vorhergehender theoretischer und experimenteller Arbeiten und liefert Werte für die adiabatische Elektronenaffinität (EAad) sowie die Schwingungsfrequenzen des Anions und des neutralen Moleküls. Weiterhin lässt die Untersuchung der Winkelverteilung auf das Vorliegen von zwei rumpfangeregten Formresonanzen (engl. core excited shape resonances) schließen und erlaubt die Analyse der Hybridisierung eines wichtigen Molekülorbitals. Die experimentelle Untersuchung von Pt3- erfolgte im SEVI-Modus (von engl. slow electron velocity map imaging), um die Schwingungsstruktur bei den für Metallclustern charakteristischen niedrigen Frequenzen aufzulösen. Die große Menge an experimentellen Daten erlaubt, unterstützt durch Dichtefunktionaltheorie(DFT)-Rechnungen, den Nachweis von zwei isomeren Formen (linear und gewinkelt) und liefert Hinweise auf den Einfluss von Jahn-Teller- und Pseudo-Jahn-Teller-Effekten. Einige Winkelverteilungen zeigen eine oszillierende Energieabhängigkeit, die, entsprechend der von Fano beschriebenen Quanten-Analogie zu Youngs Doppelspaltexperiment, als Interferenzeffekt im linearen Isomer gedeutet werden kann. Der zweite Teil der Arbeit widmet sich der Charakterisierung von CoArn+ (n=3 to 6), Con-mMnmArx+ (n=3 to 14; m=0 to 2) und gesättigten kationischen Rutheniumcluster-Carbonylen Rum(CO)m+ (n=1 to 8) mittels (Fern-)Infrarot Mehrphotonendissoziations-(IRMPD)-Spektroskopie, welche besonders geeignet ist, Strukturinfomationen für diese Komplexe zu erlangen. Die Untersuchung von CoArn+ widmet sich der Frage nach der Art der Wechselwirkung zwischen Metallkation und Edelgasatom und lässt auf einen kovalenten Anteil an der Bindung schließen. Motiviert durch die Funktion von Ruthenium als Katalysator bei der CO-Methanisierung in der Fischer-Tropsch-Synthese wurden Zusammensetzung und Struktur von gesättigten kationischen Rutheniumcarbonylen untersucht. Deren IR-Spektren wurden im Bereich der C-O Streckschwingung sowie der Ru-CO Streck- und Deformationsmoden bestimmt. Die Zuordnung der Strukturen gelang mit Hilfe von DFT-Rechnungen und bestätigt frühere Vorhersagen. Schließlich werden die IR-Spektren von Con-mMnmArx+ vorgestellt und, im Vergleich zu denen der reinen Cobaltcluster, diskutiert. Diese Daten stehen als Motivation für zukünftige theoretische Untersuchungen, die das Rätsel um die Strukturen dieser binären Metallcluster lösen könnten

Informative sensing : theory and applications

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2012.Cataloged from PDF version of thesis.Includes bibliographical references (p. 145-156).Compressed sensing is a recent theory for the sampling and reconstruction of sparse signals. Sparse signals only occupy a tiny fraction of the entire signal space and thus have a small amount of information, relative to their dimension. The theory tells us that the information can be captured faithfully with few random measurement samples, even far below the Nyquist rate. Despite the successful story, we question how the theory would change if we had a more precise prior than the simple sparsity model. Hence, we consider the settings where the prior is encoded as a probability density. In a Bayesian perspective, we see the signal recovery as an inference, in which we estimate the unmeasured dimensions of the signal given the incomplete measurements. We claim that good sensors should somehow be designed to minimize the uncertainty of the inference. In this thesis, we primarily use Shannon's entropy to measure the uncertainty and in effect pursue the InfoMax principle, rather than the restricted isometry property, in optimizing the sensors. By approximate analysis on sparse signals, we found random projections, typical in the compressed sensing literature, to be InfoMax optimal if the sparse coefficients are independent and identically distributed (i.i.d.). If not, however, we could find a different set of projections which, in signal reconstruction, consistently outperformed random or other types of measurements. In particular, if the coefficients are groupwise i.i.d., groupwise random projections with nonuniform sampling rate per group prove asymptotically Info- Max optimal. Such a groupwise i.i.d. pattern roughly appears in natural images when the wavelet basis is partitioned into groups according to the scale. Consequently, we applied the groupwise random projections to the sensing of natural images. We also considered designing an optimal color filter array for single-chip cameras. In this case, the feasible set of projections is highly restricted because multiplexing across pixels is not allowed. Nevertheless, our principle still applies. By minimizing the uncertainty of the unmeasured colors given the measured ones, we could find new color filter arrays which showed better demosaicking performance in comparison with Bayer or other existing color filter arrays.by Hyun Sung Chang.Ph.D

Convex relaxation methods for graphical models : Lagrangian and maximum entropy approaches

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2008.Includes bibliographical references (p. 241-257).Graphical models provide compact representations of complex probability distributions of many random variables through a collection of potential functions defined on small subsets of these variables. This representation is defined with respect to a graph in which nodes represent random variables and edges represent the interactions among those random variables. Graphical models provide a powerful and flexible approach to many problems in science and engineering, but also present serious challenges owing to the intractability of optimal inference and estimation over general graphs. In this thesis, we consider convex optimization methods to address two central problems that commonly arise for graphical models. First, we consider the problem of determining the most probable configuration-also known as the maximum a posteriori (MAP) estimate-of all variables in a graphical model, conditioned on (possibly noisy) measurements of some variables. This general problem is intractable, so we consider a Lagrangian relaxation (LR) approach to obtain a tractable dual problem. This involves using the Lagrangian decomposition technique to break up an intractable graph into tractable subgraphs, such as small "blocks" of nodes, embedded trees or thin subgraphs. We develop a distributed, iterative algorithm that minimizes the Lagrangian dual function by block coordinate descent. This results in an iterative marginal-matching procedure that enforces consistency among the subgraphs using an adaptation of the well-known iterative scaling algorithm. This approach is developed both for discrete variable and Gaussian graphical models. In discrete models, we also introduce a deterministic annealing procedure, which introduces a temperature parameter to define a smoothed dual function and then gradually reduces the temperature to recover the (non-differentiable) Lagrangian dual. When strong duality holds, we recover the optimal MAP estimate. We show that this occurs for a broad class of "convex decomposable" Gaussian graphical models, which generalizes the "pairwise normalizable" condition known to be important for iterative estimation in Gaussian models.(cont.) In certain "frustrated" discrete models a duality gap can occur using simple versions of our approach. We consider methods that adaptively enhance the dual formulation, by including more complex subgraphs, so as to reduce the duality gap. In many cases we are able to eliminate the duality gap and obtain the optimal MAP estimate in a tractable manner. We also propose a heuristic method to obtain approximate solutions in cases where there is a duality gap. Second, we consider the problem of learning a graphical model (both the graph and its potential functions) from sample data. We propose the maximum entropy relaxation (MER) method, which is the convex optimization problem of selecting the least informative (maximum entropy) model over an exponential family of graphical models subject to constraints that small subsets of variables should have marginal distributions that are close to the distribution of sample data. We use relative entropy to measure the divergence between marginal probability distributions. We find that MER leads naturally to selection of sparse graphical models. To identify this sparse graph efficiently, we use a "bootstrap" method that constructs the MER solution by solving a sequence of tractable subproblems defined over thin graphs, including new edges at each step to correct for large marginal divergences that violate the MER constraint. The MER problem on each of these subgraphs is efficiently solved using the primaldual interior point method (implemented so as to take advantage of efficient inference methods for thin graphical models). We also consider a dual formulation of MER that minimizes a convex function of the potentials of the graphical model. This MER dual problem can be interpreted as a robust version of maximum-likelihood parameter estimation, where the MER constraints specify the uncertainty in the sufficient statistics of the model. This also corresponds to a regularized maximum-likelihood approach, in which an information-geometric regularization term favors selection of sparse potential representations. We develop a relaxed version of the iterative scaling method to solve this MER dual problem.by Jason K. Johnson.Ph.D