Concentration of Measure Inequalities in Information Theory, Communications and Coding (Second Edition)

During the last two decades, concentration inequalities have been the subject of exciting developments in various areas, including convex geometry, functional analysis, statistical physics, high-dimensional statistics, pure and applied probability theory, information theory, theoretical computer science, and learning theory. This monograph focuses on some of the key modern mathematical tools that are used for the derivation of concentration inequalities, on their links to information theory, and on their various applications to communications and coding. In addition to being a survey, this monograph also includes various new recent results derived by the authors. The first part of the monograph introduces classical concentration inequalities for martingales, as well as some recent refinements and extensions. The power and versatility of the martingale approach is exemplified in the context of codes defined on graphs and iterative decoding algorithms, as well as codes for wireless communication. The second part of the monograph introduces the entropy method, an information-theoretic technique for deriving concentration inequalities. The basic ingredients of the entropy method are discussed first in the context of logarithmic Sobolev inequalities, which underlie the so-called functional approach to concentration of measure, and then from a complementary information-theoretic viewpoint based on transportation-cost inequalities and probability in metric spaces. Some representative results on concentration for dependent random variables are briefly summarized, with emphasis on their connections to the entropy method. Finally, we discuss several applications of the entropy method to problems in communications and coding, including strong converses, empirical distributions of good channel codes, and an information-theoretic converse for concentration of measure.Comment: Foundations and Trends in Communications and Information Theory, vol. 10, no 1-2, pp. 1-248, 2013. Second edition was published in October 2014. ISBN to printed book: 978-1-60198-906-

On the Martingale Property for Generalized Stochastic Processes

In the recent years, several groups have studied stochastic equations (e.g. SDE's, SPDE's, stochastic Volterra equations) outside the framework of the Itô calculus. Often, this led to solutions in spaces of generalized random processes or fields. It is therefore of interest to study the probabilistic properties of generalized stochastic processes, and the present paper makes some (rather naive) first steps into this direction. If we think of a generalized process as a mapping from the real line (or an interval) into aspace of generalized random variables (with some additional properties), then there is a wide range of choices for the latter: e.g., the space (S)* of Hida distributions (e.g. [HKPS]), the space (S)-1 of Kondratiev distributions (e.g. [KLS]), the Sobolev type space D* which is used within the Malliavin calculus and so on. Often, the generalized processes coming up in applications have a chaos decomposition with kernels which belong to L2(IRn), and in this paper we shall focus our attention on this situation. It will be convenient to work with a space G* which is larger than D*. This space has already been considered by several authors, cf. e.g. [PT] and the references quoted there. It turns out, that basic notions from the theory of stochastic processes like conditional expectation, martingales, sub- (super-) martingales etc. have a rather natural generalization to mappings from the realline into G* . The paper is organized as follows. In Section 2 we shall give a construction of the Itô integral (with respect to Brownian motion) of generalized stochastic processes, and compare it with the Hitsuda-Skorokhod integral (e.g. [HKPS]). In Section 3 we define generalized martingales and derive a number of properties. In particular, we prove that the generalized Itô integrals of Section 2 are indeed generalized martingales. Moreover, a representation of generalized martingales (in analogy with the Clark-Haussmann formula) will be shown, and we prove that the Wick product of two generalized martingales is again one. Finally, in Section 4 we define the notion of a generalized semimartingale and give a class of examples. In the remainder of this Introduction we provide the necessary background

A Conversation with Chris Heyde

Born in Sydney, Australia, on April 20, 1939, Chris Heyde shifted his interest from sport to mathematics thanks to inspiration from a schoolteacher. After earning an M.Sc. degree from the University of Sydney and a Ph.D. from the Australian National University (ANU), he began his academic career in the United States at Michigan State University, and then in the United Kingdom at the University of Sheffield and the University of Manchester. In 1968, Chris moved back to Australia to teach at ANU until 1975, when he joined CSIRO, where he was Acting Chief of the Division of Mathematics and Statistics. From 1983 to 1986, he was a Professor and Chairman of the Department of Statistics at the University of Melbourne. Chris then returned to ANU to become the Head of the Statistics Department, and later the Foundation Dean of the School of Mathematical Sciences (now the Mathematical Sciences Institute). Since 1993, he has also spent one semester each year teaching at the Department of Statistics, Columbia University, and has been the director of the Center for Applied Probability at Columbia University since its creation in 1993. Chris has been honored worldwide for his contributions in probability, statistics and the history of statistics. He is a Fellow of the International Statistical Institute and the Institute of Mathematical Statistics, and he is one of three people to be a member of both the Australian Academy of Science and the Australian Academy of Social Sciences. In 2003, he received the Order of Australia from the Australian government. He has been awarded the Pitman Medal and the Hannan Medal. Chris was conferred a D.Sc. honoris causa by University of Sydney in 1998. Chris has been very active in serving the statistical community, including as the Vice President of the International Statistical Institute, President of the Bernoulli Society and Vice President of the Australian Mathematical Society. He has served on numerous editorial boards, most notably as Editor of Stochastic Processes and Their Applications from 1983 to 1989, and as Editor-in-Chief of Journal of Applied Probability and Advances in Applied Probability since 1990.Comment: Published at http://dx.doi.org/10.1214/088342306000000088 in the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org

An essay on the general theory of stochastic processes

This text is a survey of the general theory of stochastic processes, with a view towards random times and enlargements of filtrations. The first five chapters present standard materials, which were developed by the French probability school and which are usually written in French. The material presented in the last three chapters is less standard and takes into account some recent developments.Comment: Published at http://dx.doi.org/10.1214/154957806000000104 in the Probability Surveys (http://www.i-journals.org/ps/) by the Institute of Mathematical Statistics (http://www.imstat.org

Multiple Schramm-Loewner Evolutions and Statistical Mechanics Martingales

A statistical mechanics argument relating partition functions to martingales is used to get a condition under which random geometric processes can describe interfaces in 2d statistical mechanics at criticality. Requiring multiple SLEs to satisfy this condition leads to some natural processes, which we study in this note. We give examples of such multiple SLEs and discuss how a choice of conformal block is related to geometric configuration of the interfaces and what is the physical meaning of mixed conformal blocks. We illustrate the general ideas on concrete computations, with applications to percolation and the Ising model.Comment: 40 pages, 6 figures. V2: well, it looks better with the addresse

Martingale proofs of many-server heavy-traffic limits for Markovian queues

This is an expository review paper illustrating the ``martingale method'' for proving many-server heavy-traffic stochastic-process limits for queueing models, supporting diffusion-process approximations. Careful treatment is given to an elementary model -- the classical infinite-server model $M/M/\infty$, but models with finitely many servers and customer abandonment are also treated. The Markovian stochastic process representing the number of customers in the system is constructed in terms of rate-1 Poisson processes in two ways: (i) through random time changes and (ii) through random thinnings. Associated martingale representations are obtained for these constructions by applying, respectively: (i) optional stopping theorems where the random time changes are the stopping times and (ii) the integration theorem associated with random thinning of a counting process. Convergence to the diffusion process limit for the appropriate sequence of scaled queueing processes is obtained by applying the continuous mapping theorem. A key FCLT and a key FWLLN in this framework are established both with and without applying martingales.Comment: Published in at http://dx.doi.org/10.1214/06-PS091 the Probability Surveys (http://www.i-journals.org/ps/) by the Institute of Mathematical Statistics (http://www.imstat.org