Distributed Functional Scalar Quantization Simplified

Distributed functional scalar quantization (DFSQ) theory provides optimality conditions and predicts performance of data acquisition systems in which a computation on acquired data is desired. We address two limitations of previous works: prohibitively expensive decoder design and a restriction to sources with bounded distributions. We rigorously show that a much simpler decoder has equivalent asymptotic performance as the conditional expectation estimator previously explored, thus reducing decoder design complexity. The simpler decoder has the feature of decoupled communication and computation blocks. Moreover, we extend the DFSQ framework with the simpler decoder to acquire sources with infinite-support distributions such as Gaussian or exponential distributions. Finally, through simulation results we demonstrate that performance at moderate coding rates is well predicted by the asymptotic analysis, and we give new insight on the rate of convergence

Chatting in distributed quantization networks

Abstract—Several key results in source coding offer the intuition that distributed encoding via vector-quantize-and-bin is only slightly suboptimal to joint encoding and oftentimes is just as good. However, when source acquisition requires the blocklength to be small, collaboration between sensors can greatly reduce distortion. For a distributed acquisition network where sensors are allowed to “chat ” using a side channel, we provide exact characterization of distortion performance and quantizer design in the high-resolution (low-distortion) regime using a framework called distributed functional scalar quantization (DFSQ). The key result is that chatting can dramatically improve performance even when the intersensor communication is at very low rate. We also solve the rate allo-cation problem when communication links have heterogeneous costs and provide examples to demonstrate that this theory predicts performance at practical communication rates. I

Kodierung von Gaußmaßen

Es sei $gamma$ ein Gaußmaß auf der Borelschen $sigma$-Algebra $mathcal B$ des separablen Banachraums $B$. Für $X:Omega o B$ gelte $P_X=gamma$. Wir untersuchen den mittleren Fehler, der bei Kodierung von $gamma$ respektive $X$ mit $Ninmathbb N$ Punkten entsteht, und bestimmen untere und obere Abschätzungen für die Asymptotik ($N oinfty$) dieses Fehlers. Hierbei betrachten wir zu $r>0$ Gütekriterien wie folgt: Deterministische Kodierung $delta_2(N,r) := inf_{y_1,ldots,y_Nin B}Emin_{k=1,ldots,N}X-y_k^r.$ Zufällige Kodierung $delta_3(N,r) := inf_ u Emin_{k=1,ldots,N}X-Y_k^r.$ Die $(Y_k)$ seien hierbei i.i.d., unabhängig von $X$, und nach $u$ verteilt. Das Infimum wird über alle Wahrscheinlichkeitsmaße $u$ gebildet. Für das Gütekriterium $delta_4(cdot,r)$ wird ausgehend von der Definition von $delta_3(cdot,r)$ $u$ nicht optimal, sondern $u=gamma$ gewählt. Das Gütekriterium $delta_1(cdot,r)$ ergibt sich aus der Quellkodierungstheorie nach Shannon. Es gilt $delta_1(cdot,r) le delta_2(cdot,r) le delta_3(cdot,r) le delta_4(cdot,r).$ Wir stellen folgenden Zusammenhang zwischen der Asymptotik von $delta_4(cdot,r)$ und den logarithmischen kleinen Abweichungen von $gamma$ her: Es gebe $kappa,a>0$ und $binR$ mit psi(varepsilon) := -log P{X1.Let $gamma$ be a Gaussian measure on the Borel $sigma$-algebra $mathcal B$ of the separable Banach space $B$. Let $X:Omega o B$ with $P_X=gamma$. We investigate the average error in coding $gamma$ resp. $X$ with $Ninmathbb N$ points and obtain lower and upper bounds for the error asymptotics ($N oinfty$). We consider, given $r>0$, fidelity criterions as follows: Deterministic Coding $delta_2(N,r) := inf_{y_1,ldots,y_Nin B}Emin_{k=1,ldots,N}X-y_k^r.$ Random Coding $delta_3(N,r) := inf_ u Emin_{k=1,ldots,N}X-Y_k^r.$ The $(Y_k)$ above are i.i.d., independent of $X$, and distributed according to $u$. The infimum is taken with respect to all probability measures $u$. For the fidelity criterion $delta_4(cdot,r)$, starting from the definition of $delta_3(cdot,r)$, $u$ is not chosen optimal, but as $u=gamma$. The fidelity criterion $delta_1(cdot,r)$ is given according to the source coding theory of Shannon. The fidelity criterions are connected through $delta_1(cdot,r) le delta_2(cdot,r) le delta_3(cdot,r) le delta_4(cdot,r).$ We obtain the following connection between the asymptotics of $delta_4(cdot,r)$ and the den logarithmic small deviations of $gamma$: Let $kappa,a>0$ and $binR$ with psi(varepsilon) := -log P{X1

Detection of sparse mixtures: fundamental limits and algorithms

In this thesis, we study the sparse mixture detection problem as a binary hypothesis testing problem. Under the null hypothesis, we observe i.i.d. samples from a known noise distribution. Under the alternative hypothesis, we observe i.i.d. samples from a mixture of the noise distribution and signal distribution. The noise and signal distributions, as well as the proportion of signal (sparsity level), are allowed to depend on the sample size such that the proportion of signal in the mixture tends to zero as the sample size tends to infinity. The sparse mixture detection problem has applications in areas such as astrophysics, covert communications, biology and machine learning. There are two basic questions in the sparse mixture detection problem, studied in the large sample size regime: 1. Under what conditions do there exist algorithms that can distinguish pure noise from the presence of signal with vanishing error probability? 2. Can one detect the presence of a signal without knowledge of the particular signal distribution or sparsity level, with vanishing error probability? The first question is that of consistent testing, while the second question is that of adaptive testing. While previous works have studied consistency and adaptivity, particularly in the case of Gaussian signal and noise distributions, it has been shown that different consistent adaptive tests can have very different error probabilities at finite sample sizes. This thesis contributes a more refined look at consistency by studying the fundamental rates at which the error probabilities for the sparse mixture detection problem can be driven to zero with the sample size under mild assumptions on the signal and noise distributions. The fundamental rates of decay of the error probabilities are derived by characterizing the error probabilities of the oracle likelihood ratio test. We illustrate our theory on the Gaussian location model, where the noise distribution is standard Gaussian and the signal distribution is a Gaussian with unit variance and positive mean. This thesis also contributes to the field of adaptive test design. We show that when the signal and noise distributions are specified under a finite alphabet, a variant of Hoeffding's test is adaptive with rates matching the oracle likelihood ratio test. We leverage our results on finite alphabet sparse mixture detection problems to study the general sparse mixture detection problem via quantization. We build adaptive tests for general sparse mixture detection problems by studying tests which quantize the data to two levels via a sample size dependent quantizer, which we term 1-bit quantized tests. As the 1-bit quantized tests have data on a binary alphabet, we are able to precisely analyze the fundamental rate of decay of error probabilities under both hypotheses using our theory. A key contribution of our work is constructing adaptive tests for the sparse mixture detection problem by combining 1-bit quantized tests using different quantizers. The first advantage of our proposed test is that it has lower time and space complexity than other known adaptive tests for the sparse mixture detection problem. The second advantage is ease of theoretical analysis. We show that unlike existing tests such as the Higher Criticism test, our adaptive test construction offers tight control of the rate of decay for the false alarm probability under mild assumptions on the quantizers and noise distribution. We show our proposed test construction is adaptive against all possible signals in Generalized Gaussian location models. Furthermore, in the special case of a Gaussian location model, we show that the proposed adaptive test has near-optimal rate of decay of the miss detection probability, as compared with the oracle likelihood ratio test when both hypotheses are assumed to be equally likely. Numerical results show our test performs competitively with existing state-of-the-art tests

Quantization in acquisition and computation networks

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2013.Cataloged from PDF version of thesis.Includes bibliographical references (p. 151-165).In modern systems, it is often desirable to extract relevant information from large amounts of data collected at different spatial locations. Applications include sensor networks, wearable health-monitoring devices and a variety of other systems for inference. Several existing source coding techniques, such as Slepian-Wolf and Wyner-Ziv coding, achieve asymptotic compression optimality in distributed systems. However, these techniques are rarely used in sensor networks because of decoding complexity and prohibitively long code length. Moreover, the fundamental limits that arise from existing techniques are intractable to describe for a complicated network topology or when the objective of the system is to perform some computation on the data rather than to reproduce the data. This thesis bridges the technological gap between the needs of real-world systems and the optimistic bounds derived from asymptotic analysis. Specifically, we characterize fundamental trade-offs when the desired computation is incorporated into the compression design and the code length is one. To obtain both performance guarantees and achievable schemes, we use high-resolution quantization theory, which is complementary to the Shannon-theoretic analyses previously used to study distributed systems. We account for varied network topologies, such as those where sensors are allowed to collaborate or the communication links are heterogeneous. In these settings, a small amount of intersensor communication can provide a significant improvement in compression performance. As a result, this work suggests new compression principles and network design for modern distributed systems. Although the ideas in the thesis are motivated by current and future sensor network implementations, the framework applies to a wide range of signal processing questions. We draw connections between the fidelity criteria studied in the thesis and distortion measures used in perceptual coding. As a consequence, we determine the optimal quantizer for expected relative error (ERE), a measure that is widely useful but is often neglected in the source coding community. We further demonstrate that applying the ERE criterion to psychophysical models can explain the Weber-Fechner law, a longstanding hypothesis of how humans perceive the external world. Our results are consistent with the hypothesis that human perception is Bayesian optimal for information acquisition conditioned on limited cognitive resources, thereby supporting the notion that the brain is efficient at acquisition and adaptation.by John Z. Sun.Ph.D

The Other Asymptotic Theory Of Lossy Source Coding

Rate-distortion theory, as initiated by Shannon in his celebrated 1948 paper, is a well known theory of lossy source coding. It is an asymptotic theory in the sense that the performance it prescribes is approachable only in the limit as code dimension increases. Less well known is the "other" asymptotic theory of lossy source coding, which goes by the names of high-rate, high-resolution and asymptotic quantization theory. This theory prescribes the performance of codes with a given dimension and asymptotically large rate. The purpose of the present paper is to compare and contrast the two theories, and to highlight some recent results in high-rate quantization theory. It is the thesis of this paper that high-rate quantization theory has surpassed rate-distortion theory in its relevance to practical code design, because of its ability to identify key characteristics of good codes and to analyze the performance of codes with complexity-reducing structure