Nearly Optimal Sparse Group Testing

Group testing is the process of pooling arbitrary subsets from a set of $n$ items so as to identify, with a minimal number of tests, a "small" subset of $d$ defective items. In "classical" non-adaptive group testing, it is known that when $d$ is substantially smaller than $n$, $\Theta(d\log(n))$ tests are both information-theoretically necessary and sufficient to guarantee recovery with high probability. Group testing schemes in the literature meeting this bound require most items to be tested $\Omega(\log(n))$ times, and most tests to incorporate $\Omega(n/d)$ items. Motivated by physical considerations, we study group testing models in which the testing procedure is constrained to be "sparse". Specifically, we consider (separately) scenarios in which (a) items are finitely divisible and hence may participate in at most $\gamma \in o(\log(n))$ tests; or (b) tests are size-constrained to pool no more than $\rho \in o(n/d)$items per test. For both scenarios we provide information-theoretic lower bounds on the number of tests required to guarantee high probability recovery. In both scenarios we provide both randomized constructions (under both $\epsilon$-error and zero-error reconstruction guarantees) and explicit constructions of designs with computationally efficient reconstruction algorithms that require a number of tests that are optimal up to constant or small polynomial factors in some regimes of $n, d, \gamma,$ and $\rho$. The randomized design/reconstruction algorithm in the $\rho$-sized test scenario is universal -- independent of the value of $d$, as long as $\rho \in o(n/d)$. We also investigate the effect of unreliability/noise in test outcomes. For the full abstract, please see the full text PDF

Group testing:an information theory perspective

The group testing problem concerns discovering a small number of defective items within a large population by performing tests on pools of items. A test is positive if the pool contains at least one defective, and negative if it contains no defectives. This is a sparse inference problem with a combinatorial flavour, with applications in medical testing, biology, telecommunications, information technology, data science, and more. In this monograph, we survey recent developments in the group testing problem from an information-theoretic perspective. We cover several related developments: efficient algorithms with practical storage and computation requirements, achievability bounds for optimal decoding methods, and algorithm-independent converse bounds. We assess the theoretical guarantees not only in terms of scaling laws, but also in terms of the constant factors, leading to the notion of the {\em rate} of group testing, indicating the amount of information learned per test. Considering both noiseless and noisy settings, we identify several regimes where existing algorithms are provably optimal or near-optimal, as well as regimes where there remains greater potential for improvement. In addition, we survey results concerning a number of variations on the standard group testing problem, including partial recovery criteria, adaptive algorithms with a limited number of stages, constrained test designs, and sublinear-time algorithms.Comment: Survey paper, 140 pages, 19 figures. To be published in Foundations and Trends in Communications and Information Theor

Poisson Group Testing: A Probabilistic Model for Boolean Compressed Sensing

We introduce a novel probabilistic group testing framework, termed Poisson group testing, in which the number of defectives follows a right-truncated Poisson distribution. The Poisson model has a number of new applications, including dynamic testing with diminishing relative rates of defectives. We consider both nonadaptive and semi-adaptive identification methods. For nonadaptive methods, we derive a lower bound on the number of tests required to identify the defectives with a probability of error that asymptotically converges to zero; in addition, we propose test matrix constructions for which the number of tests closely matches the lower bound. For semi-adaptive methods, we describe a lower bound on the expected number of tests required to identify the defectives with zero error probability. In addition, we propose a stage-wise reconstruction algorithm for which the expected number of tests is only a constant factor away from the lower bound. The methods rely only on an estimate of the average number of defectives, rather than on the individual probabilities of subjects being defective

Algorithms to Exploit Data Sparsity

While data in the real world is very high-dimensional, it generally has some underlying structure; for instance, if we think of an image as a set of pixels with associated color values, most possible settings of color values correspond to something more like random noise than what we typically think of as a picture. With an appropriate transformation of basis, this underlying structure can often be converted into sparsity in data, giving an equivalent representation of the data where the magnitude is large in only a few directions relative to the ambient dimension. This motivates a variety of theoretical questions around designing algorithms that can exploit this data sparsity to achieve better performance than what would be possible naively, and in this thesis we tackle several such questions.We first examine the question of simply approximating the level of sparsity of a signal under several different measurement models, a natural first step if the sparsity is to be exploited by other algorithms. Second, we look at a particular sparse signal recovery problem called nonadaptive probabilistic group testing, and investigate the question of exactly how sparse the signal needs to be before the methods used for recovering sparse signals outperform those used for non-sparse signals. Third, we prove novel upper bounds on the number of measurements needed to recover a sparse signal in the universal one-bit compressed sensing model of sparse signal recovery. Fourth, we give some approximations of an information-theoretic quantity called the index coding rate of a network modeled by a graph, in the special case that the graph is sparse or otherwise highly structured. For each of the problems considered, we also discuss some remaining open questions and conjectures, as well as possible directions towards their solutions

Noisy Non-Adaptive Group Testing: A (Near-)Definite Defectives Approach

The group testing problem consists of determining a small set of defective items from a larger set of items based on a number of possibly-noisy tests, and is relevant in applications such as medical testing, communication protocols, pattern matching, and many more. We study the noisy version of the problem, where the output of each standard noiseless group test is subject to independent noise, corresponding to passing the noiseless result through a binary channel. We introduce a class of algorithms that we refer to as Near-Definite Defectives (NDD), and study bounds on the required number of tests for vanishing error probability under Bernoulli random test designs. In addition, we study algorithm-independent converse results, giving lower bounds on the required number of tests under Bernoulli test designs. Under reverse $Z$-channel noise, the achievable rates and converse results match in a broad range of sparsity regimes, and under $Z$-channel noise, the two match in a narrower range of dense/low-noise regimes. We observe that although these two channels have the same Shannon capacity when viewed as a communication channel, they can behave quite differently when it comes to group testing. Finally, we extend our analysis of these noise models to the symmetric noise model, and show improvements over the best known existing bounds in broad scaling regimes.Comment: Submitted to IEEE Transactions on Information Theor

Estimation of Sparsity via Simple Measurements

We consider several related problems of estimating the 'sparsity' or number of nonzero elements $d$ in a length $n$ vector $\mathbf{x}$ by observing only $\mathbf{b} = M \odot \mathbf{x}$, where $M$ is a predesigned test matrix independent of $\mathbf{x}$, and the operation $\odot$ varies between problems. We aim to provide a $\Delta$-approximation of sparsity for some constant $\Delta$ with a minimal number of measurements (rows of $M$). This framework generalizes multiple problems, such as estimation of sparsity in group testing and compressed sensing. We use techniques from coding theory as well as probabilistic methods to show that $O(D \log D \log n)$ rows are sufficient when the operation $\odot$ is logical OR (i.e., group testing), and nearly this many are necessary, where $D$ is a known upper bound on $d$. When instead the operation $\odot$ is multiplication over $\mathbb{R}$ or a finite field $\mathbb{F}_q$, we show that respectively $\Theta(D)$ and $\Theta(D \log_q \frac{n}{D})$ measurements are necessary and sufficient.Comment: 13 pages; shortened version presented at ISIT 201

Engineering Competitive and Query-Optimal Minimal-Adaptive Randomized Group Testing Strategies

Suppose that given is a collection of $n$ elements where $d$ of them are \emph{defective}. We can query an arbitrarily chosen subset of elements which returns Yes if the subset contains at least one defective and No if the subset is free of defectives. The problem of group testing is to identify the defectives with a minimum number of such queries. By the information-theoretic lower bound at least $\log_2 \binom {n}{d} \approx d\log_2 (\frac{n}{d}) \approx d\log_2 n$ queries are needed. Using adaptive group testing, i.e., asking one query at a time, the lower bound can be easily achieved. However, strategies are preferred that work in a fixed small number of stages, where queries in a stage are asked in parallel. A group testing strategy is called \emph{competitive} if it works for completely unknown $d$ and requires only $O(d\log_2 n)$ queries. Usually competitive group testing is based on sequential queries. We have shown that actually competitive group testing with expected $O(d\log_2 n)$ queries is possible in only $2$ or $3$ stages. Then we have focused on minimizing the hidden constant factor in the query number and proposed a systematic approach for this purpose. Another main result is related to the design of query-optimal and minimal-adaptive strategies. We have shown that a $2$-stage randomized strategy with prescribed success probability can asymptotically achieve the information-theoretic lower bound for $d \ll n$ and growing much slower than $n$. Similarly, we can approach the entropy lower bound in $4$ stages when $d=o(n)$