1,871 research outputs found
Fundamental performance limits for ideal decoders in high-dimensional linear inverse problems
This paper focuses on characterizing the fundamental performance limits that
can be expected from an ideal decoder given a general model, ie, a general
subset of "simple" vectors of interest. First, we extend the so-called notion
of instance optimality of a decoder to settings where one only wishes to
reconstruct some part of the original high dimensional vector from a
low-dimensional observation. This covers practical settings such as medical
imaging of a region of interest, or audio source separation when one is only
interested in estimating the contribution of a specific instrument to a musical
recording. We define instance optimality relatively to a model much beyond the
traditional framework of sparse recovery, and characterize the existence of an
instance optimal decoder in terms of joint properties of the model and the
considered linear operator. Noiseless and noise-robust settings are both
considered. We show somewhat surprisingly that the existence of noise-aware
instance optimal decoders for all noise levels implies the existence of a
noise-blind decoder. A consequence of our results is that for models that are
rich enough to contain an orthonormal basis, the existence of an L2/L2 instance
optimal decoder is only possible when the linear operator is not substantially
dimension-reducing. This covers well-known cases (sparse vectors, low-rank
matrices) as well as a number of seemingly new situations (structured sparsity
and sparse inverse covariance matrices for instance). We exhibit an
operator-dependent norm which, under a model-specific generalization of the
Restricted Isometry Property (RIP), always yields a feasible instance
optimality property. This norm can be upper bounded by an atomic norm relative
to the considered model.Comment: To appear in IEEE Transactions on Information Theor
On the existence of optimal multi-valued decoders and their accuracy bounds for undersampled inverse problems
Undersampled inverse problems occur everywhere in the sciences including
medical imaging, radar, astronomy etc., yielding underdetermined linear or
non-linear reconstruction problems. There are now a myriad of techniques to
design decoders that can tackle such problems, ranging from optimization based
approaches, such as compressed sensing, to deep learning (DL), and variants in
between the two techniques. The variety of methods begs for a unifying approach
to determine the existence of optimal decoders and fundamental accuracy bounds,
in order to facilitate a theoretical and empirical understanding of the
performance of existing and future methods. Such a theory must allow for both
single-valued and multi-valued decoders, as underdetermined inverse problems
typically have multiple solutions. Indeed, multi-valued decoders arise due to
non-uniqueness of minimizers in optimisation problems, such as in compressed
sensing, and for DL based decoders in generative adversarial models, such as
diffusion models and ensemble models. In this work we provide a framework for
assessing the lowest possible reconstruction accuracy in terms of worst- and
average-case errors. The universal bounds bounds only depend on the measurement
model , the model class and the noise
model . For linear these bounds depend on its kernel, and in
the non-linear case the concept of kernel is generalized for undersampled
settings. Additionally, we provide multi-valued variational solutions that
obtain the lowest possible reconstruction error
Compressible Distributions for High-dimensional Statistics
We develop a principled way of identifying probability distributions whose
independent and identically distributed (iid) realizations are compressible,
i.e., can be well-approximated as sparse. We focus on Gaussian random
underdetermined linear regression (GULR) problems, where compressibility is
known to ensure the success of estimators exploiting sparse regularization. We
prove that many distributions revolving around maximum a posteriori (MAP)
interpretation of sparse regularized estimators are in fact incompressible, in
the limit of large problem sizes. A highlight is the Laplace distribution and
regularized estimators such as the Lasso and Basis Pursuit
denoising. To establish this result, we identify non-trivial undersampling
regions in GULR where the simple least squares solution almost surely
outperforms an oracle sparse solution, when the data is generated from the
Laplace distribution. We provide simple rules of thumb to characterize classes
of compressible (respectively incompressible) distributions based on their
second and fourth moments. Generalized Gaussians and generalized Pareto
distributions serve as running examples for concreteness.Comment: Was previously entitled "Compressible priors for high-dimensional
statistics"; IEEE Transactions on Information Theory (2012
Dynamical Systems in Spiking Neuromorphic Hardware
Dynamical systems are universal computers. They can perceive stimuli, remember, learn from feedback, plan sequences of actions, and coordinate complex behavioural responses. The Neural Engineering Framework (NEF) provides a general recipe to formulate models of such systems as coupled sets of nonlinear differential equations and compile them onto recurrently connected spiking neural networks â akin to a programming language for spiking models of computation. The Nengo software ecosystem supports the NEF and compiles such models onto neuromorphic hardware. In this thesis, we analyze the theory driving the success of the NEF, and expose several core principles underpinning its correctness, scalability, completeness, robustness, and extensibility. We also derive novel theoretical extensions to the framework that enable it to far more effectively leverage a wide variety of dynamics in digital hardware, and to exploit the device-level physics in analog hardware. At the same time, we propose a novel set of spiking algorithms that recruit an optimal nonlinear encoding of time, which we call the Delay Network (DN). Backpropagation across stacked layers of DNs dramatically outperforms stacked Long Short-Term Memory (LSTM) networksâa state-of-the-art deep recurrent architectureâin accuracy and training time, on a continuous-time memory task, and a chaotic time-series prediction benchmark. The basic component of this network is shown to function on state-of-the-art spiking neuromorphic hardware including Braindrop and Loihi. This implementation approaches the energy-efficiency of the human brain in the former case, and the precision of conventional computation in the latter case
Single-Shot Decoding of Linear Rate LDPC Quantum Codes With High Performance
We construct and analyze a family of low-density parity check (LDPC) quantum codes with a linear encoding rate, distance scaling as nϔ for ϔ>0 and efficient decoding schemes. The code family is based on tessellations of closed, four-dimensional, hyperbolic manifolds, as first suggested by Guth and Lubotzky. The main contribution of this work is the construction of suitable manifolds via finite presentations of Coxeter groups, their linear representations over Galois fields and topological coverings. We establish a lower bound on the encoding rate k/n of 13/72=0.180⊠and we show that the bound is tight for the examples that we construct. Numerical simulations give evidence that parallelizable decoding schemes of low computational complexity suffice to obtain high performance. These decoding schemes can deal with syndrome noise, so that parity check measurements do not have to be repeated to decode. Our data is consistent with a threshold of around 4% in the phenomenological noise model with syndrome noise in the single-shot regime
- âŠ