Support Recovery with Sparsely Sampled Free Random Matrices

Consider a Bernoulli-Gaussian complex $n$-vector whose components are $V_i = X_i B_i$, with X_i \sim \Cc\Nc(0,\Pc_x) and binary $B_i$ mutually independent and iid across $i$. This random $q$-sparse vector is multiplied by a square random matrix \Um, and a randomly chosen subset, of average size $n p$, $p \in [0,1]$, of the resulting vector components is then observed in additive Gaussian noise. We extend the scope of conventional noisy compressive sampling models where \Um is typically %A16 the identity or a matrix with iid components, to allow \Um satisfying a certain freeness condition. This class of matrices encompasses Haar matrices and other unitarily invariant matrices. We use the replica method and the decoupling principle of Guo and Verd\'u, as well as a number of information theoretic bounds, to study the input-output mutual information and the support recovery error rate in the limit of $n \to \infty$. We also extend the scope of the large deviation approach of Rangan, Fletcher and Goyal and characterize the performance of a class of estimators encompassing thresholded linear MMSE and $\ell_1$ relaxation

Rigorous Dynamics and Consistent Estimation in Arbitrarily Conditioned Linear Systems

The problem of estimating a random vector x from noisy linear measurements y = A x + w with unknown parameters on the distributions of x and w, which must also be learned, arises in a wide range of statistical learning and linear inverse problems. We show that a computationally simple iterative message-passing algorithm can provably obtain asymptotically consistent estimates in a certain high-dimensional large-system limit (LSL) under very general parameterizations. Previous message passing techniques have required i.i.d. sub-Gaussian A matrices and often fail when the matrix is ill-conditioned. The proposed algorithm, called adaptive vector approximate message passing (Adaptive VAMP) with auto-tuning, applies to all right-rotationally random A. Importantly, this class includes matrices with arbitrarily poor conditioning. We show that the parameter estimates and mean squared error (MSE) of x in each iteration converge to deterministic limits that can be precisely predicted by a simple set of state evolution (SE) equations. In addition, a simple testable condition is provided in which the MSE matches the Bayes-optimal value predicted by the replica method. The paper thus provides a computationally simple method with provable guarantees of optimality and consistency over a large class of linear inverse problems

Random positive operator valued measures

We introduce several notions of random positive operator valued measures (POVMs), and we prove that some of them are equivalent. We then study statistical properties of the effect operators for the canonical examples, obtaining limiting eigenvalue distributions with the help of free probability theory. Similarly, we obtain the large system limit for several quantities of interest in quantum information theory, such as the sharpness, the noise content, and the probability range. Finally, we study different compatibility criteria, and we compare them for generic POVMs.Comment: 33 pages. v3: small modification

Multivariate Density Estimation via Adaptive Partitioning (I): Sieve MLE

We study a non-parametric approach to multivariate density estimation. The estimators are piecewise constant density functions supported by binary partitions. The partition of the sample space is learned by maximizing the likelihood of the corresponding histogram on that partition. We analyze the convergence rate of the sieve maximum likelihood estimator, and reach a conclusion that for a relatively rich class of density functions the rate does not directly depend on the dimension. This suggests that, under certain conditions, this method is immune to the curse of dimensionality, in the sense that it is possible to get close to the parametric rate even in high dimensions. We also apply this method to several special cases, and calculate the explicit convergence rates respectively

Online Estimation and Adaptive Control for a Class of History Dependent Functional Differential Equations

This paper presents sufficient conditions for the convergence of online estimation methods and the stability of adaptive control strategies for a class of history dependent, functional differential equations. The study is motivated by the increasing interest in estimation and control techniques for robotic systems whose governing equations include history dependent nonlinearities. The functional differential equations in this paper are constructed using integral operators that depend on distributed parameters. As a consequence the resulting estimation and control equations are examples of distributed parameter systems whose states and distributed parameters evolve in finite and infinite dimensional spaces, respectively. suWell-posedness, existence, and uniqueness are discussed for the class of fully actuated robotic systems with history dependent forces in their governing equation of motion. By deriving rates of approximation for the class of history dependent operators in this paper, sufficient conditions are derived that guarantee that finite dimensional approximations of the online estimation equations converge to the solution of the infinite dimensional, distributed parameter system. The convergence and stability of a sliding mode adaptive control strategy for the history dependent, functional differential equations is established using Barbalat's lemma.Comment: submitted to Nonlinear Dynamic

Expressibility and entangling capability of parameterized quantum circuits for hybrid quantum-classical algorithms

Parameterized quantum circuits play an essential role in the performance of many variational hybrid quantum-classical (HQC) algorithms. One challenge in implementing such algorithms is to choose an effective circuit that well represents the solution space while maintaining a low circuit depth and number of parameters. To characterize and identify expressible, yet compact, parameterized circuits, we propose several descriptors, including measures of expressibility and entangling capability, that can be statistically estimated from classical simulations of parameterized quantum circuits. We compute these descriptors for different circuit structures, varying the qubit connectivity and selection of gates. From our simulations, we identify circuit fragments that perform well with respect to the descriptors. In particular, we quantify the substantial improvement in performance of two-qubit gates in a ring or all-to-all connected arrangement compared to that of those on a line. Furthermore, we quantify the improvement in expressibility and entangling capability achieved by sequences of controlled X-rotation gates compared to sequences of controlled Z-rotation gates. In addition, we investigate how expressibility "saturates" with increased circuit depth, finding that the rate and saturated-value appear to be distinguishing features of a parameterized quantum circuit template. While the correlation between each descriptor and performance of an algorithm remains to be investigated, methods and results from this study can be useful for both algorithm development and design of experiments for general variational HQC algorithms

Bayesian inverse problems with l1l_1 priors: a Randomize-then-Optimize approach

Prior distributions for Bayesian inference that rely on the $l_1$-norm of the parameters are of considerable interest, in part because they promote parameter fields with less regularity than Gaussian priors (e.g., discontinuities and blockiness). These $l_1$-type priors include the total variation (TV) prior and the Besov $B^s_{1,1}$ space prior, and in general yield non-Gaussian posterior distributions. Sampling from these posteriors is challenging, particularly in the inverse problem setting where the parameter space is high-dimensional and the forward problem may be nonlinear. This paper extends the randomize-then-optimize (RTO) method, an optimization-based sampling algorithm developed for Bayesian inverse problems with Gaussian priors, to inverse problems with $l_1$-type priors. We use a variable transformation to convert an $l_1$-type prior to a standard Gaussian prior, such that the posterior distribution of the transformed parameters is amenable to Metropolized sampling via RTO. We demonstrate this approach on several deconvolution problems and an elliptic PDE inverse problem, using TV or Besov $B^s_{1,1}$ space priors. Our results show that the transformed RTO algorithm characterizes the correct posterior distribution and can be more efficient than other sampling algorithms. The variable transformation can also be extended to other non-Gaussian priors.Comment: Preprint 24 pages, 13 figures. v1 submitted to SIAM Journal on Scientific Computing on July 5, 201

Wavelet Variance for Random Fields: an M-Estimation Framework

We present a general M-estimation framework for inference on the wavelet variance. This framework generalizes the results on the scale-wise properties of the standard estimator and extends them to deliver the joint asymptotic properties of the estimated wavelet variance vector. Moreover, this is achieved by extending the estimation of the wavelet variance to multidimensional random fields and by stating the necessary conditions for these properties to hold when the size of the wavelet variance vector goes to infinity with the sample size. Finally, these results generally hold when using bounded estimating functions thereby delivering a robust framework for the estimation of this quantity which improves over existing methods both in terms of asymptotic properties and in terms of its finite sample performance. The proposed estimator is investigated in simulation studies and different applications highlighting its good properties

Inference in Deep Networks in High Dimensions

Deep generative networks provide a powerful tool for modeling complex data in a wide range of applications. In inverse problems that use these networks as generative priors on data, one must often perform inference of the inputs of the networks from the outputs. Inference is also required for sampling during stochastic training on these generative models. This paper considers inference in a deep stochastic neural network where the parameters (e.g., weights, biases and activation functions) are known and the problem is to estimate the values of the input and hidden units from the output. While several approximate algorithms have been proposed for this task, there are few analytic tools that can provide rigorous guarantees in the reconstruction error. This work presents a novel and computationally tractable output-to-input inference method called Multi-Layer Vector Approximate Message Passing (ML-VAMP). The proposed algorithm, derived from expectation propagation, extends earlier AMP methods that are known to achieve the replica predictions for optimality in simple linear inverse problems. Our main contribution shows that the mean-squared error (MSE) of ML-VAMP can be exactly predicted in a certain large system limit (LSL) where the numbers of layers is fixed and weight matrices are random and orthogonally-invariant with dimensions that grow to infinity. ML-VAMP is thus a principled method for output-to-input inference in deep networks with a rigorous and precise performance achievability result in high dimensions.Comment: 27 page

Artificial Neural Network in Cosmic Landscape

In this paper we propose that artificial neural network, the basis of machine learning, is useful to generate the inflationary landscape from a cosmological point of view. Traditional numerical simulations of a global cosmic landscape typically need an exponential complexity when the number of fields is large. However, a basic application of artificial neural network could solve the problem based on the universal approximation theorem of the multilayer perceptron. A toy model in inflation with multiple light fields is investigated numerically as an example of such an application.Comment: v2, add some new content