266 research outputs found
Kernel Exponential Family Estimation via Doubly Dual Embedding
We investigate penalized maximum log-likelihood estimation for exponential
family distributions whose natural parameter resides in a reproducing kernel
Hilbert space. Key to our approach is a novel technique, doubly dual embedding,
that avoids computation of the partition function. This technique also allows
the development of a flexible sampling strategy that amortizes the cost of
Monte-Carlo sampling in the inference stage. The resulting estimator can be
easily generalized to kernel conditional exponential families. We establish a
connection between kernel exponential family estimation and MMD-GANs, revealing
a new perspective for understanding GANs. Compared to the score matching based
estimators, the proposed method improves both memory and time efficiency while
enjoying stronger statistical properties, such as fully capturing smoothness in
its statistical convergence rate while the score matching estimator appears to
saturate. Finally, we show that the proposed estimator empirically outperforms
state-of-the-artComment: 22 pages, 20 figures; AISTATS 201
Compressed sensing performance bounds under Poisson noise
This paper describes performance bounds for compressed sensing (CS) where the
underlying sparse or compressible (sparsely approximable) signal is a vector of
nonnegative intensities whose measurements are corrupted by Poisson noise. In
this setting, standard CS techniques cannot be applied directly for several
reasons. First, the usual signal-independent and/or bounded noise models do not
apply to Poisson noise, which is non-additive and signal-dependent. Second, the
CS matrices typically considered are not feasible in real optical systems
because they do not adhere to important constraints, such as nonnegativity and
photon flux preservation. Third, the typical -- minimization
leads to overfitting in the high-intensity regions and oversmoothing in the
low-intensity areas. In this paper, we describe how a feasible positivity- and
flux-preserving sensing matrix can be constructed, and then analyze the
performance of a CS reconstruction approach for Poisson data that minimizes an
objective function consisting of a negative Poisson log likelihood term and a
penalty term which measures signal sparsity. We show that, as the overall
intensity of the underlying signal increases, an upper bound on the
reconstruction error decays at an appropriate rate (depending on the
compressibility of the signal), but that for a fixed signal intensity, the
signal-dependent part of the error bound actually grows with the number of
measurements or sensors. This surprising fact is both proved theoretically and
justified based on physical intuition.Comment: 12 pages, 3 pdf figures; accepted for publication in IEEE
Transactions on Signal Processin
Frontiers in Nonparametric Statistics
The goal of this workshop was to discuss recent developments of nonparametric statistical inference. A particular focus was on high dimensional statistics, semiparametrics, adaptation, nonparametric bayesian statistics, shape constraint estimation and statistical inverse problems. The close interaction of these issues with optimization, machine learning and inverse problems has been addressed as well
Direct Estimation of Information Divergence Using Nearest Neighbor Ratios
We propose a direct estimation method for R\'{e}nyi and f-divergence measures
based on a new graph theoretical interpretation. Suppose that we are given two
sample sets and , respectively with and samples, where
is a constant value. Considering the -nearest neighbor (-NN)
graph of in the joint data set , we show that the average powered
ratio of the number of points to the number of points among all -NN
points is proportional to R\'{e}nyi divergence of and densities. A
similar method can also be used to estimate f-divergence measures. We derive
bias and variance rates, and show that for the class of -H\"{o}lder
smooth functions, the estimator achieves the MSE rate of
. Furthermore, by using a weighted ensemble
estimation technique, for density functions with continuous and bounded
derivatives of up to the order , and some extra conditions at the support
set boundary, we derive an ensemble estimator that achieves the parametric MSE
rate of . Our estimators are more computationally tractable than other
competing estimators, which makes them appealing in many practical
applications.Comment: 2017 IEEE International Symposium on Information Theory (ISIT
Semi-Supervised Learning of Class Balance under Class-Prior Change by Distribution Matching
In real-world classification problems, the class balance in the training
dataset does not necessarily reflect that of the test dataset, which can cause
significant estimation bias. If the class ratio of the test dataset is known,
instance re-weighting or resampling allows systematical bias correction.
However, learning the class ratio of the test dataset is challenging when no
labeled data is available from the test domain. In this paper, we propose to
estimate the class ratio in the test dataset by matching probability
distributions of training and test input data. We demonstrate the utility of
the proposed approach through experiments.Comment: ICML201
- …