3 research outputs found
Estimating Densities with Non-Parametric Exponential Families
We propose a novel approach for density estimation with exponential families
for the case when the true density may not fall within the chosen family. Our
approach augments the sufficient statistics with features designed to
accumulate probability mass in the neighborhood of the observed points,
resulting in a non-parametric model similar to kernel density estimators. We
show that under mild conditions, the resulting model uses only the sufficient
statistics if the density is within the chosen exponential family, and
asymptotically, it approximates densities outside of the chosen exponential
family. Using the proposed approach, we modify the exponential random graph
model, commonly used for modeling small-size graph distributions, to address
the well-known issue of model degeneracy.Comment: 22 pages, 5 figure
Generalization and Memorization: The Bias Potential Model
Models for learning probability distributions such as generative models and
density estimators behave quite differently from models for learning functions.
One example is found in the memorization phenomenon, namely the ultimate
convergence to the empirical distribution, that occurs in generative
adversarial networks (GANs). For this reason, the issue of generalization is
more subtle than that for supervised learning. For the bias potential model, we
show that dimension-independent generalization accuracy is achievable if early
stopping is adopted, despite that in the long term, the model either memorizes
the samples or diverges.Comment: Added new section on regularized mode
Estimating Densities with Non-Parametric Exponential Families
We propose a novel approach for density estimation with exponential families for the case when the true density may not fall within the chosen family. Our approach augments the sufficient statistics with features designed to accumulate probability mass in the neighborhood of the observed points, resulting in a nonparametric model similar to kernel density estimators. We show that under mild conditions, the resulting model uses only the sufficient statistics if the density is within the chosen exponential family, and asymptotically, it approximates densities outside of the chosen exponential family. Using the proposed approach, we modify the exponential random graph model, commonly used for modeling small-size graph distributions, The problem of density estimation is ubiquitous in machine learning and statistics. A typical approach would assume a parametric family for the distribution from which the observed data is drawn and estimate the parameters by fitting them to the data. Among the parametric families, exponential families play a prominent role, as maximum likelihood estimation from complete data for exponential families is asymptoticall