4,093 research outputs found
A Bit of Information Theory, and the Data Augmentation Algorithm Converges
The data augmentation (DA) algorithm is a simple and powerful tool in
statistical computing. In this note basic information theory is used to prove a
nontrivial convergence theorem for the DA algorithm
Expectation-maximization for logistic regression
We present a family of expectation-maximization (EM) algorithms for binary
and negative-binomial logistic regression, drawing a sharp connection with the
variational-Bayes algorithm of Jaakkola and Jordan (2000). Indeed, our results
allow a version of this variational-Bayes approach to be re-interpreted as a
true EM algorithm. We study several interesting features of the algorithm, and
of this previously unrecognized connection with variational Bayes. We also
generalize the approach to sparsity-promoting priors, and to an online method
whose convergence properties are easily established. This latter method
compares favorably with stochastic-gradient descent in situations with marked
collinearity
Parameter Expansion and Efficient Inference
This EM review article focuses on parameter expansion, a simple technique
introduced in the PX-EM algorithm to make EM converge faster while maintaining
its simplicity and stability. The primary objective concerns the connection
between parameter expansion and efficient inference. It reviews the statistical
interpretation of the PX-EM algorithm, in terms of efficient inference via bias
reduction, and further unfolds the PX-EM mystery by looking at PX-EM from
different perspectives. In addition, it briefly discusses potential
applications of parameter expansion to statistical inference and the broader
impact of statistical thinking on understanding and developing other iterative
optimization algorithms.Comment: Published in at http://dx.doi.org/10.1214/10-STS348 the Statistical
Science (http://www.imstat.org/sts/) by the Institute of Mathematical
Statistics (http://www.imstat.org
Machine Learning Models that Remember Too Much
Machine learning (ML) is becoming a commodity. Numerous ML frameworks and
services are available to data holders who are not ML experts but want to train
predictive models on their data. It is important that ML models trained on
sensitive inputs (e.g., personal images or documents) not leak too much
information about the training data.
We consider a malicious ML provider who supplies model-training code to the
data holder, does not observe the training, but then obtains white- or
black-box access to the resulting model. In this setting, we design and
implement practical algorithms, some of them very similar to standard ML
techniques such as regularization and data augmentation, that "memorize"
information about the training dataset in the model yet the model is as
accurate and predictive as a conventionally trained model. We then explain how
the adversary can extract memorized information from the model.
We evaluate our techniques on standard ML tasks for image classification
(CIFAR10), face recognition (LFW and FaceScrub), and text analysis (20
Newsgroups and IMDB). In all cases, we show how our algorithms create models
that have high predictive power yet allow accurate extraction of subsets of
their training data
- …