38,595 research outputs found
LWPR: A Scalable Method for Incremental Online Learning in High Dimensions
Locally weighted projection regression (LWPR) is a new algorithm for incremental nonlinear func-
tion approximation in high dimensional spaces with redundant and irrelevant input dimensions. At
its core, it employs nonparametric regression with locally linear models. In order to stay computa-
tionally efficient and numerically robust, each local model performs the regression analysis with a
small number of univariate regressions in selected directions in input space in the spirit of partial
least squares regression. We discuss when and how local learning techniques can successfully work
in high dimensional spaces and compare various techniques for local dimensionality reduction before
finally deriving the LWPR algorithm. The properties of LWPR are that it i) learns rapidly with
second order learning methods based on incremental training, ii) uses statistically sound stochastic
leave-one-out cross validation for learning without the need to memorize training data, iii) adjusts
its weighting kernels based only on local information in order to minimize the danger of negative
interference of incremental learning, iv) has a computational complexity that is linear in the num-
ber of inputs, and v) can deal with a large number of - possibly redundant - inputs, as shown in
various empirical evaluations with up to 50 dimensional data sets. For a probabilistic interpreta-
tion, predictive variance and confidence intervals are derived. To our knowledge, LWPR is the first
truly incremental spatially localized learning method that can successfully and efficiently operate
in very high dimensional spaces
Private Incremental Regression
Data is continuously generated by modern data sources, and a recent challenge
in machine learning has been to develop techniques that perform well in an
incremental (streaming) setting. In this paper, we investigate the problem of
private machine learning, where as common in practice, the data is not given at
once, but rather arrives incrementally over time.
We introduce the problems of private incremental ERM and private incremental
regression where the general goal is to always maintain a good empirical risk
minimizer for the history observed under differential privacy. Our first
contribution is a generic transformation of private batch ERM mechanisms into
private incremental ERM mechanisms, based on a simple idea of invoking the
private batch ERM procedure at some regular time intervals. We take this
construction as a baseline for comparison. We then provide two mechanisms for
the private incremental regression problem. Our first mechanism is based on
privately constructing a noisy incremental gradient function, which is then
used in a modified projected gradient procedure at every timestep. This
mechanism has an excess empirical risk of , where is the
dimensionality of the data. While from the results of [Bassily et al. 2014]
this bound is tight in the worst-case, we show that certain geometric
properties of the input and constraint set can be used to derive significantly
better results for certain interesting regression problems.Comment: To appear in PODS 201
- …