2 research outputs found
Learning Theory of Distributed Regression with Bias Corrected Regularization Kernel Network
Distributed learning is an effective way to analyze big data. In distributed
regression, a typical approach is to divide the big data into multiple blocks,
apply a base regression algorithm on each of them, and then simply average the
output functions learnt from these blocks. Since the average process will
decrease the variance, not the bias, bias correction is expected to improve the
learning performance if the base regression algorithm is a biased one.
Regularization kernel network is an effective and widely used method for
nonlinear regression analysis. In this paper we will investigate a bias
corrected version of regularization kernel network. We derive the error bounds
when it is applied to a single data set and when it is applied as a base
algorithm in distributed regression. We show that, under certain appropriate
conditions, the optimal learning rates can be reached in both situations
Optimal Rates of Distributed Regression with Imperfect Kernels
Distributed machine learning systems have been receiving increasing
attentions for their efficiency to process large scale data. Many distributed
frameworks have been proposed for different machine learning tasks. In this
paper, we study the distributed kernel regression via the divide and conquer
approach. This approach has been proved asymptotically minimax optimal if the
kernel is perfectly selected so that the true regression function lies in the
associated reproducing kernel Hilbert space. However, this is usually, if not
always, impractical because kernels that can only be selected via prior
knowledge or a tuning process are hardly perfect. Instead it is more common
that the kernel is good enough but imperfect in the sense that the true
regression can be well approximated by but does not lie exactly in the kernel
space. We show distributed kernel regression can still achieves capacity
independent optimal rate in this case. To this end, we first establish a
general framework that allows to analyze distributed regression with response
weighted base algorithms by bounding the error of such algorithms on a single
data set, provided that the error bounds has factored the impact of the
unexplained variance of the response variable. Then we perform a leave one out
analysis of the kernel ridge regression and bias corrected kernel ridge
regression, which in combination with the aforementioned framework allows us to
derive sharp error bounds and capacity independent optimal rates for the
associated distributed kernel regression algorithms. As a byproduct of the
thorough analysis, we also prove the kernel ridge regression can achieve rates
faster than (where is the sample size) in the noise free setting
which, to our best knowledge, are first observed and novel in regression
learning.Comment: 2 figure