2 research outputs found
Risk Convergence of Centered Kernel Ridge Regression with Large Dimensional Data
This paper carries out a large dimensional analysis of a variation of kernel
ridge regression that we call \emph{centered kernel ridge regression} (CKRR),
also known in the literature as kernel ridge regression with offset. This
modified technique is obtained by accounting for the bias in the regression
problem resulting in the old kernel ridge regression but with \emph{centered}
kernels. The analysis is carried out under the assumption that the data is
drawn from a Gaussian distribution and heavily relies on tools from random
matrix theory (RMT). Under the regime in which the data dimension and the
training size grow infinitely large with fixed ratio and under some mild
assumptions controlling the data statistics, we show that both the empirical
and the prediction risks converge to a deterministic quantities that describe
in closed form fashion the performance of CKRR in terms of the data statistics
and dimensions. Inspired by this theoretical result, we subsequently build a
consistent estimator of the prediction risk based on the training data which
allows to optimally tune the design parameters. A key insight of the proposed
analysis is the fact that asymptotically a large class of kernels achieve the
same minimum prediction risk. This insight is validated with both synthetic and
real data.Comment: Submitted to IEEE Transactions on Signal Processin
Kernel regression in high dimensions: Refined analysis beyond double descent
In this paper, we provide a precise characterization of generalization
properties of high dimensional kernel ridge regression across the under- and
over-parameterized regimes, depending on whether the number of training data n
exceeds the feature dimension d. By establishing a bias-variance decomposition
of the expected excess risk, we show that, while the bias is (almost)
independent of d and monotonically decreases with n, the variance depends on n,
d and can be unimodal or monotonically decreasing under different
regularization schemes. Our refined analysis goes beyond the double descent
theory by showing that, depending on the data eigen-profile and the level of
regularization, the kernel regression risk curve can be a double-descent-like,
bell-shaped, or monotonic function of n. Experiments on synthetic and real data
are conducted to support our theoretical findings.Comment: This paper was accepted by AISTATS-202