4 research outputs found
Locally Adaptive and Differentiable Regression
Over-parameterized models like deep nets and random forests have become very
popular in machine learning. However, the natural goals of continuity and
differentiability, common in regression models, are now often ignored in modern
overparametrized, locally-adaptive models. We propose a general framework to
construct a global continuous and differentiable model based on a weighted
average of locally learned models in corresponding local regions. This model is
competitive in dealing with data with different densities or scales of function
values in different local regions. We demonstrate that when we mix kernel ridge
and polynomial regression terms in the local models, and stitch them together
continuously, we achieve faster statistical convergence in theory and improved
performance in various practical settings
Predictive Model Degrees of Freedom in Linear Regression
Overparametrized interpolating models have drawn increasing attention from
machine learning. Some recent studies suggest that regularized interpolating
models can generalize well. This phenomenon seemingly contradicts the
conventional wisdom that interpolation tends to overfit the data and performs
poorly on test data. Further, it appears to defy the bias-variance trade-off.
As one of the shortcomings of the existing theory, the classical notion of
model degrees of freedom fails to explain the intrinsic difference among the
interpolating models since it focuses on estimation of in-sample prediction
error. This motivates an alternative measure of model complexity which can
differentiate those interpolating models and take different test points into
account. In particular, we propose a measure with a proper adjustment based on
the squared covariance between the predictions and observations. Our analysis
with least squares method reveals some interesting properties of the measure,
which can reconcile the "double descent" phenomenon with the classical theory.
This opens doors to an extended definition of model degrees of freedom in
modern predictive settings.Comment: 47 pages, 18 figure