CORE
🇺🇦Â
 make metadata, not war
Services
Research
Services overview
Explore all CORE services
Access to raw data
API
Dataset
FastSync
Content discovery
Recommender
Discovery
OAI identifiers
OAI Resolver
Managing content
Dashboard
Bespoke contracts
Consultancy services
Support us
Support us
Membership
Sponsorship
Community governance
Advisory Board
Board of supporters
Research network
About
About us
Our mission
Team
Blog
FAQs
Contact us
Near-Interpolators: Rapid Norm Growth and the Trade-Off between Interpolation and Generalization
Authors
Wei Hu
Rishi Sonthalia
Yutong Wang
Publication date
11 March 2024
Publisher
View
on
arXiv
Abstract
We study the generalization capability of nearly-interpolating linear regressors:
β
\boldsymbol{\beta}
β
's whose training error
Ï„
\tau
Ï„
is positive but small, i.e., below the noise floor. Under a random matrix theoretic assumption on the data distribution and an eigendecay assumption on the data covariance matrix
Σ
\boldsymbol{\Sigma}
Σ
, we demonstrate that any near-interpolator exhibits rapid norm growth: for
Ï„
\tau
Ï„
fixed,
β
\boldsymbol{\beta}
β
has squared
â„“
2
\ell_2
â„“
2
​
-norm
E
[
∥
β
∥
2
2
]
=
Ω
(
n
α
)
\mathbb{E}[\|{\boldsymbol{\beta}}\|_{2}^{2}] = \Omega(n^{\alpha})
E
[
∥
β
∥
2
2
​
]
=
Ω
(
n
α
)
where
n
n
n
is the number of samples and
α
>
1
\alpha >1
α
>
1
is the exponent of the eigendecay, i.e.,
λ
i
(
Σ
)
∼
i
−
α
\lambda_i(\boldsymbol{\Sigma}) \sim i^{-\alpha}
λ
i
​
(
Σ
)
∼
i
−
α
. This implies that existing data-independent norm-based bounds are necessarily loose. On the other hand, in the same regime we precisely characterize the asymptotic trade-off between interpolation and generalization. Our characterization reveals that larger norm scaling exponents
α
\alpha
α
correspond to worse trade-offs between interpolation and generalization. We verify empirically that a similar phenomenon holds for nearly-interpolating shallow neural networks.Comment: AISTATS 202
Similar works
Full text
Available Versions
arXiv.org e-Print Archive
See this paper in CORE
Go to the repository landing page
Download from data provider
oai:arXiv.org:2403.07264
Last time updated on 28/09/2024