51 research outputs found
Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension
The success of over-parameterized neural networks trained to near-zero
training error has caused great interest in the phenomenon of benign
overfitting, where estimators are statistically consistent even though they
interpolate noisy training data. While benign overfitting in fixed dimension
has been established for some learning methods, current literature suggests
that for regression with typical kernel methods and wide neural networks,
benign overfitting requires a high-dimensional setting where the dimension
grows with the sample size. In this paper, we show that the smoothness of the
estimators, and not the dimension, is the key: benign overfitting is possible
if and only if the estimator's derivatives are large enough. We generalize
existing inconsistency results to non-interpolating models and more kernels to
show that benign overfitting with moderate derivatives is impossible in fixed
dimension. Conversely, we show that rate-optimal benign overfitting is possible
for regression with a sequence of spiky-smooth kernels with large derivatives.
Using neural tangent kernels, we translate our results to wide neural networks.
We prove that while infinite-width networks do not overfit benignly with the
ReLU activation, this can be fixed by adding small high-frequency fluctuations
to the activation function. Our experiments verify that such neural networks,
while overfitting, can indeed generalize well even on low-dimensional data
sets.Comment: We provide Python code to reproduce all of our experimental results
at https://github.com/moritzhaas/mind-the-spike
Compressive Wave Computation
This paper considers large-scale simulations of wave propagation phenomena.
We argue that it is possible to accurately compute a wavefield by decomposing
it onto a largely incomplete set of eigenfunctions of the Helmholtz operator,
chosen at random, and that this provides a natural way of parallelizing wave
simulations for memory-intensive applications.
This paper shows that L1-Helmholtz recovery makes sense for wave computation,
and identifies a regime in which it is provably effective: the one-dimensional
wave equation with coefficients of small bounded variation. Under suitable
assumptions we show that the number of eigenfunctions needed to evolve a sparse
wavefield defined on N points, accurately with very high probability, is
bounded by C log(N) log(log(N)), where C is related to the desired accuracy and
can be made to grow at a much slower rate than N when the solution is sparse.
The PDE estimates that underlie this result are new to the authors' knowledge
and may be of independent mathematical interest; they include an L1 estimate
for the wave equation, an estimate of extension of eigenfunctions, and a bound
for eigenvalue gaps in Sturm-Liouville problems.
Numerical examples are presented in one spatial dimension and show that as
few as 10 percents of all eigenfunctions can suffice for accurate results.
Finally, we argue that the compressive viewpoint suggests a competitive
parallel algorithm for an adjoint-state inversion method in reflection
seismology.Comment: 45 pages, 4 figure
- …