212 research outputs found
Does Momentum Change the Implicit Regularization on Separable Data?
The momentum acceleration technique is widely adopted in many optimization
algorithms. However, there is no theoretical answer on how the momentum affects
the generalization performance of the optimization algorithms. This paper
studies this problem by analyzing the implicit regularization of momentum-based
optimization. We prove that on the linear classification problem with separable
data and exponential-tailed loss, gradient descent with momentum (GDM)
converges to the L2 max-margin solution, which is the same as vanilla gradient
descent. That means gradient descent with momentum acceleration still converges
to a low-complexity model, which guarantees their generalization. We then
analyze the stochastic and adaptive variants of GDM (i.e., SGDM and
deterministic Adam) and show they also converge to the L2 max-margin solution.
Technically, to overcome the difficulty of the error accumulation in analyzing
the momentum, we construct new potential functions to analyze the gap between
the model parameter and the max-margin solution. Numerical experiments are
conducted and support our theoretical results
Deep Latent Regularity Network for Modeling Stochastic Partial Differential Equations
Stochastic partial differential equations (SPDEs) are crucial
for modelling dynamics with randomness in many areas including economics, physics, and atmospheric sciences. Recently, using deep learning approaches to learn the PDE solution for accelerating PDE simulation becomes increasingly
popular. However, SPDEs have two unique properties that
require new design on the models. First, the model to approximate the solution of SPDE should be generalizable over
both initial conditions and the random sampled forcing term.
Second, the random forcing terms usually have poor regularity whose statistics may diverge (e.g., the space-time white
noise). To deal with the problems, in this work, we design
a deep neural network called Deep Latent Regularity Net
(DLR-Net). DLR-Net includes a regularity feature block as
the main component, which maps the initial condition and the
random forcing term to a set of regularity features. The processing of regularity features is inspired by regularity structure theory and the features provably compose a set of basis to
represent the SPDE solution. The regularity features are then
fed into a small backbone neural operator to get the output.
We conduct experiments on various SPDEs including the dynamic Φ^{4}_{1} model and the stochastic 2D Navier-Stokes equation to predict their solutions, and the results demonstrate that
the proposed DLR-Net can achieve SOTA accuracy compared
with the baselines. Moreover, the inference time is over 20
times faster than the traditional numerical solver and is comparable with the baseline deep learning models
- …