6,113 research outputs found
AReS and MaRS - Adversarial and MMD-Minimizing Regression for SDEs
Stochastic differential equations are an important modeling class in many
disciplines. Consequently, there exist many methods relying on various
discretization and numerical integration schemes. In this paper, we propose a
novel, probabilistic model for estimating the drift and diffusion given noisy
observations of the underlying stochastic system. Using state-of-the-art
adversarial and moment matching inference techniques, we avoid the
discretization schemes of classical approaches. This leads to significant
improvements in parameter accuracy and robustness given random initial guesses.
On four established benchmark systems, we compare the performance of our
algorithms to state-of-the-art solutions based on extended Kalman filtering and
Gaussian processes.Comment: Published at the Thirty-sixth International Conference on Machine
Learning (ICML 2019
Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks
Stochastic gradient descent (SGD) is widely believed to perform implicit
regularization when used to train deep neural networks, but the precise manner
in which this occurs has thus far been elusive. We prove that SGD minimizes an
average potential over the posterior distribution of weights along with an
entropic regularization term. This potential is however not the original loss
function in general. So SGD does perform variational inference, but for a
different loss than the one used to compute the gradients. Even more
surprisingly, SGD does not even converge in the classical sense: we show that
the most likely trajectories of SGD for deep networks do not behave like
Brownian motion around critical points. Instead, they resemble closed loops
with deterministic components. We prove that such "out-of-equilibrium" behavior
is a consequence of highly non-isotropic gradient noise in SGD; the covariance
matrix of mini-batch gradients for deep networks has a rank as small as 1% of
its dimension. We provide extensive empirical validation of these claims,
proven in the appendix
- …