2,728 research outputs found
Locally Adaptive Bayesian P-Splines with a Normal-Exponential-Gamma Prior
The necessity to replace smoothing approaches with a global amount of smoothing arises in a variety of situations such as effects with highly varying curvature or effects with discontinuities. We present an implementation of locally adaptive spline smoothing using a class of heavy-tailed shrinkage priors. These priors utilize scale mixtures of normals with locally varying exponential-gamma distributed variances for the differences of the P-spline coefficients. A fully Bayesian hierarchical structure is derived with inference about the posterior being based on Markov Chain Monte Carlo techniques. Three increasingly flexible and automatic approaches are introduced to estimate the spatially varying structure of the variances. In an extensive simulation study, the performance of our approach on a number of benchmark functions is shown to be at least equivalent, but mostly better than previous approaches and fits both functions of smoothly varying complexity and discontinuous functions well. Results from two applications also reflecting these two situations support the simulation results
Differential geometric regularization for supervised learning of classifiers
We study the problem of supervised learning for both binary and multiclass classification from a unified geometric perspective. In particular, we propose a geometric regularization technique to find the submanifold corresponding to an estimator of the class probability P(y|\vec x). The regularization term measures the volume of this submanifold, based on the intuition that overfitting produces rapid local oscillations and hence large volume of the estimator. This technique can be applied to regularize any classification function that satisfies two requirements: firstly, an estimator of the class probability can be obtained; secondly, first and second derivatives of the class probability estimator can be calculated. In experiments, we apply our regularization technique to standard loss functions for classification, our RBF-based implementation compares favorably to widely used regularization methods for both binary and multiclass classification.http://proceedings.mlr.press/v48/baia16.pdfPublished versio
Understanding and Combating Robust Overfitting via Input Loss Landscape Analysis and Regularization
Adversarial training is widely used to improve the robustness of deep neural
networks to adversarial attack. However, adversarial training is prone to
overfitting, and the cause is far from clear. This work sheds light on the
mechanisms underlying overfitting through analyzing the loss landscape w.r.t.
the input. We find that robust overfitting results from standard training,
specifically the minimization of the clean loss, and can be mitigated by
regularization of the loss gradients. Moreover, we find that robust overfitting
turns severer during adversarial training partially because the gradient
regularization effect of adversarial training becomes weaker due to the
increase in the loss landscapes curvature. To improve robust generalization, we
propose a new regularizer to smooth the loss landscape by penalizing the
weighted logits variation along the adversarial direction. Our method
significantly mitigates robust overfitting and achieves the highest robustness
and efficiency compared to similar previous methods. Code is available at
https://github.com/TreeLLi/Combating-RO-AdvLC.Comment: published in journal Pattern Recognition:
https://www.sciencedirect.com/science/article/pii/S0031320322007087?via%3Dihu
- …