7 research outputs found
Stochastic Optimization of Areas UnderPrecision-Recall Curves with Provable Convergence
Areas under ROC (AUROC) and precision-recall curves (AUPRC) are common
metrics for evaluating classification performance for imbalanced problems.
Compared with AUROC, AUPRC is a more appropriate metric for highly imbalanced
datasets. While stochastic optimization of AUROC has been studied extensively,
principled stochastic optimization of AUPRC has been rarely explored. In this
work, we propose a principled technical method to optimize AUPRC for deep
learning. Our approach is based on maximizing the averaged precision (AP),
which is an unbiased point estimator of AUPRC. We cast the objective into a sum
of {\it dependent compositional functions} with inner functions dependent on
random variables of the outer level. We propose efficient adaptive and
non-adaptive stochastic algorithms named SOAP with {\it provable convergence
guarantee under mild conditions} by leveraging recent advances in stochastic
compositional optimization. Extensive experimental results on image and graph
datasets demonstrate that our proposed method outperforms prior methods on
imbalanced problems in terms of AUPRC. To the best of our knowledge, our work
represents the first attempt to optimize AUPRC with provable convergence. The
SOAP has been implemented in the libAUC library at~\url{https://libauc.org/}.Comment: 24 pages, 10 figure
Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descent
Recently there are a considerable amount of work devoted to the study of the
algorithmic stability and generalization for stochastic gradient descent (SGD).
However, the existing stability analysis requires to impose restrictive
assumptions on the boundedness of gradients, strong smoothness and convexity of
loss functions. In this paper, we provide a fine-grained analysis of stability
and generalization for SGD by substantially relaxing these assumptions.
Firstly, we establish stability and generalization for SGD by removing the
existing bounded gradient assumptions. The key idea is the introduction of a
new stability measure called on-average model stability, for which we develop
novel bounds controlled by the risks of SGD iterates. This yields
generalization bounds depending on the behavior of the best model, and leads to
the first-ever-known fast bounds in the low-noise setting using stability
approach. Secondly, the smoothness assumption is relaxed by considering loss
functions with Holder continuous (sub)gradients for which we show that optimal
bounds are still achieved by balancing computation and stability. To our best
knowledge, this gives the first-ever-known stability and generalization bounds
for SGD with even non-differentiable loss functions. Finally, we study learning
problems with (strongly) convex objectives but non-convex loss functions.Comment: to appear in ICML 202