1 research outputs found
Parametrized Accelerated Methods Free of Condition Number
Analyses of accelerated (momentum-based) gradient descent usually assume
bounded condition number to obtain exponential convergence rates. However, in
many real problems, e.g., kernel methods or deep neural networks, the condition
number, even locally, can be unbounded, unknown or mis-estimated. This poses
problems in both implementing and analyzing accelerated algorithms. In this
paper, we address this issue by proposing parametrized accelerated methods by
considering the condition number as a free parameter. We provide spectral-level
analysis for several important accelerated algorithms, obtain explicit
expressions and improve worst case convergence rates. Moreover, we show that
those algorithm converge exponentially even when the condition number is
unknown or mis-estimated.Comment: 23 pages, 3 figure