41 research outputs found
Conic Descent Redux for Memory-Efficient Optimization
Conic programming has well-documented merits in a gamut of signal processing
and machine learning tasks. This contribution revisits a recently developed
first-order conic descent (CD) solver, and advances it in three aspects:
intuition, theory, and algorithmic implementation. It is found that CD can
afford an intuitive geometric derivation that originates from the dual problem.
This opens the door to novel algorithmic designs, with a momentum variant of
CD, momentum conic descent (MOCO) exemplified. Diving deeper into the dual
behavior CD and MOCO reveals: i) an analytically justified stopping criterion;
and, ii) the potential to design preconditioners to speed up dual convergence.
Lastly, to scale semidefinite programming (SDP) especially for low-rank
solutions, a memory efficient MOCO variant is developed and numerically
validated
Scalable Bayesian Meta-Learning through Generalized Implicit Gradients
Meta-learning owns unique effectiveness and swiftness in tackling emerging
tasks with limited data. Its broad applicability is revealed by viewing it as a
bi-level optimization problem. The resultant algorithmic viewpoint however,
faces scalability issues when the inner-level optimization relies on
gradient-based iterations. Implicit differentiation has been considered to
alleviate this challenge, but it is restricted to an isotropic Gaussian prior,
and only favors deterministic meta-learning approaches. This work markedly
mitigates the scalability bottleneck by cross-fertilizing the benefits of
implicit differentiation to probabilistic Bayesian meta-learning. The novel
implicit Bayesian meta-learning (iBaML) method not only broadens the scope of
learnable priors, but also quantifies the associated uncertainty. Furthermore,
the ultimate complexity is well controlled regardless of the inner-level
optimization trajectory. Analytical error bounds are established to demonstrate
the precision and efficiency of the generalized implicit gradient over the
explicit one. Extensive numerical tests are also carried out to empirically
validate the performance of the proposed method.Comment: Accepted as a poster paper in the main track of Proceedings of the
37th AAAI Conference on Artificial Intelligence (AAAI-23