2,437 research outputs found
BOME! Bilevel Optimization Made Easy: A Simple First-Order Approach
Bilevel optimization (BO) is useful for solving a variety of important
machine learning problems including but not limited to hyperparameter
optimization, meta-learning, continual learning, and reinforcement learning.
Conventional BO methods need to differentiate through the low-level
optimization process with implicit differentiation, which requires expensive
calculations related to the Hessian matrix. There has been a recent quest for
first-order methods for BO, but the methods proposed to date tend to be
complicated and impractical for large-scale deep learning applications. In this
work, we propose a simple first-order BO algorithm that depends only on
first-order gradient information, requires no implicit differentiation, and is
practical and efficient for large-scale non-convex functions in deep learning.
We provide non-asymptotic convergence analysis of the proposed method to
stationary points for non-convex objectives and present empirical results that
show its superior practical performance
Solving ill-posed bilevel programs
This paper deals with ill-posed bilevel programs, i.e., problems admitting multiple lower-level solutions for some upper-level parameters. Many publications have been devoted to the standard optimistic case of this problem, where the difficulty is essentially moved from the objective function to the feasible set. This new problem is simpler but there is no guaranty to obtain local optimal solutions for the original optimistic problem by this process. Considering the intrinsic non-convexity of bilevel programs, computing local optimal solutions is the best one can hope to get in most cases. To achieve this goal, we start by establishing an equivalence between the original optimistic problem an a certain set-valued optimization problem. Next, we develop optimality conditions for the latter problem and show that they generalize all the results currently known in the literature on optimistic bilevel optimization. Our approach is then extended to multiobjective bilevel optimization, and completely new results are derived for problems with vector-valued upper- and lower-level objective functions. Numerical implementations of the results of this paper are provided on some examples, in order to demonstrate how the original optimistic problem can be solved in practice, by means of a special set-valued optimization problem
- …