Recently, bi-level optimization (BLO) has taken center stage in some very
exciting developments in the area of signal processing (SP) and machine
learning (ML). Roughly speaking, BLO is a classical optimization problem that
involves two levels of hierarchy (i.e., upper and lower levels), wherein
obtaining the solution to the upper-level problem requires solving the
lower-level one. BLO has become popular largely because it is powerful in
modeling problems in SP and ML, among others, that involve optimizing nested
objective functions. Prominent applications of BLO range from resource
allocation for wireless systems to adversarial machine learning. In this work,
we focus on a class of tractable BLO problems that often appear in SP and ML
applications. We provide an overview of some basic concepts of this class of
BLO problems, such as their optimality conditions, standard algorithms
(including their optimization principles and practical implementations), as
well as how they can be leveraged to obtain state-of-the-art results for a
number of key SP and ML applications. Further, we discuss some recent advances
in BLO theory, its implications for applications, and point out some
limitations of the state-of-the-art that require significant future research
efforts. Overall, we hope that this article can serve to accelerate the
adoption of BLO as a generic tool to model, analyze, and innovate on a wide
array of emerging SP and ML applications