20 research outputs found
Convex optimization over intersection of simple sets: improved convergence rate guarantees via an exact penalty approach
We consider the problem of minimizing a convex function over the intersection
of finitely many simple sets which are easy to project onto. This is an
important problem arising in various domains such as machine learning. The main
difficulty lies in finding the projection of a point in the intersection of
many sets. Existing approaches yield an infeasible point with an
iteration-complexity of for nonsmooth problems with no
guarantees on the in-feasibility. By reformulating the problem through exact
penalty functions, we derive first-order algorithms which not only guarantees
that the distance to the intersection is small but also improve the complexity
to and for smooth functions. For
composite and smooth problems, this is achieved through a saddle-point
reformulation where the proximal operators required by the primal-dual
algorithms can be computed in closed form. We illustrate the benefits of our
approach on a graph transduction problem and on graph matching
Optimizing Nondecomposable Data Dependent Regularizers via Lagrangian Reparameterization offers Significant Performance and Efficiency Gains
Data dependent regularization is known to benefit a wide variety of problems
in machine learning. Often, these regularizers cannot be easily decomposed into
a sum over a finite number of terms, e.g., a sum over individual example-wise
terms. The measure, Area under the ROC curve (AUCROC) and Precision
at a fixed recall (P@R) are some prominent examples that are used in many
applications. We find that for most medium to large sized datasets, scalability
issues severely limit our ability in leveraging the benefits of such
regularizers. Importantly, the key technical impediment despite some recent
progress is that, such objectives remain difficult to optimize via
backpropapagation procedures. While an efficient general-purpose strategy for
this problem still remains elusive, in this paper, we show that for many
data-dependent nondecomposable regularizers that are relevant in applications,
sizable gains in efficiency are possible with minimal code-level changes; in
other words, no specialized tools or numerical schemes are needed. Our
procedure involves a reparameterization followed by a partial dualization --
this leads to a formulation that has provably cheap projection operators. We
present a detailed analysis of runtime and convergence properties of our
algorithm. On the experimental side, we show that a direct use of our scheme
significantly improves the state of the art IOU measures reported for MSCOCO
Stuff segmentation dataset
Proximally Constrained Methods for Weakly Convex Optimization with Weakly Convex Constraints
Optimization models with non-convex constraints arise in many tasks in
machine learning, e.g., learning with fairness constraints or Neyman-Pearson
classification with non-convex loss. Although many efficient methods have been
developed with theoretical convergence guarantees for non-convex unconstrained
problems, it remains a challenge to design provably efficient algorithms for
problems with non-convex functional constraints. This paper proposes a class of
subgradient methods for constrained optimization where the objective function
and the constraint functions are are weakly convex. Our methods solve a
sequence of strongly convex subproblems, where a proximal term is added to both
the objective function and each constraint function. Each subproblem can be
solved by various algorithms for strongly convex optimization. Under a uniform
Slater's condition, we establish the computation complexities of our methods
for finding a nearly stationary point