4,207 research outputs found
Global convergence rate analysis of unconstrained optimization methods based on probabilistic models
We present global convergence rates for a line-search method which is based
on random first-order models and directions whose quality is ensured only with
certain probability. We show that in terms of the order of the accuracy, the
evaluation complexity of such a method is the same as its counterparts that use
deterministic accurate models; the use of probabilistic models only increases
the complexity by a constant, which depends on the probability of the models
being good. We particularize and improve these results in the convex and
strongly convex case.
We also analyze a probabilistic cubic regularization variant that allows
approximate probabilistic second-order models and show improved complexity
bounds compared to probabilistic first-order methods; again, as a function of
the accuracy, the probabilistic cubic regularization bounds are of the same
(optimal) order as for the deterministic case
Successive Convex Approximation Algorithms for Sparse Signal Estimation with Nonconvex Regularizations
In this paper, we propose a successive convex approximation framework for
sparse optimization where the nonsmooth regularization function in the
objective function is nonconvex and it can be written as the difference of two
convex functions. The proposed framework is based on a nontrivial combination
of the majorization-minimization framework and the successive convex
approximation framework proposed in literature for a convex regularization
function. The proposed framework has several attractive features, namely, i)
flexibility, as different choices of the approximate function lead to different
type of algorithms; ii) fast convergence, as the problem structure can be
better exploited by a proper choice of the approximate function and the
stepsize is calculated by the line search; iii) low complexity, as the
approximate function is convex and the line search scheme is carried out over a
differentiable function; iv) guaranteed convergence to a stationary point. We
demonstrate these features by two example applications in subspace learning,
namely, the network anomaly detection problem and the sparse subspace
clustering problem. Customizing the proposed framework by adopting the
best-response type approximation, we obtain soft-thresholding with exact line
search algorithms for which all elements of the unknown parameter are updated
in parallel according to closed-form expressions. The attractive features of
the proposed algorithms are illustrated numerically.Comment: submitted to IEEE Journal of Selected Topics in Signal Processing,
special issue in Robust Subspace Learnin
- …