30,012 research outputs found
A successive difference-of-convex approximation method for a class of nonconvex nonsmooth optimization problems
We consider a class of nonconvex nonsmooth optimization problems whose
objective is the sum of a smooth function and a finite number of nonnegative
proper closed possibly nonsmooth functions (whose proximal mappings are easy to
compute), some of which are further composed with linear maps. This kind of
problems arises naturally in various applications when different regularizers
are introduced for inducing simultaneous structures in the solutions. Solving
these problems, however, can be challenging because of the coupled nonsmooth
functions: the corresponding proximal mapping can be hard to compute so that
standard first-order methods such as the proximal gradient algorithm cannot be
applied efficiently. In this paper, we propose a successive
difference-of-convex approximation method for solving this kind of problems. In
this algorithm, we approximate the nonsmooth functions by their Moreau
envelopes in each iteration. Making use of the simple observation that Moreau
envelopes of nonnegative proper closed functions are continuous {\em
difference-of-convex} functions, we can then approximately minimize the
approximation function by first-order methods with suitable majorization
techniques. These first-order methods can be implemented efficiently thanks to
the fact that the proximal mapping of {\em each} nonsmooth function is easy to
compute. Under suitable assumptions, we prove that the sequence generated by
our method is bounded and any accumulation point is a stationary point of the
objective. We also discuss how our method can be applied to concrete
applications such as nonconvex fused regularized optimization problems and
simultaneously structured matrix optimization problems, and illustrate the
performance numerically for these two specific applications
Component selection and smoothing in multivariate nonparametric regression
We propose a new method for model selection and model fitting in multivariate
nonparametric regression models, in the framework of smoothing spline ANOVA.
The ``COSSO'' is a method of regularization with the penalty functional being
the sum of component norms, instead of the squared norm employed in the
traditional smoothing spline method. The COSSO provides a unified framework for
several recent proposals for model selection in linear models and smoothing
spline ANOVA models. Theoretical properties, such as the existence and the rate
of convergence of the COSSO estimator, are studied. In the special case of a
tensor product design with periodic functions, a detailed analysis reveals that
the COSSO does model selection by applying a novel soft thresholding type
operation to the function components. We give an equivalent formulation of the
COSSO estimator which leads naturally to an iterative algorithm. We compare the
COSSO with MARS, a popular method that builds functional ANOVA models, in
simulations and real examples. The COSSO method can be extended to
classification problems and we compare its performance with those of a number
of machine learning algorithms on real datasets. The COSSO gives very
competitive performance in these studies.Comment: Published at http://dx.doi.org/10.1214/009053606000000722 in the
Annals of Statistics (http://www.imstat.org/aos/) by the Institute of
Mathematical Statistics (http://www.imstat.org
Optimal Point Placement for Mesh Smoothing
We study the problem of moving a vertex in an unstructured mesh of
triangular, quadrilateral, or tetrahedral elements to optimize the shapes of
adjacent elements. We show that many such problems can be solved in linear time
using generalized linear programming. We also give efficient algorithms for
some mesh smoothing problems that do not fit into the generalized linear
programming paradigm.Comment: 12 pages, 3 figures. A preliminary version of this paper was
presented at the 8th ACM/SIAM Symp. on Discrete Algorithms (SODA '97). This
is the final version, and will appear in a special issue of J. Algorithms for
papers from SODA '9
Well-Centered Triangulation
Meshes composed of well-centered simplices have nice orthogonal dual meshes
(the dual Voronoi diagram). This is useful for certain numerical algorithms
that prefer such primal-dual mesh pairs. We prove that well-centered meshes
also have optimality properties and relationships to Delaunay and minmax angle
triangulations. We present an iterative algorithm that seeks to transform a
given triangulation in two or three dimensions into a well-centered one by
minimizing a cost function and moving the interior vertices while keeping the
mesh connectivity and boundary vertices fixed. The cost function is a direct
result of a new characterization of well-centeredness in arbitrary dimensions
that we present. Ours is the first optimization-based heuristic for
well-centeredness, and the first one that applies in both two and three
dimensions. We show the results of applying our algorithm to small and large
two-dimensional meshes, some with a complex boundary, and obtain a
well-centered tetrahedralization of the cube. We also show numerical evidence
that our algorithm preserves gradation and that it improves the maximum and
minimum angles of acute triangulations created by the best known previous
method.Comment: Content has been added to experimental results section. Significant
edits in introduction and in summary of current and previous results. Minor
edits elsewher
Continuation of Nesterov's Smoothing for Regression with Structured Sparsity in High-Dimensional Neuroimaging
Predictive models can be used on high-dimensional brain images for diagnosis
of a clinical condition. Spatial regularization through structured sparsity
offers new perspectives in this context and reduces the risk of overfitting the
model while providing interpretable neuroimaging signatures by forcing the
solution to adhere to domain-specific constraints. Total Variation (TV)
enforces spatial smoothness of the solution while segmenting predictive regions
from the background. We consider the problem of minimizing the sum of a smooth
convex loss, a non-smooth convex penalty (whose proximal operator is known) and
a wide range of possible complex, non-smooth convex structured penalties such
as TV or overlapping group Lasso. Existing solvers are either limited in the
functions they can minimize or in their practical capacity to scale to
high-dimensional imaging data. Nesterov's smoothing technique can be used to
minimize a large number of non-smooth convex structured penalties but
reasonable precision requires a small smoothing parameter, which slows down the
convergence speed. To benefit from the versatility of Nesterov's smoothing
technique, we propose a first order continuation algorithm, CONESTA, which
automatically generates a sequence of decreasing smoothing parameters. The
generated sequence maintains the optimal convergence speed towards any globally
desired precision. Our main contributions are: To propose an expression of the
duality gap to probe the current distance to the global optimum in order to
adapt the smoothing parameter and the convergence speed. We provide a
convergence rate, which is an improvement over classical proximal gradient
smoothing methods. We demonstrate on both simulated and high-dimensional
structural neuroimaging data that CONESTA significantly outperforms many
state-of-the-art solvers in regard to convergence speed and precision.Comment: 11 pages, 6 figures, accepted in IEEE TMI, IEEE Transactions on
Medical Imaging 201
Fast stable direct fitting and smoothness selection for Generalized Additive Models
Existing computationally efficient methods for penalized likelihood GAM
fitting employ iterative smoothness selection on working linear models (or
working mixed models). Such schemes fail to converge for a non-negligible
proportion of models, with failure being particularly frequent in the presence
of concurvity. If smoothness selection is performed by optimizing `whole model'
criteria these problems disappear, but until now attempts to do this have
employed finite difference based optimization schemes which are computationally
inefficient, and can suffer from false convergence. This paper develops the
first computationally efficient method for direct GAM smoothness selection. It
is highly stable, but by careful structuring achieves a computational
efficiency that leads, in simulations, to lower mean computation times than the
schemes based on working-model smoothness selection. The method also offers a
reliable way of fitting generalized additive mixed models
Robust Forecasting of Non-Stationary Time Series
This paper proposes a robust forecasting method for non-stationary time series. The time series is modelled using non-parametric heteroscedastic regression, and fitted by a localized MM-estimator, combining high robustness and large efficiency. The proposed method is shown to produce reliable forecasts in the presence of outliers, non-linearity, and heteroscedasticity. In the absence of outliers, the forecasts are only slightly less precise than those based on a localized Least Squares estimator. An additional advantage of the MM-estimator is that it provides a robust estimate of the local variability of the time series.Heteroscedasticity;Non-parametric regression;Prediction;Outliers;Robustness
- …