2,756 research outputs found
Fixed-point and coordinate descent algorithms for regularized kernel methods
In this paper, we study two general classes of optimization algorithms for
kernel methods with convex loss function and quadratic norm regularization, and
analyze their convergence. The first approach, based on fixed-point iterations,
is simple to implement and analyze, and can be easily parallelized. The second,
based on coordinate descent, exploits the structure of additively separable
loss functions to compute solutions of line searches in closed form. Instances
of these general classes of algorithms are already incorporated into state of
the art machine learning software for large scale problems. We start from a
solution characterization of the regularized problem, obtained using
sub-differential calculus and resolvents of monotone operators, that holds for
general convex loss functions regardless of differentiability. The two
methodologies described in the paper can be regarded as instances of non-linear
Jacobi and Gauss-Seidel algorithms, and are both well-suited to solve large
scale problems
The representer theorem for Hilbert spaces: a necessary and sufficient condition
A family of regularization functionals is said to admit a linear representer
theorem if every member of the family admits minimizers that lie in a fixed
finite dimensional subspace. A recent characterization states that a general
class of regularization functionals with differentiable regularizer admits a
linear representer theorem if and only if the regularization term is a
non-decreasing function of the norm. In this report, we improve over such
result by replacing the differentiability assumption with lower semi-continuity
and deriving a proof that is independent of the dimensionality of the space
Minimization of multi-penalty functionals by alternating iterative thresholding and optimal parameter choices
Inspired by several recent developments in regularization theory,
optimization, and signal processing, we present and analyze a numerical
approach to multi-penalty regularization in spaces of sparsely represented
functions. The sparsity prior is motivated by the largely expected
geometrical/structured features of high-dimensional data, which may not be
well-represented in the framework of typically more isotropic Hilbert spaces.
In this paper, we are particularly interested in regularizers which are able to
correctly model and separate the multiple components of additively mixed
signals. This situation is rather common as pure signals may be corrupted by
additive noise. To this end, we consider a regularization functional composed
by a data-fidelity term, where signal and noise are additively mixed, a
non-smooth and non-convex sparsity promoting term, and a penalty term to model
the noise. We propose and analyze the convergence of an iterative alternating
algorithm based on simple iterative thresholding steps to perform the
minimization of the functional. By means of this algorithm, we explore the
effect of choosing different regularization parameters and penalization norms
in terms of the quality of recovering the pure signal and separating it from
additive noise. For a given fixed noise level numerical experiments confirm a
significant improvement in performance compared to standard one-parameter
regularization methods. By using high-dimensional data analysis methods such as
Principal Component Analysis, we are able to show the correct geometrical
clustering of regularized solutions around the expected solution. Eventually,
for the compressive sensing problems considered in our experiments we provide a
guideline for a choice of regularization norms and parameters.Comment: 32 page
Reproducing Kernel Banach Spaces with the l1 Norm
Targeting at sparse learning, we construct Banach spaces B of functions on an
input space X with the properties that (1) B possesses an l1 norm in the sense
that it is isometrically isomorphic to the Banach space of integrable functions
on X with respect to the counting measure; (2) point evaluations are continuous
linear functionals on B and are representable through a bilinear form with a
kernel function; (3) regularized learning schemes on B satisfy the linear
representer theorem. Examples of kernel functions admissible for the
construction of such spaces are given.Comment: 28 pages, an extra section was adde
- …