828 research outputs found
A Generalized Representer Theorem for Hilbert Space - Valued Functions
The necessary and sufficient conditions for existence of a generalized representer theorem are presented for learning Hilbert space - valued functions. Representer theorems involving explicit basis functions and Reproducing Kernels are a common occurrence in various machine learning algorithms like generalized least squares, support vector machines, Gaussian process regression, and kernel-based deep neural networks to name a few. Due to the more general structure of the underlying variational problems, the theory is also relevant to other application areas like optimal control, signal processing and decision making. The following presents a generalized representer theorem using the theory of closed, densely defined linear operators and subspace valued maps as a means to address variational optimization problems in learning and control. The implications of the theorem are presented with examples of multi-input - multi-output problems from kernel-based deep neural networks, stochastic regression and sparsity learning problems
The representer theorem for Hilbert spaces: a necessary and sufficient condition
A family of regularization functionals is said to admit a linear representer
theorem if every member of the family admits minimizers that lie in a fixed
finite dimensional subspace. A recent characterization states that a general
class of regularization functionals with differentiable regularizer admits a
linear representer theorem if and only if the regularization term is a
non-decreasing function of the norm. In this report, we improve over such
result by replacing the differentiability assumption with lower semi-continuity
and deriving a proof that is independent of the dimensionality of the space
Periodic Splines and Gaussian Processes for the Resolution of Linear Inverse Problems
This paper deals with the resolution of inverse problems in a periodic
setting or, in other terms, the reconstruction of periodic continuous-domain
signals from their noisy measurements. We focus on two reconstruction
paradigms: variational and statistical. In the variational approach, the
reconstructed signal is solution to an optimization problem that establishes a
tradeoff between fidelity to the data and smoothness conditions via a quadratic
regularization associated to a linear operator. In the statistical approach,
the signal is modeled as a stationary random process defined from a Gaussian
white noise and a whitening operator; one then looks for the optimal estimator
in the mean-square sense. We give a generic form of the reconstructed signals
for both approaches, allowing for a rigorous comparison of the two.We fully
characterize the conditions under which the two formulations yield the same
solution, which is a periodic spline in the case of sampling measurements. We
also show that this equivalence between the two approaches remains valid on
simulations for a broad class of problems. This extends the practical range of
applicability of the variational method
A New Approach to Collaborative Filtering: Operator Estimation with Spectral Regularization
We present a general approach for collaborative filtering (CF) using spectral
regularization to learn linear operators from "users" to the "objects" they
rate. Recent low-rank type matrix completion approaches to CF are shown to be
special cases. However, unlike existing regularization based CF methods, our
approach can be used to also incorporate information such as attributes of the
users or the objects -- a limitation of existing regularization based CF
methods. We then provide novel representer theorems that we use to develop new
estimation methods. We provide learning algorithms based on low-rank
decompositions, and test them on a standard CF dataset. The experiments
indicate the advantages of generalizing the existing regularization based CF
methods to incorporate related information about users and objects. Finally, we
show that certain multi-task learning methods can be also seen as special cases
of our proposed approach
- …