105,684 research outputs found
An analytic comparison of regularization methods for Gaussian Processes
Gaussian Processes (GPs) are a popular approach to predict the output of a
parameterized experiment. They have many applications in the field of Computer
Experiments, in particular to perform sensitivity analysis, adaptive design of
experiments and global optimization. Nearly all of the applications of GPs
require the inversion of a covariance matrix that, in practice, is often
ill-conditioned. Regularization methodologies are then employed with
consequences on the GPs that need to be better understood.The two principal
methods to deal with ill-conditioned covariance matrices are i) pseudoinverse
and ii) adding a positive constant to the diagonal (the so-called nugget
regularization).The first part of this paper provides an algebraic comparison
of PI and nugget regularizations. Redundant points, responsible for covariance
matrix singularity, are defined. It is proven that pseudoinverse
regularization, contrarily to nugget regularization, averages the output values
and makes the variance zero at redundant points. However, pseudoinverse and
nugget regularizations become equivalent as the nugget value vanishes. A
measure for data-model discrepancy is proposed which serves for choosing a
regularization technique.In the second part of the paper, a distribution-wise
GP is introduced that interpolates Gaussian distributions instead of data
points. Distribution-wise GP can be seen as an improved regularization method
for GPs
M-Power Regularized Least Squares Regression
Regularization is used to find a solution that both fits the data and is
sufficiently smooth, and thereby is very effective for designing and refining
learning algorithms. But the influence of its exponent remains poorly
understood. In particular, it is unclear how the exponent of the reproducing
kernel Hilbert space~(RKHS) regularization term affects the accuracy and the
efficiency of kernel-based learning algorithms. Here we consider regularized
least squares regression (RLSR) with an RKHS regularization raised to the power
of m, where m is a variable real exponent. We design an efficient algorithm for
solving the associated minimization problem, we provide a theoretical analysis
of its stability, and we compare its advantage with respect to computational
complexity, speed of convergence and prediction accuracy to the classical
kernel ridge regression algorithm where the regularization exponent m is fixed
at 2. Our results show that the m-power RLSR problem can be solved efficiently,
and support the suggestion that one can use a regularization term that grows
significantly slower than the standard quadratic growth in the RKHS norm
A Consistent Regularization Approach for Structured Prediction
We propose and analyze a regularization approach for structured prediction
problems. We characterize a large class of loss functions that allows to
naturally embed structured outputs in a linear space. We exploit this fact to
design learning algorithms using a surrogate loss approach and regularization
techniques. We prove universal consistency and finite sample bounds
characterizing the generalization properties of the proposed methods.
Experimental results are provided to demonstrate the practical usefulness of
the proposed approach.Comment: 39 pages, 2 Tables, 1 Figur
- …