8 research outputs found
Model selection of polynomial kernel regression
Polynomial kernel regression is one of the standard and state-of-the-art
learning strategies. However, as is well known, the choices of the degree of
polynomial kernel and the regularization parameter are still open in the realm
of model selection. The first aim of this paper is to develop a strategy to
select these parameters. On one hand, based on the worst-case learning rate
analysis, we show that the regularization term in polynomial kernel regression
is not necessary. In other words, the regularization parameter can decrease
arbitrarily fast when the degree of the polynomial kernel is suitable tuned. On
the other hand,taking account of the implementation of the algorithm, the
regularization term is required. Summarily, the effect of the regularization
term in polynomial kernel regression is only to circumvent the " ill-condition"
of the kernel matrix. Based on this, the second purpose of this paper is to
propose a new model selection strategy, and then design an efficient learning
algorithm. Both theoretical and experimental analysis show that the new
strategy outperforms the previous one. Theoretically, we prove that the new
learning strategy is almost optimal if the regression function is smooth.
Experimentally, it is shown that the new strategy can significantly reduce the
computational burden without loss of generalization capability.Comment: 29 pages, 4 figure
Learning with sample dependent hypothesis spaces
AbstractMany learning algorithms use hypothesis spaces which are trained from samples, but little theoretical work has been devoted to the study of these algorithms. In this paper we show that mathematical analysis for these algorithms is essentially different from that for algorithms with hypothesis spaces independent of the sample or depending only on the sample size. The difficulty lies in the lack of a proper characterization of approximation error. To overcome this difficulty, we propose an idea of using a larger function class (not necessarily linear space) containing the union of all possible hypothesis spaces (varying with the sample) to measure the approximation ability of the algorithm. We show how this idea provides error analysis for two particular classes of learning algorithms in kernel methods: learning the kernel via regularization and coefficient based regularization. We demonstrate the power of this approach by its wide applicability
Fast Polynomial Kernel Classification for Massive Data
In the era of big data, it is highly desired to develop efficient machine
learning algorithms to tackle massive data challenges such as storage
bottleneck, algorithmic scalability, and interpretability. In this paper, we
develop a novel efficient classification algorithm, called fast polynomial
kernel classification (FPC), to conquer the scalability and storage challenges.
Our main tools are a suitable selected feature mapping based on polynomial
kernels and an alternating direction method of multipliers (ADMM) algorithm for
a related non-smooth convex optimization problem. Fast learning rates as well
as feasibility verifications including the convergence of ADMM and the
selection of center points are established to justify theoretical behaviors of
FPC. Our theoretical assertions are verified by a series of simulations and
real data applications. The numerical results demonstrate that FPC
significantly reduces the computational burden and storage memory of the
existing learning schemes such as support vector machines and boosting, without
sacrificing their generalization abilities much.Comment: arXiv admin note: text overlap with arXiv:1402.4735 by other author