Model selection of polynomial kernel regression

Polynomial kernel regression is one of the standard and state-of-the-art learning strategies. However, as is well known, the choices of the degree of polynomial kernel and the regularization parameter are still open in the realm of model selection. The first aim of this paper is to develop a strategy to select these parameters. On one hand, based on the worst-case learning rate analysis, we show that the regularization term in polynomial kernel regression is not necessary. In other words, the regularization parameter can decrease arbitrarily fast when the degree of the polynomial kernel is suitable tuned. On the other hand,taking account of the implementation of the algorithm, the regularization term is required. Summarily, the effect of the regularization term in polynomial kernel regression is only to circumvent the " ill-condition" of the kernel matrix. Based on this, the second purpose of this paper is to propose a new model selection strategy, and then design an efficient learning algorithm. Both theoretical and experimental analysis show that the new strategy outperforms the previous one. Theoretically, we prove that the new learning strategy is almost optimal if the regression function is smooth. Experimentally, it is shown that the new strategy can significantly reduce the computational burden without loss of generalization capability.Comment: 29 pages, 4 figure

Smooth regression quantile estimation

In this thesis, attention will be mainly focused on the local linear kernel regression quantile estimation. Different estimators within this class have been proposed, developed asymptotically and applied to real applications. I include algorithmdesign and selection of smoothing parameters. Chapter 2 studies two estimators, first a single-kernel estimator based on "check function" and a bandwidth selection rule is proposed based on the asymptotic MSE of this estimator. Second a recursive double-kernel estimator which extends Fan et al's (1996) density estimator, and two algorithms are given for bandwidth selection. In Chapter 3, a comparison is carried out of local constant fitting and local linear fitting using MSEs of the estimates as a criterion. Chapter 4 gives a theoretical summary and a simulation study of local linear kernel estimation of conditional distribution function. This has a special interest in itself as well as being related to regression quantiles. In Chapter 5, a kernel-version method of LMS (Cole and Green, 1992) is considered. The method proposed, which is still a semi-parametric one, is based on a general idea of local linear kernel approach of log-likelihood model. Chapter 6 proposes a two-step method of smoothing regression quantiles called BPK. The method considered is based on the idea of combining k- NN method with Healy's et al (1988) partition rule, and correlated regression model are involved. In Chapter 7, methods of regression quantile estimation are compared for different underlying models and design densities in a simulation study. The ISE criterion of interior and boundary points is used as a basis for these comparisons. Three methods are recommended for quantile regression in practice, and they are double kernel method, LMS method and Box partition kernel method (BPK). In Chapter 8, attention is turned to a novel idea of local polynomial roughness penalty regression model, where a purely theoretical framework is considered

Finding kernel function for stock market prediction with support vector regression

Stock market prediction is one of the fascinating issues of stock market research. Accurate stock prediction becomes the biggest challenge in investment industry because the distribution of stock data is changing over the time. Time series forcasting, Neural Network (NN) and Support Vector Machine (SVM) are once commonly used for prediction on stock price. In this study, the data mining operation called time series forecasting is implemented. The large amount of stock data collected from Kuala Lumpur Stock Exchange is used for the experiment to test the validity of SVMs regression. SVM is a new machine learning technique with principle of structural minimization risk, which have greater generalization ability and proved success in time series prediction. Two kernel functions namely Radial Basis Function and polynomial are compared for finding the accurate prediction values. Besides that, backpropagation neural network are also used to compare the predictions performance. Several experiments are conducted and some analyses on the experimental results are done. The results show that SVM with polynomial kernels provide a promising alternative tool in KLSE stock market prediction

Orthogonalized smoothing for rescaled spike and slab models

Rescaled spike and slab models are a new Bayesian variable selection method for linear regression models. In high dimensional orthogonal settings such models have been shown to possess optimal model selection properties. We review background theory and discuss applications of rescaled spike and slab models to prediction problems involving orthogonal polynomials. We first consider global smoothing and discuss potential weaknesses. Some of these deficiencies are remedied by using local regression. The local regression approach relies on an intimate connection between local weighted regression and weighted generalized ridge regression. An important implication is that one can trace the effective degrees of freedom of a curve as a way to visualize and classify curvature. Several motivating examples are presented.Comment: Published in at http://dx.doi.org/10.1214/074921708000000192 the IMS Collections (http://www.imstat.org/publications/imscollections.htm) by the Institute of Mathematical Statistics (http://www.imstat.org

Use Case Point Approach Based Software Effort Estimation using Various Support Vector Regression Kernel Methods

The job of software effort estimation is a critical one in the early stages of the software development life cycle when the details of requirements are usually not clearly identified. Various optimization techniques help in improving the accuracy of effort estimation. The Support Vector Regression (SVR) is one of several different soft-computing techniques that help in getting optimal estimated values. The idea of SVR is based upon the computation of a linear regression function in a high dimensional feature space where the input data are mapped via a nonlinear function. Further, the SVR kernel methods can be applied in transforming the input data and then based on these transformations, an optimal boundary between the possible outputs can be obtained. The main objective of the research work carried out in this paper is to estimate the software effort using use case point approach. The use case point approach relies on the use case diagram to estimate the size and effort of software projects. Then, an attempt has been made to optimize the results obtained from use case point analysis using various SVR kernel methods to achieve better prediction accuracy.Comment: 13 pages, 6 figures, 11 Tables, International Journal of Information Processing (IJIP