Fast Kernel Sparse Representation

Abstract—Two efficient algorithms are proposed to seek the sparse representation on high-dimensional Hilbert space. By proving that all the calculations in Orthogonal Match Pursuit (OMP) are essentially inner-product combinations, we modify the OMP algorithm to apply the kernel-trick. The proposed Kernel OMP (KOMP) is much faster than the existing methods, and illustrates higher accuracy in some scenarios. Furthermore, inspired by the success of group-sparsity, we enforce a rigid group-sparsity constraint on KOMP which leads to a non-iterative variation. The constrained cousin of KOMP, dubbed as Single-Step KOMP (S-KOMP), merely takes one step to achieve the sparse coefficients. A remarkable improvement (up to 2, 750 times) in efficiency is reported for S-KOMP, with only a negligible loss of accuracy. I

Simplex basis function based sparse least squares support vector regression

In this paper, a novel sparse least squares support vector regression algorithm, referred to as LSSVR-SBF, is introduced which uses a new low rank kernel based on simplex basis function, which has a set of nonlinear parameters. It is shown that the proposed model can be represented as a sparse linear regression model based on simplex basis functions. We propose a fast algorithm for least squares support vector regression solution at the cost of O(N) by avoiding direct kernel matrix inversion. An iterative estimation algorithm has been proposed to optimize the nonlinear parameters associated with the simplex basis functions with the aim of minimizing model mean square errors using the gradient descent algorithm. The proposed fast least square solution and the gradient descent algorithm are alternatively applied. Finally it is shown that the model has a dual representation as a piecewise linear model with respect to the system input. Numerical experiments are carried out to demonstrate the effectiveness of the proposed approaches

Samplets: Construction and scattered data compression

We introduce the concept of samplets by transferring the construction of Tausch-White wavelets to scattered data. This way, we obtain a multiresolution analysis tailored to discrete data which directly enables data compression, feature detection and adaptivity. The cost for constructing the samplet basis and for the fast samplet transform, respectively, is $O(N)$, where $N$ is the number of data points. Samplets with vanishing moments can be used to compress kernel matrices, arising, for instance, kernel based learning and scattered data approximation. The result are sparse matrices with only $O(N \log N )$ remaining entries. We provide estimates for the compression error and present an algorithm that computes the compressed kernel matrix with computational cost $O(N \log N )$. The accuracy of the approximation is controlled by the number of vanishing moments. Besides the cost efficient storage of kernel matrices, the sparse representation enables the application of sparse direct solvers for the numerical solution of corresponding linear systems. In addition to a comprehensive introduction to samplets and their properties, we present numerical studies to benchmark the approach. Our results demonstrate that samplets mark a considerable step in the direction of making large scattered data sets accessible for multiresolution analysis

Kernelized Sparse Self-Representation for Clustering and Recommendation

Sparse models have demonstrated substantial success in applications for data analysis such as clustering, classification and denoising. However, most of the current work is built upon the assumption that data is distributed in a union of subspaces, whereas limited work has been conducted on nonlinear datasets where data reside in a union of manifolds rather than a union of subspaces. To understand data nonlinearity using sparse models, in this paper, we propose to exploit the self-representation property of nonlinear data in an implicit feature space using kernel methods. We propose a kernelized sparse self-representation model, denoted as KSSR, and a novel Kernelized Fast Iterative Soft-Thresholding Algorithm, denoted as K-FISTA, to recover the underlying nonlinear structure among the data. We evaluate our method for clustering problems on both synthetic and real-world datasets, and demonstrate its superior performance compared to the other state-of-the-art methods. We also apply our method for collaborative filtering in recommender systems, and demonstrate its great potential for novel applications beyond clustering

The Sample Complexity of Dictionary Learning

A large set of signals can sometimes be described sparsely using a dictionary, that is, every element can be represented as a linear combination of few elements from the dictionary. Algorithms for various signal processing applications, including classification, denoising and signal separation, learn a dictionary from a set of signals to be represented. Can we expect that the representation found by such a dictionary for a previously unseen example from the same source will have L_2 error of the same magnitude as those for the given examples? We assume signals are generated from a fixed distribution, and study this questions from a statistical learning theory perspective. We develop generalization bounds on the quality of the learned dictionary for two types of constraints on the coefficient selection, as measured by the expected L_2 error in representation when the dictionary is used. For the case of l_1 regularized coefficient selection we provide a generalization bound of the order of O(sqrt(np log(m lambda)/m)), where n is the dimension, p is the number of elements in the dictionary, lambda is a bound on the l_1 norm of the coefficient vector and m is the number of samples, which complements existing results. For the case of representing a new signal as a combination of at most k dictionary elements, we provide a bound of the order O(sqrt(np log(m k)/m)) under an assumption on the level of orthogonality of the dictionary (low Babel function). We further show that this assumption holds for most dictionaries in high dimensions in a strong probabilistic sense. Our results further yield fast rates of order 1/m as opposed to 1/sqrt(m) using localized Rademacher complexity. We provide similar results in a general setting using kernels with weak smoothness requirements

Recent Progress in Image Deblurring

This paper comprehensively reviews the recent development of image deblurring, including non-blind/blind, spatially invariant/variant deblurring techniques. Indeed, these techniques share the same objective of inferring a latent sharp image from one or several corresponding blurry images, while the blind deblurring techniques are also required to derive an accurate blur kernel. Considering the critical role of image restoration in modern imaging systems to provide high-quality images under complex environments such as motion, undesirable lighting conditions, and imperfect system components, image deblurring has attracted growing attention in recent years. From the viewpoint of how to handle the ill-posedness which is a crucial issue in deblurring tasks, existing methods can be grouped into five categories: Bayesian inference framework, variational methods, sparse representation-based methods, homography-based modeling, and region-based methods. In spite of achieving a certain level of development, image deblurring, especially the blind case, is limited in its success by complex application conditions which make the blur kernel hard to obtain and be spatially variant. We provide a holistic understanding and deep insight into image deblurring in this review. An analysis of the empirical evidence for representative methods, practical issues, as well as a discussion of promising future directions are also presented.Comment: 53 pages, 17 figure