A sparse multinomial probit model for classification

A recent development in penalized probit modelling using a hierarchical Bayesian approach has led to a sparse binomial (two-class) probit classifier that can be trained via an EM algorithm. A key advantage of the formulation is that no tuning of hyperparameters relating to the penalty is needed thus simplifying the model selection process. The resulting model demonstrates excellent classification performance and a high degree of sparsity when used as a kernel machine. It is, however, restricted to the binary classification problem and can only be used in the multinomial situation via a one-against-all or one-against-many strategy. To overcome this, we apply the idea to the multinomial probit model. This leads to a direct multi-classification approach and is shown to give a sparse solution with accuracy and sparsity comparable with the current state-of-the-art. Comparative numerical benchmark examples are used to demonstrate the method

Relational visual cluster validity

The assessment of cluster validity plays a very important role in cluster analysis. Most commonly used cluster validity methods are based on statistical hypothesis testing or finding the best clustering scheme by computing a number of different cluster validity indices. A number of visual methods of cluster validity have been produced to display directly the validity of clusters by mapping data into two- or three-dimensional space. However, these methods may lose too much information to correctly estimate the results of clustering algorithms. Although the visual cluster validity (VCV) method of Hathaway and Bezdek can successfully solve this problem, it can only be applied for object data, i.e. feature measurements. There are very few validity methods that can be used to analyze the validity of data where only a similarity or dissimilarity relation exists – relational data. To tackle this problem, this paper presents a relational visual cluster validity (RVCV) method to assess the validity of clustering relational data. This is done by combining the results of the non-Euclidean relational fuzzy c-means (NERFCM) algorithm with a modification of the VCV method to produce a visual representation of cluster validity. RVCV can cluster complete and incomplete relational data and adds to the visual cluster validity theory. Numeric examples using synthetic and real data are presente

A kernel method for non-linear systems identification – infinite degree volterra series estimation

Volterra series expansions are widely used in analyzing and solving the problems of non-linear dynamical systems. However, the problem that the number of terms to be determined increases exponentially with the order of the expansion restricts its practical application. In practice, Volterra series expansions are truncated severely so that they may not give accurate representations of the original system. To address this problem, kernel methods are shown to be deserving of exploration. In this report, we make use of an existing result from the theory of approximation in reproducing kernel Hilbert space (RKHS) that has not yet been exploited in the systems identification field. An exponential kernel method, based on an RKHS called a generalized Fock space, is introduced, to model non-linear dynamical systems and to specify the corresponding Volterra series expansion. In this way a non-linear dynamical system can be modelled using a finite memory length, infinite degree Volterra series expansion, thus reducing the source of approximation error solely to truncation in time. We can also, in principle, recover any coefficient in the Volterra series

A kernel method for non-linear systems indentification - infinite degree volerra series estimation

Volterra series expansions are widely used in analyzing and solving the problems of non-linear dynamical systems. However, the problem that the number of terms to be determined increases exponentially with the order of the expansion restricts its practical application. In practice, Volterra series expansions are truncated severely so that they may not give accurate representations of the original system. To address this problem, kernel methods are shown to be deserving of exploration. In this report, we make use of an existing result from the theory of approximation in reproducing kernel Hilbert space (RKHS) that has not yet been exploited in the systems identification field. An exponential kernel method, based on an RKHS called a generalized Fock space, is introduced, to model non-linear dynamical systems and to specify the corresponding Volterra series expansion. In this way a non-linear dynamical system can be modelled using a finite memory length, infinite degree Volterra series expansion, thus reducing the source of approximation error solely to truncation in time. We can also, in principle, recover any coefficient in the Volterra series

Iterative sparse interpolation in reproducing kernel Hilbert spaces

The problem of interpolating data in reproducing kernel Hilbert spaces is well known to be ill-conditioned. In the presence of noise, regularisation can be applied to find a good solution. In the noise-free case, regularisation has the effect of over-smoothing the function and few data points are interpolated. In this paper an alternative framework, based on sparsity, is proposed for interpolation of noise-free data. Iterative construction of a sparse sequence of interpolants is shown to be well defined and produces good results

Steepest descent for a linear operator equation of the second kind with application to Tikhonov regularisation

Let H1 H2 be Hilbert spaces, T a bounded linear operator on H1 into H2 such that the range of T, R (T), is closed. Lrt T* denote the adjoint of T. In this paper, we review the generalised solution and method of steepest descent, for the linear operator equation, Tx=b,b E H2. Further, we establish the convergence of the method of steepest descent to the unique solution (T*T=.......

Some lemmas on reproducing kernel Hilbert spaces

Reproducing kernal Hilbert spaces (RKHS) provide a framework for approximation from finite data using the idea of bounded linear functionals. The approximation problem in this case can be viewed as the inverse problem of finding the optimum operator from the Euclidean space of observations to some subspace of the RKHS. In constructing the appropriate invers operator, use is made of both adjoint operators in RKHS and various norms. In this report a number of lemmas are given with respect to such adjoint operators and norms

Reduction of kernel models

Kernel models can be expensive to compute and in a non-stationary environment can become unmanageably large. Here we present several previously reported techniques for reducing the complexity of these models in a common framework. This reformulation leads to the development of further related reduction techniques and clarifies the relationships between these and the existing techniques