Deep Multiple Kernel Learning

Deep learning methods have predominantly been applied to large artificial neural networks. Despite their state-of-the-art performance, these large networks typically do not generalize well to datasets with limited sample sizes. In this paper, we take a different approach by learning multiple layers of kernels. We combine kernels at each layer and then optimize over an estimate of the support vector machine leave-one-out error rather than the dual objective function. Our experiments on a variety of datasets show that each layer successively increases performance with only a few base kernels.Comment: 4 pages, 1 figure, 1 table, conference pape

Neural Generalization of Multiple Kernel Learning

Multiple Kernel Learning is a conventional way to learn the kernel function in kernel-based methods. MKL algorithms enhance the performance of kernel methods. However, these methods have a lower complexity compared to deep learning models and are inferior to these models in terms of recognition accuracy. Deep learning models can learn complex functions by applying nonlinear transformations to data through several layers. In this paper, we show that a typical MKL algorithm can be interpreted as a one-layer neural network with linear activation functions. By this interpretation, we propose a Neural Generalization of Multiple Kernel Learning (NGMKL), which extends the conventional multiple kernel learning framework to a multi-layer neural network with nonlinear activation functions. Our experiments on several benchmarks show that the proposed method improves the complexity of MKL algorithms and leads to higher recognition accuracy

Hybrid intelligent deep kernel incremental extreme learning machine based on differential evolution and multiple population grey wolf optimization methods

Focussing on the problem that redundant nodes in the kernel incremental extreme learning machine (KI-ELM) which leads to ineffective iteration increase and reduce the learning efficiency, a novel improved hybrid intelligent deep kernel incremental extreme learning machine (HI-DKIELM) based on a hybrid intelligent algorithms and kernel incremental extreme learning machine is proposed. At first, hybrid intelligent algorithms are proposed based on differential evolution (DE) and multiple population grey wolf optimization (MPGWO) methods which used to optimize the hidden layer neuron parameters and then to determine the effective hidden layer neurons number. The learning efficiency of the algorithm is improved by reducing the network complexity. Then, we bring in the deep network structure to the kernel incremental extreme learning machine to extract the original input data layer by layer gradually. The experiment results show that the HI-DKIELM methods proposed in this paper with more compact network structure have higher prediction accuracy and better ability of generation compared with other ELM methods

FFT-Based Deep Learning Deployment in Embedded Systems

Deep learning has delivered its powerfulness in many application domains, especially in image and speech recognition. As the backbone of deep learning, deep neural networks (DNNs) consist of multiple layers of various types with hundreds to thousands of neurons. Embedded platforms are now becoming essential for deep learning deployment due to their portability, versatility, and energy efficiency. The large model size of DNNs, while providing excellent accuracy, also burdens the embedded platforms with intensive computation and storage. Researchers have investigated on reducing DNN model size with negligible accuracy loss. This work proposes a Fast Fourier Transform (FFT)-based DNN training and inference model suitable for embedded platforms with reduced asymptotic complexity of both computation and storage, making our approach distinguished from existing approaches. We develop the training and inference algorithms based on FFT as the computing kernel and deploy the FFT-based inference model on embedded platforms achieving extraordinary processing speed.Comment: Design, Automation, and Test in Europe (DATE) For source code, please contact Mahdi Nazemi at <[email protected]

The Benefit of Multitask Representation Learning

We discuss a general method to learn data representations from multiple tasks. We provide a justification for this method in both settings of multitask learning and learning-to-learn. The method is illustrated in detail in the special case of linear feature learning. Conditions on the theoretical advantage offered by multitask representation learning over independent task learning are established. In particular, focusing on the important example of half-space learning, we derive the regime in which multitask representation learning is beneficial over independent task learning, as a function of the sample size, the number of tasks and the intrinsic data dimensionality. Other potential applications of our results include multitask feature learning in reproducing kernel Hilbert spaces and multilayer, deep networks.Comment: To appear in Journal of Machine Learning Research (JMLR). 31 page

A theory of representation learning in deep neural networks gives a deep generalisation of kernel methods

The successes of modern deep neural networks (DNNs) are founded on their ability to transform inputs across multiple layers to build good high-level representations. It is therefore critical to understand this process of representation learning. However, we cannot use standard theoretical approaches involving infinite width limits, as they eliminate representation learning. We therefore develop a new infinite width limit, the representation learning limit, that exhibits representation learning mirroring that in finite-width networks, yet at the same time, remains extremely tractable. For instance, the representation learning limit gives exactly multivariate Gaussian posteriors in deep Gaussian processes with a wide range of kernels, including all isotropic (distance-dependent) kernels. We derive an elegant objective that describes how each network layer learns representations that interpolate between input and output. Finally, we use this limit and objective to develop a flexible, deep generalisation of kernel methods, that we call deep kernel machines (DKMs). We show that DKMs can be scaled to large datasets using methods inspired by inducing point methods from the Gaussian process literature, and we show that DKMs exhibit superior performance to other kernel-based approaches