From neural PCA to deep unsupervised learning

A network supporting deep unsupervised learning is presented. The network is an autoencoder with lateral shortcut connections from the encoder to decoder at each level of the hierarchy. The lateral shortcut connections allow the higher levels of the hierarchy to focus on abstract invariant features. While standard autoencoders are analogous to latent variable models with a single layer of stochastic variables, the proposed network is analogous to hierarchical latent variables models. Learning combines denoising autoencoder and denoising sources separation frameworks. Each layer of the network contributes to the cost function a term which measures the distance of the representations produced by the encoder and the decoder. Since training signals originate from all levels of the network, all layers can learn efficiently even in deep networks. The speedup offered by cost terms from higher levels of the hierarchy and the ability to learn invariant features are demonstrated in experiments.Comment: A revised version of an article that has been accepted for publication in Advances in Independent Component Analysis and Learning Machines (2015), edited by Ella Bingham, Samuel Kaski, Jorma Laaksonen and Jouko Lampine

Artificial Neural Networks, Support Vector Machine And Energy Detection For Spectrum Sensing Based On Real Signals

A Cognitive Radio (CR) is an intelligent wireless communication system, which is able to improve the utilization of the spectral environment. Spectrum sensing (SS) is one of the most important phases in the cognitive radio cycle, this operation consists in detecting signals presence in a particular frequency band. In order to detect primary user (PU) existence, this paper proposes a low cost and low power consumption spectrum sensing implementation. Our proposed platform is tested based on real world signals. Those signals are generated by a Raspberry Pi card and a 433 MHz Wireless transmitter (ASK (Amplitude-Shift Keying) and FSK (Frequency-Shift Keying) modulation type).  RTL-SDR dongle is used as a reception interface. In this work, we compare the performance of three methods for SS operation: The energy detection technique, the Artificial neural network (ANN) and the support vector machine (SVM). So, the received data could be classified as a PU or not (noise) by the ED method, and by training and testing on a proposed ANN and SVM classification model. The proposed algorithms are implemented under MATLAB software. In order to determine the best architecture, in the case of ANN, two different training algorithms are compared. Furthermore, we have investigated the effect of several SVM functions. The main objective is to find out the best method for signal detection between the three methods. The performance evaluation of our proposed system are the probability of detection and the false alarm probability . This Comparative work has shown that the SS operation by SVM can be more accurate than ANN and ED

Practical recommendations for gradient-based training of deep architectures

Learning algorithms related to artificial neural networks and in particular for Deep Learning may seem to involve many bells and whistles, called hyper-parameters. This chapter is meant as a practical guide with recommendations for some of the most commonly used hyper-parameters, in particular in the context of learning algorithms based on back-propagated gradient and gradient-based optimization. It also discusses how to deal with the fact that more interesting results can be obtained when allowing one to adjust many hyper-parameters. Overall, it describes elements of the practice used to successfully and efficiently train and debug large-scale and often deep multi-layer neural networks. It closes with open questions about the training difficulties observed with deeper architectures

A linear approach for sparse coding by a two-layer neural network

Many approaches to transform classification problems from non-linear to linear by feature transformation have been recently presented in the literature. These notably include sparse coding methods and deep neural networks. However, many of these approaches require the repeated application of a learning process upon the presentation of unseen data input vectors, or else involve the use of large numbers of parameters and hyper-parameters, which must be chosen through cross-validation, thus increasing running time dramatically. In this paper, we propose and experimentally investigate a new approach for the purpose of overcoming limitations of both kinds. The proposed approach makes use of a linear auto-associative network (called SCNN) with just one hidden layer. The combination of this architecture with a specific error function to be minimized enables one to learn a linear encoder computing a sparse code which turns out to be as similar as possible to the sparse coding that one obtains by re-training the neural network. Importantly, the linearity of SCNN and the choice of the error function allow one to achieve reduced running time in the learning phase. The proposed architecture is evaluated on the basis of two standard machine learning tasks. Its performances are compared with those of recently proposed non-linear auto-associative neural networks. The overall results suggest that linear encoders can be profitably used to obtain sparse data representations in the context of machine learning problems, provided that an appropriate error function is used during the learning phase