Learning activation functions from data using cubic spline interpolation

Neural networks require a careful design in order to perform properly on a given task. In particular, selecting a good activation function (possibly in a data-dependent fashion) is a crucial step, which remains an open problem in the research community. Despite a large amount of investigations, most current implementations simply select one fixed function from a small set of candidates, which is not adapted during training, and is shared among all neurons throughout the different layers. However, neither two of these assumptions can be supposed optimal in practice. In this paper, we present a principled way to have data-dependent adaptation of the activation functions, which is performed independently for each neuron. This is achieved by leveraging over past and present advances on cubic spline interpolation, allowing for local adaptation of the functions around their regions of use. The resulting algorithm is relatively cheap to implement, and overfitting is counterbalanced by the inclusion of a novel damping criterion, which penalizes unwanted oscillations from a predefined shape. Experimental results validate the proposal over two well-known benchmarks.Comment: Submitted to the 27th Italian Workshop on Neural Networks (WIRN 2017

The Cascade Orthogonal Neural Network

In the paper new non-conventional growing neural network is proposed. It coincides with the Cascade- Correlation Learning Architecture structurally, but uses ortho-neurons as basic structure units, which can be adjusted using linear tuning procedures. As compared with conventional approximating neural networks proposed approach allows significantly to reduce time required for weight coefficients adjustment and the training dataset size

Suitable MLP Network Activation Functions For Breast Cancer And Thyroid Disease Detection.

This paper presents a comparison study of various MLP activation functions for detection and classification problems

Do ReLU Networks Have An Edge When Approximating Compactly-Supported Functions?

We study the problem of approximating compactly-supported integrable functions while implementing their support set using feedforward neural networks. Our first main result transcribes this "structured" approximation problem into a universality problem. We do this by constructing a refinement of the usual topology on the space $L^1_{\operatorname{loc}}(\mathbb{R}^d,\mathbb{R}^D)$ of locally-integrable functions in which compactly-supported functions can only be approximated in $L^1$-norm by functions with matching discretized support. We establish the universality of ReLU feedforward networks with bilinear pooling layers in this refined topology. Consequentially, we find that ReLU feedforward networks with bilinear pooling can approximate compactly supported functions while implementing their discretized support. We derive a quantitative uniform version of our universal approximation theorem on the dense subclass of compactly-supported Lipschitz functions. This quantitative result expresses the depth, width, and the number of bilinear pooling layers required to construct this ReLU network via the target function's regularity, the metric capacity and diameter of its essential support, and the dimensions of the inputs and output spaces. Conversely, we show that polynomial regressors and analytic feedforward networks are not universal in this space.Comment: 23 Pages: Main Text - 16 pages, Appendix - 7.5 pages, - Bibliography - 5 page

Training Input-Output Recurrent Neural Networks through Spectral Methods

We consider the problem of training input-output recurrent neural networks (RNN) for sequence labeling tasks. We propose a novel spectral approach for learning the network parameters. It is based on decomposition of the cross-moment tensor between the output and a non-linear transformation of the input, based on score functions. We guarantee consistent learning with polynomial sample and computational complexity under transparent conditions such as non-degeneracy of model parameters, polynomial activations for the neurons, and a Markovian evolution of the input sequence. We also extend our results to Bidirectional RNN which uses both previous and future information to output the label at each time point, and is employed in many NLP tasks such as POS tagging

Neuro-Fuzzy Computing System with the Capacity of Implementation on Memristor-Crossbar and Optimization-Free Hardware Training

In this paper, first we present a new explanation for the relation between logical circuits and artificial neural networks, logical circuits and fuzzy logic, and artificial neural networks and fuzzy inference systems. Then, based on these results, we propose a new neuro-fuzzy computing system which can effectively be implemented on the memristor-crossbar structure. One important feature of the proposed system is that its hardware can directly be trained using the Hebbian learning rule and without the need to any optimization. The system also has a very good capability to deal with huge number of input-out training data without facing problems like overtraining.Comment: 16 pages, 11 images, submitted to IEEE Trans. on Fuzzy system

Adaptive PI Hermite neural control for MIMO uncertain nonlinear systems

[[abstract]]This paper presents an adaptive PI Hermite neural control (APIHNC) system for multi-input multi-output (MIMO) uncertain nonlinear systems. The proposed APIHNC system is composed of a neural controller and a robust compensator. The neural controller uses a three-layer Hermite neural network (HNN) to online mimic an ideal controller and the robust compensator is designed to eliminate the effect of the approximation error introduced by the neural controller upon the system stability in the Lyapunov sense. Moreover, a proportional–integral learning algorithm is derived to speed up the convergence of the tracking error. Finally, the proposed APIHNC system is applied to an inverted double pendulums and a two-link robotic manipulator. Simulation results verify that the proposed APIHNC system can achieve high-precision tracking performance. It should be emphasized that the proposed APIHNC system is clearly and easily used for real-time applications.[[notice]]補正完畢[[incitationindex]]SCI[[booktype]]紙本[[booktype]]電子

On Training Efficiency and Computational Costs of a Feed Forward Neural Network: A Review

A comprehensive review on the problem of choosing a suitable activation function for the hidden layer of a feed forward neural network has been widely investigated. Since the nonlinear component of a neural network is the main contributor to the network mapping capabilities, the different choices that may lead to enhanced performances, in terms of training, generalization, or computational costs, are analyzed, both in general-purpose and in embedded computing environments. Finally, a strategy to convert a network configuration between different activation functions without altering the network mapping capabilities will be presented