1,043 research outputs found
A survey on modern trainable activation functions
In neural networks literature, there is a strong interest in identifying and
defining activation functions which can improve neural network performance. In
recent years there has been a renovated interest of the scientific community in
investigating activation functions which can be trained during the learning
process, usually referred to as "trainable", "learnable" or "adaptable"
activation functions. They appear to lead to better network performance.
Diverse and heterogeneous models of trainable activation function have been
proposed in the literature. In this paper, we present a survey of these models.
Starting from a discussion on the use of the term "activation function" in
literature, we propose a taxonomy of trainable activation functions, highlight
common and distinctive proprieties of recent and past models, and discuss main
advantages and limitations of this type of approach. We show that many of the
proposed approaches are equivalent to adding neuron layers which use fixed
(non-trainable) activation functions and some simple local rule that
constraints the corresponding weight layers.Comment: Published in "Neural Networks" journal (Elsevier
Learning activation functions from data using cubic spline interpolation
Neural networks require a careful design in order to perform properly on a
given task. In particular, selecting a good activation function (possibly in a
data-dependent fashion) is a crucial step, which remains an open problem in the
research community. Despite a large amount of investigations, most current
implementations simply select one fixed function from a small set of
candidates, which is not adapted during training, and is shared among all
neurons throughout the different layers. However, neither two of these
assumptions can be supposed optimal in practice. In this paper, we present a
principled way to have data-dependent adaptation of the activation functions,
which is performed independently for each neuron. This is achieved by
leveraging over past and present advances on cubic spline interpolation,
allowing for local adaptation of the functions around their regions of use. The
resulting algorithm is relatively cheap to implement, and overfitting is
counterbalanced by the inclusion of a novel damping criterion, which penalizes
unwanted oscillations from a predefined shape. Experimental results validate
the proposal over two well-known benchmarks.Comment: Submitted to the 27th Italian Workshop on Neural Networks (WIRN 2017
Learning backward induction: a neural network agent approach
This paper addresses the question of whether neural networks (NNs), a realistic cognitive model of human information processing, can learn to backward induce in a two-stage game with a unique subgame-perfect Nash equilibrium. The NNs were found to predict the Nash equilibrium approximately 70% of the time in new games. Similarly to humans, the neural network agents are also found to suffer from subgame and truncation inconsistency, supporting the contention that they are appropriate models of general learning in humans. The agents were found to behave in a bounded rational manner as a result of the endogenous emergence of decision heuristics. In particular a very simple heuristic socialmax, that chooses the cell with the highest social payoff explains their behavior approximately 60% of the time, whereas the ownmax heuristic that simply chooses the cell with the maximum payoff for that agent fares worse explaining behavior roughly 38%, albeit still significantly better than chance. These two heuristics were found to be ecologically valid for the backward induction problem as they predicted the Nash equilibrium in 67% and 50% of the games respectively. Compared to various standard classification algorithms, the NNs were found to be only slightly more accurate than standard discriminant analyses. However, the latter do not model the dynamic learning process and have an ad hoc postulated functional form. In contrast, a NN agent’s behavior evolves with experience and is capable of taking on any functional form according to the universal approximation theorem.
- …