1,843 research outputs found
Deep Learning with S-shaped Rectified Linear Activation Units
Rectified linear activation units are important components for
state-of-the-art deep convolutional networks. In this paper, we propose a novel
S-shaped rectified linear activation unit (SReLU) to learn both convex and
non-convex functions, imitating the multiple function forms given by the two
fundamental laws, namely the Webner-Fechner law and the Stevens law, in
psychophysics and neural sciences. Specifically, SReLU consists of three
piecewise linear functions, which are formulated by four learnable parameters.
The SReLU is learned jointly with the training of the whole deep network
through back propagation. During the training phase, to initialize SReLU in
different layers, we propose a "freezing" method to degenerate SReLU into a
predefined leaky rectified linear unit in the initial several training epochs
and then adaptively learn the good initial values. SReLU can be universally
used in the existing deep networks with negligible additional parameters and
computation cost. Experiments with two popular CNN architectures, Network in
Network and GoogLeNet on scale-various benchmarks including CIFAR10, CIFAR100,
MNIST and ImageNet demonstrate that SReLU achieves remarkable improvement
compared to other activation functions.Comment: Accepted by AAAI-1
Stable Electromyographic Sequence Prediction During Movement Transitions using Temporal Convolutional Networks
Transient muscle movements influence the temporal structure of myoelectric
signal patterns, often leading to unstable prediction behavior from
movement-pattern classification methods. We show that temporal convolutional
network sequential models leverage the myoelectric signal's history to discover
contextual temporal features that aid in correctly predicting movement
intentions, especially during interclass transitions. We demonstrate
myoelectric classification using temporal convolutional networks to effect 3
simultaneous hand and wrist degrees-of-freedom in an experiment involving nine
human-subjects. Temporal convolutional networks yield significant
performance improvements over other state-of-the-art methods in terms of both
classification accuracy and stability.Comment: 4 pages, 5 figures, accepted for Neural Engineering (NER) 2019
Conferenc
- …