128,048 research outputs found
EIGEN: Ecologically-Inspired GENetic Approach for Neural Network Structure Searching from Scratch
Designing the structure of neural networks is considered one of the most
challenging tasks in deep learning, especially when there is few prior
knowledge about the task domain. In this paper, we propose an
Ecologically-Inspired GENetic (EIGEN) approach that uses the concept of
succession, extinction, mimicry, and gene duplication to search neural network
structure from scratch with poorly initialized simple network and few
constraints forced during the evolution, as we assume no prior knowledge about
the task domain. Specifically, we first use primary succession to rapidly
evolve a population of poorly initialized neural network structures into a more
diverse population, followed by a secondary succession stage for fine-grained
searching based on the networks from the primary succession. Extinction is
applied in both stages to reduce computational cost. Mimicry is employed during
the entire evolution process to help the inferior networks imitate the behavior
of a superior network and gene duplication is utilized to duplicate the learned
blocks of novel structures, both of which help to find better network
structures. Experimental results show that our proposed approach can achieve
similar or better performance compared to the existing genetic approaches with
dramatically reduced computation cost. For example, the network discovered by
our approach on CIFAR-100 dataset achieves 78.1% test accuracy under 120 GPU
hours, compared to 77.0% test accuracy in more than 65, 536 GPU hours in [35].Comment: CVPR 201
Easy over Hard: A Case Study on Deep Learning
While deep learning is an exciting new technique, the benefits of this method
need to be assessed with respect to its computational cost. This is
particularly important for deep learning since these learners need hours (to
weeks) to train the model. Such long training time limits the ability of (a)~a
researcher to test the stability of their conclusion via repeated runs with
different random seeds; and (b)~other researchers to repeat, improve, or even
refute that original work.
For example, recently, deep learning was used to find which questions in the
Stack Overflow programmer discussion forum can be linked together. That deep
learning system took 14 hours to execute. We show here that applying a very
simple optimizer called DE to fine tune SVM, it can achieve similar (and
sometimes better) results. The DE approach terminated in 10 minutes; i.e. 84
times faster hours than deep learning method.
We offer these results as a cautionary tale to the software analytics
community and suggest that not every new innovation should be applied without
critical analysis. If researchers deploy some new and expensive process, that
work should be baselined against some simpler and faster alternatives.Comment: 12 pages, 6 figures, accepted at FSE201
- …