36,648 research outputs found
Exploration and Implementation of Neural Ordinary Differential Equations
Neural ordinary differential equations (ODEs) have recently emerged as a novel ap- proach to deep learning, leveraging the knowledge of two previously separate domains, neural networks and differential equations. In this paper, we first examine the back- ground and lay the foundation for traditional artificial neural networks. We then present neural ODEs from a rigorous mathematical perspective, and explore their advantages and trade-offs compared to traditional neural nets
Bottom-Up and Top-Down Reasoning with Hierarchical Rectified Gaussians
Convolutional neural nets (CNNs) have demonstrated remarkable performance in
recent history. Such approaches tend to work in a unidirectional bottom-up
feed-forward fashion. However, practical experience and biological evidence
tells us that feedback plays a crucial role, particularly for detailed spatial
understanding tasks. This work explores bidirectional architectures that also
reason with top-down feedback: neural units are influenced by both lower and
higher-level units.
We do so by treating units as rectified latent variables in a quadratic
energy function, which can be seen as a hierarchical Rectified Gaussian model
(RGs). We show that RGs can be optimized with a quadratic program (QP), that
can in turn be optimized with a recurrent neural network (with rectified linear
units). This allows RGs to be trained with GPU-optimized gradient descent. From
a theoretical perspective, RGs help establish a connection between CNNs and
hierarchical probabilistic models. From a practical perspective, RGs are well
suited for detailed spatial tasks that can benefit from top-down reasoning. We
illustrate them on the challenging task of keypoint localization under
occlusions, where local bottom-up evidence may be misleading. We demonstrate
state-of-the-art results on challenging benchmarks.Comment: To appear in CVPR 201
Neural nets - their use and abuse for small data sets
Neural nets can be used for non-linear classification and regression models. They have a big advantage
over conventional statistical tools in that it is not necessary to assume any mathematical form for the
functional relationship between the variables. However, they also have a few associated problems chief of
which are probably the risk of over-parametrization in the absence of P-values, the lack of appropriate
diagnostic tools and the difficulties associated with model interpretation. The first of these problems is
particularly important in the case of small data sets. These problems are investigated in the context of real
market research data involving non-linear regression and discriminant analysis. In all cases we compare
the results of the non-linear neural net models with those of conventional linear statistical methods. Our
conclusion is that the theory and software for neural networks has some way to go before the above
problems will be solved
- …