283 research outputs found
Invariance of Weight Distributions in Rectified MLPs
An interesting approach to analyzing neural networks that has received
renewed attention is to examine the equivalent kernel of the neural network.
This is based on the fact that a fully connected feedforward network with one
hidden layer, a certain weight distribution, an activation function, and an
infinite number of neurons can be viewed as a mapping into a Hilbert space. We
derive the equivalent kernels of MLPs with ReLU or Leaky ReLU activations for
all rotationally-invariant weight distributions, generalizing a previous result
that required Gaussian weight distributions. Additionally, the Central Limit
Theorem is used to show that for certain activation functions, kernels
corresponding to layers with weight distributions having mean and finite
absolute third moment are asymptotically universal, and are well approximated
by the kernel corresponding to layers with spherical Gaussian weights. In deep
networks, as depth increases the equivalent kernel approaches a pathological
fixed point, which can be used to argue why training randomly initialized
networks can be difficult. Our results also have implications for weight
initialization.Comment: ICML 201
English character recognition algorithm by improving the weights of MLP neural network with dragonfly algorithm
Character Recognition (CR) is taken into consideration for years. Meanwhile, the neural network plays an important role in recognizing handwritten characters. Many character identification reports have been publishing in English, but still the minimum training timing and high accuracy of handwriting English symbols and characters by utilizing a method of neural networks are represents as open problems. Therefore, creating a character recognition system manually and automatically is very important. In this research, an attempt has been done to incubate an automatic symbols and character system for recognition for English with minimum training and a very high recognition accuracy and classification timing. In the proposed idea for improving the weights of the MLP neural network method in the process of teaching and learning character recognition, the dragonfly optimization algorithm has been used. The innovation of the proposed detection system is that with a combination of dragonfly optimization technique and MLP neural networks, the precisions of the system are recovered, and the computing time is minimized. The approach which was used in this study to identify English characters has high accuracy and minimum training time
Determination of baseflow quantity by using unmanned aerial vehicle (UAV) and Google Earth
Baseflow is most important in low-flow hydrological features [1]. It is a function of a large number of variables that include factors such as topography, geology, soil, vegetation, and climate. In many catchments, base flow is an important component of streamflow and, therefore, base flow separations have been widely studied and have a long history in science. Baseflow separation methods can be divided into two main groups: non-tracer-based and tracer- based separation methods of hydrology. Besides, the base flow is determined by fitting a unit hydrograph model with information from the recession limbs of the hydrograph and extrapolating it backward
The effect of adaptive parameters on the performance of back propagation
The Back Propagation algorithm or its variation on Multilayered Feedforward
Networks is widely used in many applications. However, this algorithm is
well-known to have difficulties with local minima problem particularly caused by
neuron saturation in the hidden layer. Most existing approaches modify the learning
model in order to add a random factor to the model, which overcomes the tendency
to sink into local minima. However, the random perturbations of the search direction
and various kinds of stochastic adjustment to the current set of weights are not
effective in enabling a network to escape from local minima which cause the network
fail to converge to a global minimum within a reasonable number of iterations. Thus,
this research proposed a new method known as Back Propagation Gradient Descent
with Adaptive Gain, Adaptive Momentum and Adaptive Learning Rate
(BPGD-AGAMAL) which modifies the existing Back Propagation Gradient Descent
algorithm by adaptively changing the gain, momentum coefficient and learning rate.
In this method, each training pattern has its own activation functions of neurons in
the hidden layer. The activation functions are adjusted by the adaptation of gain
parameters together with adaptive momentum and learning rate value during the
learning process. The efficiency of the proposed algorithm is compared with
conventional Back Propagation Gradient Descent and Back Propagation Gradient
Descent with Adaptive Gain by means of simulation on six benchmark problems
namely breast cancer, card, glass, iris, soybean, and thyroid. The results show that
the proposed algorithm extensively improves the learning process of conventional
Back Propagation algorithm
Non-local Neural Networks
Both convolutional and recurrent operations are building blocks that process
one local neighborhood at a time. In this paper, we present non-local
operations as a generic family of building blocks for capturing long-range
dependencies. Inspired by the classical non-local means method in computer
vision, our non-local operation computes the response at a position as a
weighted sum of the features at all positions. This building block can be
plugged into many computer vision architectures. On the task of video
classification, even without any bells and whistles, our non-local models can
compete or outperform current competition winners on both Kinetics and Charades
datasets. In static image recognition, our non-local models improve object
detection/segmentation and pose estimation on the COCO suite of tasks. Code is
available at https://github.com/facebookresearch/video-nonlocal-net .Comment: CVPR 2018, code is available at:
https://github.com/facebookresearch/video-nonlocal-ne
- …