54,426 research outputs found
Shooting Neural Networks Algorithm for Solving Boundary Value Problems in ODEs
The objective of this paper is to use Neural Networks for solving boundary value problems (BVPs) in Ordinary Differential Equations (ODEs). The Neural networks use the principle of Back propagation. Five examples are considered to show effectiveness of using the shooting techniques and neural network for solving the BVPs in ODEs. The convergence properties of the technique, which depend on the convergence of the integration technique and accuracy of the interpolation technique are considered
Interpolation Consistency Training for Semi-Supervised Learning
We introduce Interpolation Consistency Training (ICT), a simple and
computation efficient algorithm for training Deep Neural Networks in the
semi-supervised learning paradigm. ICT encourages the prediction at an
interpolation of unlabeled points to be consistent with the interpolation of
the predictions at those points. In classification problems, ICT moves the
decision boundary to low-density regions of the data distribution. Our
experiments show that ICT achieves state-of-the-art performance when applied to
standard neural network architectures on the CIFAR-10 and SVHN benchmark
datasets. Our theoretical analysis shows that ICT corresponds to a certain type
of data-adaptive regularization with unlabeled points which reduces overfitting
to labeled points under high confidence values.Comment: Extended version of IJCAI 2019 paper. Semi-supervised Learning, Deep
Learning, Neural Networks. All the previous results are unchanged; we added
new theoretical and empirical result
Kernel interpolation generalizes poorly
One of the most interesting problems in the recent renaissance of the studies
in kernel regression might be whether the kernel interpolation can generalize
well, since it may help us understand the `benign overfitting henomenon'
reported in the literature on deep networks. In this paper, under mild
conditions, we show that for any , the generalization error of
kernel interpolation is lower bounded by . In other
words, the kernel interpolation generalizes poorly for a large class of
kernels. As a direct corollary, we can show that overfitted wide neural
networks defined on sphere generalize poorly
Can neural networks extrapolate? Discussion of a theorem by Pedro Domingos
Neural networks trained on large datasets by minimizing a loss have become
the state-of-the-art approach for resolving data science problems, particularly
in computer vision, image processing and natural language processing. In spite
of their striking results, our theoretical understanding about how neural
networks operate is limited. In particular, what are the interpolation
capabilities of trained neural networks? In this paper we discuss a theorem of
Domingos stating that "every machine learned by continuous gradient descent is
approximately a kernel machine". According to Domingos, this fact leads to
conclude that all machines trained on data are mere kernel machines. We first
extend Domingo's result in the discrete case and to networks with vector-valued
output. We then study its relevance and significance on simple examples. We
find that in simple cases, the "neural tangent kernel" arising in Domingos'
theorem does provide understanding of the networks' predictions. Furthermore,
when the task given to the network grows in complexity, the interpolation
capability of the network can be effectively explained by Domingos' theorem,
and therefore is limited. We illustrate this fact on a classic perception
theory problem: recovering a shape from its boundary
Plateau in Monotonic Linear Interpolation -- A "Biased" View of Loss Landscape for Deep Networks
Monotonic linear interpolation (MLI) - on the line connecting a random
initialization with the minimizer it converges to, the loss and accuracy are
monotonic - is a phenomenon that is commonly observed in the training of neural
networks. Such a phenomenon may seem to suggest that optimization of neural
networks is easy. In this paper, we show that the MLI property is not
necessarily related to the hardness of optimization problems, and empirical
observations on MLI for deep neural networks depend heavily on biases. In
particular, we show that interpolating both weights and biases linearly leads
to very different influences on the final output, and when different classes
have different last-layer biases on a deep network, there will be a long
plateau in both the loss and accuracy interpolation (which existing theory of
MLI cannot explain). We also show how the last-layer biases for different
classes can be different even on a perfectly balanced dataset using a simple
model. Empirically we demonstrate that similar intuitions hold on practical
networks and realistic datasets.Comment: ICLR 202
Radial basis functions versus geostatistics in spatial interpolations
A key problem in environmental monitoring is the spatial interpolation. The main current approach in spatial interpolation is geostatistical. Geostatistics is neither the only nor the best spatial interpolation method. Actually there is no “best” method, universally valid. Choosing a particular method implies to make assumptions. The understanding of initial assumption, of the methods used, and the correct interpretation of the interpolation results are key elements of the spatial interpolation process. A powerful alternative to geostatistics in spatial interpolation is the use of the soft computing methods. They offer the potential for a more flexible, less assumption dependent approach. Artificial Neural Networks are well suited for this kind of problems, due to their ability to handle non-linear, noisy, and inconsistent data. The present paper intends to prove the advantage of using Radial Basis Functions (RBF) instead of geostatistics in spatial interpolations, based on a detailed analyze and modeling of the SIC2004 (Spatial Interpolation Comparison) dataset.IFIP International Conference on Artificial Intelligence in Theory and Practice - Neural NetsRed de Universidades con Carreras en Informática (RedUNCI
Radial basis functions versus geostatistics in spatial interpolations
A key problem in environmental monitoring is the spatial interpolation. The main current approach in spatial interpolation is geostatistical. Geostatistics is neither the only nor the best spatial interpolation method. Actually there is no “best” method, universally valid. Choosing a particular method implies to make assumptions. The understanding of initial assumption, of the methods used, and the correct interpretation of the interpolation results are key elements of the spatial interpolation process. A powerful alternative to geostatistics in spatial interpolation is the use of the soft computing methods. They offer the potential for a more flexible, less assumption dependent approach. Artificial Neural Networks are well suited for this kind of problems, due to their ability to handle non-linear, noisy, and inconsistent data. The present paper intends to prove the advantage of using Radial Basis Functions (RBF) instead of geostatistics in spatial interpolations, based on a detailed analyze and modeling of the SIC2004 (Spatial Interpolation Comparison) dataset.IFIP International Conference on Artificial Intelligence in Theory and Practice - Neural NetsRed de Universidades con Carreras en Informática (RedUNCI
New acceleration technique for the backpropagation algorithm
Artificial neural networks have been studied for many years in the hope of achieving human like performance in the area of pattern recognition, speech synthesis and higher level of cognitive process. In the connectionist model there are several interconnected processing elements called the neurons that have limited processing capability. Even though the rate of information transmitted between these elements is limited, the complex interconnection and the cooperative interaction between these elements results in a vastly increased computing power; The neural network models are specified by an organized network topology of interconnected neurons. These networks have to be trained in order them to be used for a specific purpose. Backpropagation is one of the popular methods of training the neural networks. There has been a lot of improvement over the speed of convergence of standard backpropagation algorithm in the recent past. Herein we have presented a new technique for accelerating the existing backpropagation without modifying it. We have used the fourth order interpolation method for the dominant eigen values, by using these we change the slope of the activation function. And by doing so we increase the speed of convergence of the backpropagation algorithm; Our experiments have shown significant improvement in the convergence time for problems widely used in benchmarKing Three to ten fold decrease in convergence time is achieved. Convergence time decreases as the complexity of the problem increases. The technique adjusts the energy state of the system so as to escape from local minima
- …