3,093 research outputs found
Neural networks in geophysical applications
Neural networks are increasingly popular in geophysics.
Because they are universal approximators, these
tools can approximate any continuous function with an
arbitrary precision. Hence, they may yield important
contributions to finding solutions to a variety of geophysical applications.
However, knowledge of many methods and techniques
recently developed to increase the performance
and to facilitate the use of neural networks does not seem
to be widespread in the geophysical community. Therefore,
the power of these tools has not yet been explored to
their full extent. In this paper, techniques are described
for faster training, better overall performance, i.e., generalization,and the automatic estimation of network size
and architecture
An Incremental Construction of Deep Neuro Fuzzy System for Continual Learning of Non-stationary Data Streams
Existing FNNs are mostly developed under a shallow network configuration
having lower generalization power than those of deep structures. This paper
proposes a novel self-organizing deep FNN, namely DEVFNN. Fuzzy rules can be
automatically extracted from data streams or removed if they play limited role
during their lifespan. The structure of the network can be deepened on demand
by stacking additional layers using a drift detection method which not only
detects the covariate drift, variations of input space, but also accurately
identifies the real drift, dynamic changes of both feature space and target
space. DEVFNN is developed under the stacked generalization principle via the
feature augmentation concept where a recently developed algorithm, namely
gClass, drives the hidden layer. It is equipped by an automatic feature
selection method which controls activation and deactivation of input attributes
to induce varying subsets of input features. A deep network simplification
procedure is put forward using the concept of hidden layer merging to prevent
uncontrollable growth of dimensionality of input space due to the nature of
feature augmentation approach in building a deep network structure. DEVFNN
works in the sample-wise fashion and is compatible for data stream
applications. The efficacy of DEVFNN has been thoroughly evaluated using seven
datasets with non-stationary properties under the prequential test-then-train
protocol. It has been compared with four popular continual learning algorithms
and its shallow counterpart where DEVFNN demonstrates improvement of
classification accuracy. Moreover, it is also shown that the concept drift
detection method is an effective tool to control the depth of network structure
while the hidden layer merging scenario is capable of simplifying the network
complexity of a deep network with negligible compromise of generalization
performance.Comment: This paper has been published in IEEE Transactions on Fuzzy System
Importance Estimation with Random Gradient for Neural Network Pruning
Global Neuron Importance Estimation is used to prune neural networks for
efficiency reasons. To determine the global importance of each neuron or
convolutional kernel, most of the existing methods either use activation or
gradient information or both, which demands abundant labelled examples. In this
work, we use heuristics to derive importance estimation similar to Taylor First
Order (TaylorFO) approximation based methods. We name our methods TaylorFO-abs
and TaylorFO-sq. We propose two additional methods to improve these importance
estimation methods. Firstly, we propagate random gradients from the last layer
of a network, thus avoiding the need for labelled examples. Secondly, we
normalize the gradient magnitude of the last layer output before propagating,
which allows all examples to contribute similarly to the importance score. Our
methods with additional techniques perform better than previous methods when
tested on ResNet and VGG architectures on CIFAR-100 and STL-10 datasets.
Furthermore, our method also complements the existing methods and improves
their performances when combined with them.Comment: 7 pages, 2 figures, ICLR 2023 Workshop on Sparsity in Neural
Networks. arXiv admin note: text overlap with arXiv:2306.1320
Photometric redshifts with Quasi Newton Algorithm (MLPQNA). Results in the PHAT1 contest
Context. Since the advent of modern multiband digital sky surveys,
photometric redshifts (photo-z's) have become relevant if not crucial to many
fields of observational cosmology, from the characterization of cosmic
structures, to weak and strong lensing. Aims. We describe an application to an
astrophysical context, namely the evaluation of photometric redshifts, of
MLPQNA, a machine learning method based on Quasi Newton Algorithm. Methods.
Theoretical methods for photo-z's evaluation are based on the interpolation of
a priori knowledge (spectroscopic redshifts or SED templates) and represent an
ideal comparison ground for neural networks based methods. The MultiLayer
Perceptron with Quasi Newton learning rule (MLPQNA) described here is a
computing effective implementation of Neural Networks for the first time
exploited to solve regression problems in the astrophysical context and is
offered to the community through the DAMEWARE (DAta Mining & ExplorationWeb
Application REsource) infrastructure. Results. The PHAT contest (Hildebrandt et
al. 2010) provides a standard dataset to test old and new methods for
photometric redshift evaluation and with a set of statistical indicators which
allow a straightforward comparison among different methods. The MLPQNA model
has been applied on the whole PHAT1 dataset of 1984 objects after an
optimization of the model performed by using as training set the 515 available
spectroscopic redshifts. When applied to the PHAT1 dataset, MLPQNA obtains the
best bias accuracy (0.0006) and very competitive accuracies in terms of scatter
(0.056) and outlier percentage (16.3%), scoring as the second most effective
empirical method among those which have so far participated to the contest.
MLPQNA shows better generalization capabilities than most other empirical
methods especially in presence of underpopulated regions of the Knowledge Base.Comment: Accepted for publication in Astronomy & Astrophysics; 9 pages, 2
figure
- …