11,389 research outputs found
Widely Linear Kernels for Complex-Valued Kernel Activation Functions
Complex-valued neural networks (CVNNs) have been shown to be powerful
nonlinear approximators when the input data can be properly modeled in the
complex domain. One of the major challenges in scaling up CVNNs in practice is
the design of complex activation functions. Recently, we proposed a novel
framework for learning these activation functions neuron-wise in a
data-dependent fashion, based on a cheap one-dimensional kernel expansion and
the idea of kernel activation functions (KAFs). In this paper we argue that,
despite its flexibility, this framework is still limited in the class of
functions that can be modeled in the complex domain. We leverage the idea of
widely linear complex kernels to extend the formulation, allowing for a richer
expressiveness without an increase in the number of adaptable parameters. We
test the resulting model on a set of complex-valued image classification
benchmarks. Experimental results show that the resulting CVNNs can achieve
higher accuracy while at the same time converging faster.Comment: Accepted at ICASSP 201
Functional Multi-Layer Perceptron: a Nonlinear Tool for Functional Data Analysis
In this paper, we study a natural extension of Multi-Layer Perceptrons (MLP)
to functional inputs. We show that fundamental results for classical MLP can be
extended to functional MLP. We obtain universal approximation results that show
the expressive power of functional MLP is comparable to that of numerical MLP.
We obtain consistency results which imply that the estimation of optimal
parameters for functional MLP is statistically well defined. We finally show on
simulated and real world data that the proposed model performs in a very
satisfactory way.Comment: http://www.sciencedirect.com/science/journal/0893608
Metaheuristic design of feedforward neural networks: a review of two decades of research
Over the past two decades, the feedforward neural network (FNN) optimization has been a key interest among the researchers and practitioners of multiple disciplines. The FNN optimization is often viewed from the various perspectives: the optimization of weights, network architecture, activation nodes, learning parameters, learning environment, etc. Researchers adopted such different viewpoints mainly to improve the FNN's generalization ability. The gradient-descent algorithm such as backpropagation has been widely applied to optimize the FNNs. Its success is evident from the FNN's application to numerous real-world problems. However, due to the limitations of the gradient-based optimization methods, the metaheuristic algorithms including the evolutionary algorithms, swarm intelligence, etc., are still being widely explored by the researchers aiming to obtain generalized FNN for a given problem. This article attempts to summarize a broad spectrum of FNN optimization methodologies including conventional and metaheuristic approaches. This article also tries to connect various research directions emerged out of the FNN optimization practices, such as evolving neural network (NN), cooperative coevolution NN, complex-valued NN, deep learning, extreme learning machine, quantum NN, etc. Additionally, it provides interesting research challenges for future research to cope-up with the present information processing era
Ensemble Kalman filter for neural network based one-shot inversion
We study the use of novel techniques arising in machine learning for inverse
problems. Our approach replaces the complex forward model by a neural network,
which is trained simultaneously in a one-shot sense when estimating the unknown
parameters from data, i.e. the neural network is trained only for the unknown
parameter. By establishing a link to the Bayesian approach to inverse problems,
an algorithmic framework is developed which ensures the feasibility of the
parameter estimate w.r. to the forward model. We propose an efficient,
derivative-free optimization method based on variants of the ensemble Kalman
inversion. Numerical experiments show that the ensemble Kalman filter for
neural network based one-shot inversion is a promising direction combining
optimization and machine learning techniques for inverse problems
- …