Annealing Optimization for Progressive Learning with Stochastic Approximation

In this work, we introduce a learning model designed to meet the needs of applications in which computational resources are limited, and robustness and interpretability are prioritized. Learning problems can be formulated as constrained stochastic optimization problems, with the constraints originating mainly from model assumptions that define a trade-off between complexity and performance. This trade-off is closely related to over-fitting, generalization capacity, and robustness to noise and adversarial attacks, and depends on both the structure and complexity of the model, as well as the properties of the optimization methods used. We develop an online prototype-based learning algorithm based on annealing optimization that is formulated as an online gradient-free stochastic approximation algorithm. The learning model can be viewed as an interpretable and progressively growing competitive-learning neural network model to be used for supervised, unsupervised, and reinforcement learning. The annealing nature of the algorithm contributes to minimal hyper-parameter tuning requirements, poor local minima prevention, and robustness with respect to the initial conditions. At the same time, it provides online control over the performance-complexity trade-off by progressively increasing the complexity of the learning model as needed, through an intuitive bifurcation phenomenon. Finally, the use of stochastic approximation enables the study of the convergence of the learning algorithm through mathematical tools from dynamical systems and control, and allows for its integration with reinforcement learning algorithms, constructing an adaptive state-action aggregation scheme.Comment: arXiv admin note: text overlap with arXiv:2102.0583

Wavelet-Based Hierarchical Organization of Large Image Databases: ISAR and Face Recognition

We present a method for constructing efficient hierarchical organization of image databases for fast recognition and classification. The method combines a wavelet preprocessor with a Tree-Structured-Vector-Quantization for clustering. We show results of application of the method to ISAR data from ships and to face recognition based on photograph databases. In the ISAR case we show how the method constructs a multi-resolution aspect graph for each target. This paper was presented at the "SPIE's 12th Annual International Symposium on Aerospace/Defense Sensing, Simulation, and Controls (Aerosense'98)", April 13-17, 1998, Orlando, Florida.</center

Ground Vehicle Acoustic Signal Processing Based on Biological Hearing Models

This thesis presents a prototype vehicle acoustic signal classification system with low classification error and short processing delay. To analyze the spectrum of the vehicle acoustic signal, we adopt biologically motivated feature extraction models - cochlear filter and A1-cortical wavelet transform. The multi-resolution representation obtained from these two models is used in the later classification system. Different VQ based clustering algorithms are implemented and tested for real world vehicle acoustic signals. Among them, Learning VQ achieves the optimal Bayes classification performance, but its long search and training time make it not suitable for real time implementation. TSVQ needs a logarithmic search time and its tree structure naturally imitates the aggressive hearing in biological hearing systems, but it has a higher classification error. Finally, a high performance parallel TSVQ (PTSVQ)is introduced, which has classification performance close to the optimal LVQ, while maintains logarithmic search time. Experiments on ACIDS database show that both PTSVQ and LVQ achieve high classification rate. PTSVQ has additional advantages such as easy online training and insensitivity to initial conditions. All these features make PTSVQ the most promising candidate for practical system implementation.Another problem investigated in this thesis is combined DOA and classification, which is motivated by the biological sound localization model developed by Professor S. Shamma: the Stereausis neural network. This model is used to perform DOA estimation for multiple vehicle recordings. The angle estimation is further used to construct a spectral separation template. Experiments with the separated spectrum show significant improvement in classification performance. The biologically inspired separation scheme is quite different from traditional beamforming. However, it integrates all three biological hearing models into a unified framework, and it shows great potential for multiple target DOA and ID systems in the future

Convergence of a Neural Network Classifier

Kohonen's Learning Vector Quantization (LVQ) is a neural network architecture that performs nonparametric classification. It classifies observations by comparing them to k templates called Voronoi vectors. The locations of these vectors are determined from past labeled data through a learning algorithm. When learning is complete, the class of a new observation is the same as the class of the closest Voronoi vector. Hence LVQ is similar to nearest neighbors, except that instead of all of the past obervations being searched only the k Voronoi vectors are searched. In this paper, we show that the LVQ learning algorithm converges to locally asymptotic stable equilibria of an ordinary differential equation. We show that the learning algorithm performance stochastic approximation. Convergence of the Voronoi vectors is guaranteed under the appropriate conditions on the underlying statistics of the classification problem. We also present a modification to the learning algorithm which we argue results in the convergence of the LVQ error to the Bayesian optimal error as the appropriate parameters become large