Adaptive weighted least squares algorithm for Volterra signal modeling

Published versio

Adaptive Image Restoration: Perception Based Neural Nework Models and Algorithms.

Abstract This thesis describes research into the field of image restoration. Restoration is a process by which an image suffering some form of distortion or degradation can be recovered to its original form. Two primary concepts within this field have been investigated. The first concept is the use of a Hopfield neural network to implement the constrained least square error method of image restoration. In this thesis, the author reviews previous neural network restoration algorithms in the literature and builds on these algorithms to develop a new faster version of the Hopfield neural network algorithm for image restoration. The versatility of the neural network approach is then extended by the author to deal with the cases of spatially variant distortion and adaptive regularisation. It is found that using the Hopfield-based neural network approach, an image suffering spatially variant degradation can be accurately restored without a substantial penalty in restoration time. In addition, the adaptive regularisation restoration technique presented in this thesis is shown to produce superior results when compared to non-adaptive techniques and is particularly effective when applied to the difficult, yet important, problem of semi-blind deconvolution. The second concept investigated in this thesis, is the difficult problem of incorporating concepts involved in human visual perception into image restoration techniques. In this thesis, the author develops a novel image error measure which compares two images based on the differences between local regional statistics rather than pixel level differences. This measure more closely corresponds to the way humans perceive the differences between two images. Two restoration algorithms are developed by the author based on versions of the novel image error measure. It is shown that the algorithms which utilise this error measure have improved performance and produce visually more pleasing images in the cases of colour and grayscale images under high noise conditions. Most importantly, the perception based algorithms are shown to be extremely tolerant of faults in the restoration algorithm and hence are very robust. A number of experiments have been performed to demonstrate the performance of the various algorithms presented

Analog Photonics Computing for Information Processing, Inference and Optimisation

This review presents an overview of the current state-of-the-art in photonics computing, which leverages photons, photons coupled with matter, and optics-related technologies for effective and efficient computational purposes. It covers the history and development of photonics computing and modern analogue computing platforms and architectures, focusing on optimization tasks and neural network implementations. The authors examine special-purpose optimizers, mathematical descriptions of photonics optimizers, and their various interconnections. Disparate applications are discussed, including direct encoding, logistics, finance, phase retrieval, machine learning, neural networks, probabilistic graphical models, and image processing, among many others. The main directions of technological advancement and associated challenges in photonics computing are explored, along with an assessment of its efficiency. Finally, the paper discusses prospects and the field of optical quantum computing, providing insights into the potential applications of this technology.Comment: Invited submission by Journal of Advanced Quantum Technologies; accepted version 5/06/202

Particle Swarm Optimization: Basic Concepts, Variants and Applications in Power Systems

Many areas in power systems require solving one or more nonlinear optimization problems. While analytical methods might suffer from slow convergence and the curse of dimensionality, heuristics-based swarm intelligence can be an efficient alternative. Particle swarm optimization (PSO), part of the swarm intelligence family, is known to effectively solve large-scale nonlinear optimization problems. This paper presents a detailed overview of the basic concepts of PSO and its variants. Also, it provides a comprehensive survey on the power system applications that have benefited from the powerful nature of PSO as an optimization technique. For each application, technical details that are required for applying PSO, such as its type, particle formulation (solution representation), and the most efficient fitness functions are also discussed

Recurrent neural network for optimization with application to computer vision.

by Cheung Kwok-wai.Thesis (M.Phil.)--Chinese University of Hong Kong, 1993.Includes bibliographical references (leaves [146-154]).Chapter Chapter 1 --- IntroductionChapter 1.1 --- Programmed computing vs. neurocomputing --- p.1-1Chapter 1.2 --- Development of neural networks - feedforward and feedback models --- p.1-2Chapter 1.3 --- State of art of applying recurrent neural network towards computer vision problem --- p.1-3Chapter 1.4 --- Objective of the Research --- p.1-6Chapter 1.5 --- Plan of the thesis --- p.1-7Chapter Chapter 2 --- BackgroundChapter 2.1 --- Short history on development of Hopfield-like neural network --- p.2-1Chapter 2.2 --- Hopfield network model --- p.2-3Chapter 2.2.1 --- Neuron's transfer function --- p.2-3Chapter 2.2.2 --- Updating sequence --- p.2-6Chapter 2.3 --- Hopfield energy function and network convergence properties --- p.2-1Chapter 2.4 --- Generalized Hopfield network --- p.2-13Chapter 2.4.1 --- Network order and generalized Hopfield network --- p.2-13Chapter 2.4.2 --- Associated energy function and network convergence property --- p.2-13Chapter 2.4.3 --- Hardware implementation consideration --- p.2-15Chapter Chapter 3 --- Recurrent neural network for optimizationChapter 3.1 --- Mapping to Neural Network formulation --- p.3-1Chapter 3.2 --- Network stability verse Self-reinforcement --- p.3-5Chapter 3.2.1 --- Quadratic problem and Hopfield network --- p.3-6Chapter 3.2.2 --- Higher-order case and reshaping strategy --- p.3-8Chapter 3.2.3 --- Numerical Example --- p.3-10Chapter 3.3 --- Local minimum limitation and existing solutions in the literature --- p.3-12Chapter 3.3.1 --- Simulated Annealing --- p.3-13Chapter 3.3.2 --- Mean Field Annealing --- p.3-15Chapter 3.3.3 --- Adaptively changing neural network --- p.3-16Chapter 3.3.4 --- Correcting Current Method --- p.3-16Chapter 3.4 --- Conclusions --- p.3-17Chapter Chapter 4 --- A Novel Neural Network for Global Optimization - Tunneling NetworkChapter 4.1 --- Tunneling Algorithm --- p.4-1Chapter 4.1.1 --- Description of Tunneling Algorithm --- p.4-1Chapter 4.1.2 --- Tunneling Phase --- p.4-2Chapter 4.2 --- A Neural Network with tunneling capability Tunneling network --- p.4-8Chapter 4.2.1 --- Network Specifications --- p.4-8Chapter 4.2.2 --- Tunneling function for Hopfield network and the corresponding updating rule --- p.4-9Chapter 4.3 --- Tunneling network stability and global convergence property --- p.4-12Chapter 4.3.1 --- Tunneling network stability --- p.4-12Chapter 4.3.2 --- Global convergence property --- p.4-15Chapter 4.3.2.1 --- Markov chain model for Hopfield network --- p.4-15Chapter 4.3.2.2 --- Classification of the Hopfield markov chain --- p.4-16Chapter 4.3.2.3 --- Markov chain model for tunneling network and its convergence towards global minimum --- p.4-18Chapter 4.3.3 --- Variation of pole strength and its effect --- p.4-20Chapter 4.3.3.1 --- Energy Profile analysis --- p.4-21Chapter 4.3.3.2 --- Size of attractive basin and pole strength required --- p.4-24Chapter 4.3.3.3 --- A new type of pole eases the implementation problem --- p.4-30Chapter 4.4 --- Simulation Results and Performance comparison --- p.4-31Chapter 4.4.1 --- Simulation Experiments --- p.4-32Chapter 4.4.2 --- Simulation Results and Discussions --- p.4-37Chapter 4.4.2.1 --- Comparisons on optimal path obtained and the convergence rate --- p.4-37Chapter 4.4.2.2 --- On decomposition of Tunneling network --- p.4-38Chapter 4.5 --- Suggested hardware implementation of Tunneling network --- p.4-48Chapter 4.5.1 --- Tunneling network hardware implementation --- p.4-48Chapter 4.5.2 --- Alternative implementation theory --- p.4-52Chapter 4.6 --- Conclusions --- p.4-54Chapter Chapter 5 --- Recurrent Neural Network for Gaussian FilteringChapter 5.1 --- Introduction --- p.5-1Chapter 5.1.1 --- Silicon Retina --- p.5-3Chapter 5.1.2 --- An Active Resistor Network for Gaussian Filtering of Image --- p.5-5Chapter 5.1.3 --- Motivations of using recurrent neural network --- p.5-7Chapter 5.1.4 --- Difference between the active resistor network model and recurrent neural network model for gaussian filtering --- p.5-8Chapter 5.2 --- From Problem formulation to Neural Network formulation --- p.5-9Chapter 5.2.1 --- One Dimensional Case --- p.5-9Chapter 5.2.2 --- Two Dimensional Case --- p.5-13Chapter 5.3 --- Simulation Results and Discussions --- p.5-14Chapter 5.3.1 --- Spatial impulse response of the 1-D network --- p.5-14Chapter 5.3.2 --- Filtering property of the 1-D network --- p.5-14Chapter 5.3.3 --- Spatial impulse response of the 2-D network and some filtering results --- p.5-15Chapter 5.4 --- Conclusions --- p.5-16Chapter Chapter 6 --- Recurrent Neural Network for Boundary DetectionChapter 6.1 --- Introduction --- p.6-1Chapter 6.2 --- From Problem formulation to Neural Network formulation --- p.6-3Chapter 6.2.1 --- Problem Formulation --- p.6-3Chapter 6.2.2 --- Recurrent Neural Network Model used --- p.6-4Chapter 6.2.3 --- Neural Network formulation --- p.6-5Chapter 6.3 --- Simulation Results and Discussions --- p.6-7Chapter 6.3.1 --- Feasibility study and Performance comparison --- p.6-7Chapter 6.3.2 --- Smoothing and Boundary Detection --- p.6-9Chapter 6.3.3 --- Convergence improvement by network decomposition --- p.6-10Chapter 6.3.4 --- Hardware implementation consideration --- p.6-10Chapter 6.4 --- Conclusions --- p.6-11Chapter Chapter 7 --- Conclusions and Future ResearchesChapter 7.1 --- Contributions and Conclusions --- p.7-1Chapter 7.2 --- Limitations and Suggested Future Researches --- p.7-3References --- p.R-lAppendix I The assignment of the boundary connection of 2-D recurrent neural network for gaussian filtering --- p.Al-1Appendix II Formula for connection weight assignment of 2-D recurrent neural network for gaussian filtering and the proof on symmetric property --- p.A2-1Appendix III Details on reshaping strategy --- p.A3-

Multiresolution neural networks for image edge detection and restoration

One of the methods for building an automatic visual system is to borrow the properties of the human visual system (HVS). Artificial neural networks are based on this doctrine and they have been applied to image processing and computer vision. This work focused on the plausibility of using a class of Hopfield neural networks for edge detection and image restoration. To this end, a quadratic energy minimization framework is presented. Central to this framework are relaxation operations, which can be implemented using the class of Hopfield neural networks. The role of the uncertainty principle in vision is described, which imposes a limit on the simultaneous localisation in both class and position space. It is shown how a multiresolution approach allows the trade off between position and class resolution and ensures both robustness in noise and efficiency of computation. As edge detection and image restoration are ill-posed, some a priori knowledge is needed to regularize these problems. A multiresolution network is proposed to tackle the uncertainty problem and the regularization of these ill-posed image processing problems. For edge detection, orientation information is used to construct a compatibility function for the strength of the links of the proposed Hopfield neural network. Edge detection 'results are presented for a number of synthetic and natural images which show that the iterative network gives robust results at low signal-to-noise ratios (0 dB) and is at least as good as many previous methods at capturing complex region shapes. For restoration, mean square error is used as the quadratic energy function of the Hopfield neural network. The results of the edge detection are used for adaptive restoration. Also shown are the results of restoration using the proposed iterative network framework

Bayesian plug & play methods for inverse problems in imaging.

Thèse de Doctorat de Mathématiques Appliquées (Université de Paris)Tesis de Doctorado en Ingeniería Eléctrica (Universidad de la República)This thesis deals with Bayesian methods for solving ill-posed inverse problems in imaging with learnt image priors. The first part of this thesis (Chapter 3) concentrates on two particular problems, namely joint denoising and decompression and multi-image super-resolution. After an extensive study of the noise statistics for these problem in the transformed (wavelet or Fourier) domain, we derive two novel algorithms to solve this particular inverse problem. One of them is based on a multi-scale self-similarity prior and can be seen as a transform-domain generalization of the celebrated non-local bayes algorithm to the case of non-Gaussian noise. The second one uses a neural-network denoiser to implicitly encode the image prior, and a splitting scheme to incorporate this prior into an optimization algorithm to find a MAP-like estimator. The second part of this thesis concentrates on the Variational AutoEncoder (VAE) model and some of its variants that show its capabilities to explicitly capture the probability distribution of high-dimensional datasets such as images. Based on these VAE models, we propose two ways to incorporate them as priors for general inverse problems in imaging : • The first one (Chapter 4) computes a joint (space-latent) MAP estimator named Joint Posterior Maximization using an Autoencoding Prior (JPMAP). We show theoretical and experimental evidence that the proposed objective function satisfies a weak bi-convexity property which is sufficient to guarantee that our optimization scheme converges to a stationary point. Experimental results also show the higher quality of the solutions obtained by our JPMAP approach with respect to other non-convex MAP approaches which more often get stuck in spurious local optima. • The second one (Chapter 5) develops a Gibbs-like posterior sampling algorithm for the exploration of posterior distributions of inverse problems using multiple chains and a VAE as image prior. We showhowto use those samples to obtain MMSE estimates and their corresponding uncertainty.Cette thèse traite des méthodes bayésiennes pour résoudre des problèmes inverses mal posés en imagerie avec des distributions a priori d’images apprises. La première partie de cette thèse (Chapitre 3) se concentre sur deux problèmes partic-uliers, à savoir le débruitage et la décompression conjoints et la super-résolutionmulti-images. Après une étude approfondie des statistiques de bruit pour ces problèmes dans le domaine transformé (ondelettes ou Fourier), nous dérivons deuxnouveaux algorithmes pour résoudre ce problème inverse particulie. L’un d’euxest basé sur une distributions a priori d’auto-similarité multi-échelle et peut êtrevu comme une généralisation du célèbre algorithme de Non-Local Bayes au cas dubruit non gaussien. Le second utilise un débruiteur de réseau de neurones pourcoder implicitement la distribution a priori, et un schéma de division pour incor-porer cet distribution dans un algorithme d’optimisation pour trouver un estima-teur de type MAP. La deuxième partie de cette thèse se concentre sur le modèle Variational Auto Encoder (VAE) et certaines de ses variantes qui montrent ses capacités à capturer explicitement la distribution de probabilité d’ensembles de données de grande dimension tels que les images. Sur la base de ces modèles VAE, nous proposons deuxmanières de les incorporer comme distribution a priori pour les problèmes inverses généraux en imagerie: •Le premier (Chapitre 4) calcule un estimateur MAP conjoint (espace-latent) nommé Joint Posterior Maximization using an Autoencoding Prior (JPMAP). Nous montrons des preuves théoriques et expérimentales que la fonction objectif proposée satisfait une propriété de bi-convexité faible qui est suffisante pour garantir que notre schéma d’optimisation converge vers un pointstationnaire. Les résultats expérimentaux montrent également la meilleurequalité des solutions obtenues par notre approche JPMAP par rapport à d’autresapproches MAP non convexes qui restent le plus souvent bloquées dans desminima locaux. •Le second (Chapitre 5) développe un algorithme d’échantillonnage a poste-riori de type Gibbs pour l’exploration des distributions a posteriori de problèmes inverses utilisant des chaînes multiples et un VAE comme distribution a priori. Nous montrons comment utiliser ces échantillons pour obtenir desestimations MMSE et leur incertitude correspondante.En esta tesis se estudian métodos bayesianos para resolver problemas inversos mal condicionados en imágenes usando distribuciones a priori entrenadas. La primera parte de esta tesis (Capítulo 3) se concentra en dos problemas particulares, a saber, el de eliminación de ruido y descompresión conjuntos, y el de superresolución a partir de múltiples imágenes. Después de un extenso estudio de las estadísticas del ruido para estos problemas en el dominio transformado (wavelet o Fourier),derivamos dos algoritmos nuevos para resolver este problema inverso en particular. Uno de ellos se basa en una distribución a priori de autosimilitud multiescala y puede verse como una generalización al dominio wavelet del célebre algoritmo Non-Local Bayes para el caso de ruido no Gaussiano. El segundo utiliza un algoritmo de eliminación de ruido basado en una red neuronal para codificar implícitamente la distribución a priori de las imágenes y un esquema de relajación para incorporar esta distribución en un algoritmo de optimización y así encontrar un estimador similar al MAP. La segunda parte de esta tesis se concentra en el modelo Variational AutoEncoder (VAE) y algunas de sus variantes que han mostrado capacidad para capturar explícitamente la distribución de probabilidad de conjuntos de datos en alta dimensión como las imágenes. Basándonos en estos modelos VAE, proponemos dos formas de incorporarlos como distribución a priori para problemas inversos genéricos en imágenes : •El primero (Capítulo 4) calcula un estimador MAP conjunto (espacio imagen y latente) llamado Joint Posterior Maximization using an Autoencoding Prior (JPMAP). Mostramos evidencia teórica y experimental de que la función objetivo propuesta satisface una propiedad de biconvexidad débil que es suficiente para garantizar que nuestro esquema de optimización converge a un punto estacionario. Los resultados experimentales también muestran la mayor calidad de las soluciones obtenidas por nuestro enfoque JPMAP con respecto a otros enfoques MAP no convexos que a menudo se atascan en mínimos locales espurios. •El segundo (Capítulo 5) desarrolla un algoritmo de muestreo tipo Gibbs parala exploración de la distribución a posteriori de problemas inversos utilizando múltiples cadenas y un VAE como distribución a priori. Mostramos cómo usar esas muestras para obtener estimaciones de MMSE y su correspondiente incertidumbr