Newton\u27s Method Backpropagation for Complex-Valued Holomorphic Neural Networks: Algebraic and Analytic Properties

The study of Newton\u27s method in complex-valued neural networks (CVNNs) faces many difficulties. In this dissertation, we derive Newton\u27s method backpropagation algorithms for complex-valued holomorphic multilayer perceptrons (MLPs), and we investigate the convergence of the one-step Newton steplength algorithm for the minimization of real-valued complex functions via Newton\u27s method. The problem of singular Hessian matrices provides an obstacle to the use of Newton\u27s method backpropagation to train CVNNs. We approach this problem by developing an adaptive underrelaxation factor algorithm that avoids singularity of the Hessian matrices for the minimization of real-valued complex polynomial functions. To provide experimental support for the use of our algorithms, we perform a comparison of using sigmoidal functions versus their Taylor polynomial approximations as activation functions by using the Newton and pseudo-Newton backpropagation algorithms developed here and the known gradient descent backpropagation algorithm. Our experiments indicate that the Newton\u27s method based algorithms, combined with the use of polynomial activation functions, provide significant improvement in the number of training iterations required over the existing algorithms. We also test our underrelaxation factor algorithm using a small-scale polynomial neuron and a polynomial MLP. Finally, we investigate the application of an algebraic root-finding technique to the case of a polynomial MLP to develop a theoretical framework for the location of initial weight vectors that will guarantee successful training

A multifrequency method for the solution of the acoustic inverse scattering problem

We are interested in solving the time-harmonic inverse acoustic scattering problem for planar sound-soft obstacles. In this work, we introduce four methods for solving inverse scattering problems. The first method is a variation of the method introduced by Johansson and Sleeman. This method solves the inverse problem when we have the far field pattern given for only one incident wave. It is an iterative method based on a pair of integral equations used to obtain the far field pattern of a known single object. The method proposed in this thesis has a better computational performance than the method of Johansson and Sleeman. The second method we present is a multi-frequency method called the recursive linearization algorithm. This method solves the inverse problem when the far field pattern is given for multiple frequencies. The idea of this method is that from an initial guess, we solve the single frequency inverse problem for the lowest frequency. We use the result obtained as the initial guess to solve the problem for the next highest frequency. We repeat this process until we use the data from all frequencies. To solve the problem at each frequency, we use the first method proposed. To improve the quality of the reconstruction of the shadowed part of the object, we solve the inverse scattering problem of reconstructing an unknown sound-soft obstacle in the presence of known scatterers. We show that depending on the position of the scatterers, we may be able to obtain very accurate reconstructions of the entire unknown object. Next, we introduce a method for solving the inverse problem of reconstructing a convex sound-soft obstacle, given measures of the far field pattern at two frequencies that are not in the resonance region of the object. This method is based on the use of an approximation formula for the far field pattern using geometric optics. We are able to prove that for the reconstruction of the circle of radius $R$ and center at the origin, the size of the interval of convergence of this method is proportional to the inverse of the wavenumber. This procedure is effective at reconstructing the illuminated part of the object; however, it requires an initial guess close to the object for frequencies out of the resonance region. Finally, we propose a globalization technique to obtain a better initial guess to solve the inverse problem at frequencies out of the resonance region. In this technique, given the far field pattern of a convex object at two frequencies out of the resonance region, we use our extrapolation operator to generate synthetic data for low frequencies. We apply the recursive linearization algorithm, using as a single frequency solver the method that is based on geometric optics. We obtain an approximation of the object that can be used as the initial guess to apply the recursive linearization algorithm using the first method introduced as the single frequency solver

Expansions and factorizations of matrices and their applications

Abstract. Linear algebra is a foundation to decompositions and algorithms for extracting simple structures from complex data. In this thesis, we investigate and apply modern techniques from linear algebra to solve problems arising in signal processing and computer science. In particular, we focus on data that takes the shape of a matrix and we explore how to represent it as products of circulant and diagonal matrices. To this end, we study matrix decompositions, approximations, and structured matrix expansions whose elements are products of circulant and diagonal matrices. Computationally, we develop a matrix expansion with DCD matrices for approximating a given matrix. Remarkably, DCD matrices, i.e., a product of diagonal matrix, circulant matrix, and another diagonal matrix, give an natural extension to rank-one matrices. Inspired from the singular value decomposition, we introduce a notion of a matrix rank closely related to the expansion and compute the rank of some specific structured matrices. Specifically, Toeplitz matrix is a sum of two DCD matrices. Here, we present a greedy algorithmic framework to devise the expansion numerically. Finally, we show that the practical uses of the DCD expansion can be complemented by the proposed framework and perform two experiments with a view towards applications

Mathematics and Algorithms in Tomography

This is the eighth Oberwolfach conference on the mathematics of tomography. Modalities represented at the workshop included X-ray tomography, sonar, radar, seismic imaging, ultrasound, electron microscopy, impedance imaging, photoacoustic tomography, elastography, vector tomography, and texture analysis

Learned infinite elements for helioseismology

This thesis presents efficient techniques for integrating the information contained in the Dirichlet-to-Neumann (DtN) map of time-harmonic waves propagating in a stratified medium into finite element discretizations. This task arises in the context of domain decomposition methods, e.g. when reducing a problem posed on an unbounded domain to a bounded computational domain on which the problem can then be discretized. Our focus is on stratified media like the Sun, that allow for strong reflection of waves and for which suitable methods are lacking. We present learned infinite elements as a possible approach to deal with such media utilizing the assumption of a separable geometry. In this case, the DtN map is separable, however, it remains a non-local operator with a dense matrix representation, which renders its direct use computationally inefficient. Therefore, we approximate the DtN only indirectly by adding additional degrees of freedom to the linear system in such a way that the Schur complement w.r.t. the latter provides an optimal approximation of DtN and sparsity of the linear system is preserved. This optimality is ensured via the solution of a small minimization problem, which incorporates solutions of one-dimensional time-harmonic wave equations and allows for great flexibility w.r.t. properties of the medium. In the first half of the thesis we provide an error analysis of the proposed method in a generic framework which demonstrates that exponentially fast convergence rates can be expected. Numerical experiments for the Helmholtz equation and an in-depth study on modelling the solar atmosphere with learned infinite elements demonstrate the high accuracy and flexibility of the proposed method in practical applications. In the second half of the thesis, the potential of learned infinite elements in the context of sweeping preconditioners for the efficient iterative solution of large linear systems is investigated. Even though learned infinite elements are very suitable for separable media, they can only be used for tiny perturbations thereof since the corresponding DtN maps turn out to be extremely sensitive to perturbations in the presence of strong reflections.2021-12-2

Data driven regularization models of non-linear ill-posed inverse problems in imaging

Imaging technologies are widely used in application fields such as natural sciences, engineering, medicine, and life sciences. A broad class of imaging problems reduces to solve ill-posed inverse problems (IPs). Traditional strategies to solve these ill-posed IPs rely on variational regularization methods, which are based on minimization of suitable energies, and make use of knowledge about the image formation model (forward operator) and prior knowledge on the solution, but lack in incorporating knowledge directly from data. On the other hand, the more recent learned approaches can easily learn the intricate statistics of images depending on a large set of data, but do not have a systematic method for incorporating prior knowledge about the image formation model. The main purpose of this thesis is to discuss data-driven image reconstruction methods which combine the benefits of these two different reconstruction strategies for the solution of highly nonlinear ill-posed inverse problems. Mathematical formulation and numerical approaches for image IPs, including linear as well as strongly nonlinear problems are described. More specifically we address the Electrical impedance Tomography (EIT) reconstruction problem by unrolling the regularized Gauss-Newton method and integrating the regularization learned by a data-adaptive neural network. Furthermore we investigate the solution of non-linear ill-posed IPs introducing a deep-PnP framework that integrates the graph convolutional denoiser into the proximal Gauss-Newton method with a practical application to the EIT, a recently introduced promising imaging technique. Efficient algorithms are then applied to the solution of the limited electrods problem in EIT, combining compressive sensing techniques and deep learning strategies. Finally, a transformer-based neural network architecture is adapted to restore the noisy solution of the Computed Tomography problem recovered using the filtered back-projection method

Approximate controllability and observability measures in control systems design

The selection of systems of inputs and outputs (input and output structure) forms part of early system design, which is important since it preconditions the potential for control design. Existing methodologies for input, output structure selection rely on criteria expressing distance from uncontrollability (unobservability). The thesis introduces novel measures for evaluating and estimating the distance to uncontrollability and relatively unobservability. At first, the modal measuring approach is studied in detail, providing a framework for the ”best” structure selection. Although controllability (observability) is invariant under state feedback (output injection), the corresponding degrees expressing distance from uncontrollability (unobservability) are not. Hence, the thesis introduces new criteria for the distance problem from uncontrollability (unobservability) which is invariant under feedback transformations. The approach uses the restricted input-state (state-output) matrix pencil and then deploys exterior algebra that reduces the overall problem to the standard problem of distance of a set of polynomials from non-coprimeness. Results on the distance of the Sylvester Resultants from singularity provide the new measures. Since distance to singularity of the corresponding Sylvester matrix is the key in evaluating the distance to uncontrolability it is of the particular interest in the present work. In order to find the solution two novel methods are introduced in the thesis, namely the alternating projection algorithm and a structured singular value approach. A least-squares alternating projection algorithm, motivated by a factorisation result involving the Sylvester resultant matrix, is proposed for calculating the ”best” approximate GCD of a coprime polynomial set. The properties of the proposed algorithm are investigated and the method is compared with alternative optimisation techniques which can be employed to solve the problem. It is also shown that the problem of an approximate GCD calculation is equivalent tothe solution of a structured singular value (µ) problem arising in robust control for which numerous techniques are available. Motivated by the powerful concept of the structured singular values, the proposed method is extended to the special case of an implicit system that has a wide application in the behavioural analysis of complex systems. Moreover, µ-value approach has a potential application for the general distance problem to uncontrollability that is numerically hard to obtain. Overall, the proposed framework significantly simplifies and generalises the input-output structure selection procedure and evaluates alternative solutions for a variety of distance problems that appear in Control Theory

DIFFUSE OPTICAL TOMOGRAPHY: AN ADVANCED MULTISTAGE INVERSE METHOD AND AN OPTIMIZATION METHOD OF SOURCE AND DETECTOR ARRANGEMENTS

Ph.DDOCTOR OF PHILOSOPH