Second order adjoints for solving PDE-constrained optimization problems

Inverse problems are of utmost importance in many fields of science and engineering. In the variational approach inverse problems are formulated as PDE-constrained optimization problems, where the optimal estimate of the uncertain parameters is the minimizer of a certain cost functional subject to the constraints posed by the model equations. The numerical solution of such optimization problems requires the computation of derivatives of the model output with respect to model parameters. The first order derivatives of a cost functional (defined on the model output) with respect to a large number of model parameters can be calculated efficiently through first order adjoint sensitivity analysis. Second order adjoint models give second derivative information in the form of matrix-vector products between the Hessian of the cost functional and user defined vectors. Traditionally, the construction of second order derivatives for large scale models has been considered too costly. Consequently, data assimilation applications employ optimization algorithms that use only first order derivative information, like nonlinear conjugate gradients and quasi-Newton methods. In this paper we discuss the mathematical foundations of second order adjoint sensitivity analysis and show that it provides an efficient approach to obtain Hessian-vector products. We study the benefits of using of second order information in the numerical optimization process for data assimilation applications. The numerical studies are performed in a twin experiment setting with a two-dimensional shallow water model. Different scenarios are considered with different discretization approaches, observation sets, and noise levels. Optimization algorithms that employ second order derivatives are tested against widely used methods that require only first order derivatives. Conclusions are drawn regarding the potential benefits and the limitations of using high-order information in large scale data assimilation problems

Essays in Problems in Sequential Decisions and Large-Scale Randomized Algorithms

In the first part of this dissertation, we consider two problems in sequential decision making. The first problem we consider is sequential selection of a monotone subsequence from a random permutation. We find a two term asymptotic expansion for the optimal expected value of a sequentially selected monotone subsequence from a random permutation of length $n$. The second problem we consider deals with the multiplicative relaxation or constriction of the classical problem of the number of records in a sequence of $n$ independent and identically distributed observations. In the relaxed case, we find a central limit theorem (CLT) with a different normalization than Renyi\u27s classical CLT, and in the constricted case we find convergence in distribution to an unbounded random variable. In the second part of this dissertation, we put forward two large-scale randomized algorithms. We propose a two-step sensing scheme for the low-rank matrix recovery problem which requires far less storage space and has much lower computational complexity than other state-of-art methods based on nuclear norm minimization. We introduce a fast iterative reweighted least squares algorithm, \textit{Guluru}, based on subsampled randomized Hadamard transform, to solve a wide class of generalized linear models

Design Methods for Reducing Failure Probabilities with Examples from Electrical Engineering

This thesis addresses the quantification of uncertainty and optimization under uncertainty. We focus on uncertainties in the manufacturing process of devices, e.g. caused by manufacturing imperfections, natural material deviations or environmental influences. These uncertainties may lead to deviations in the geometry or the materials, which may cause deviations in the operation of the device. The term yield refers to the fraction of realizations in a manufacturing process under uncertainty, fulfilling all performance requirements. It is the counterpart of the failure probability (yield = 1 - failure probability) and serves as a measure for (un)certainty. The main goal of this work is to efficiently estimate and to maximize the yield. In this way, we increase the reliability of designs which reduces rejects of devices due to malfunction and hence saves resources, money and time. One main challenge in the field of yield estimation is the reduction of computing effort, maintaining high accuracy. In this work we propose two hybrid yield estimation methods. Both are sampling based and evaluate most of the sample points on a surrogate model, while only a small subset of so-called critical sample points is evaluated on the original high fidelity model. The SC-Hybrid approach is based on stochastic collocation and adjoint error indicators. The non-intrusive GPR-Hybrid approach uses Gaussian process regression and allows surrogate model updates on the fly. For efficient yield optimization we propose the adaptive Newton-Monte-Carlo (Newton-MC) method, where the sample size is adaptively increased. Another topic is the optimization of problems with mixed gradient information, i.e., problems, where the derivatives of the objective function are available with respect to some optimization variables, but not for all. The usage of gradient based solvers like the adaptive Newton-MC would require the costly approximation of the derivatives. We propose two methods for this case: the Hermite least squares and the Hermite BOBYQA optimization. Both are modifications of the originally derivative free BOBYQA (Bound constrained Optimization BY Quadratic Approximation) method, but are able to handle derivative information and use least squares regression instead of interpolation. In addition, an advantage of the Hermite-type approaches is their robustness in case of noisy objective functions. The global convergence of these methods is proven. In the context of yield optimization the case of mixed gradient information is particularly relevant, if - besides Gaussian distributed uncertain optimization variables - there are deterministic or non-Gaussian distributed uncertain optimization variables. The proposed methods can be applied to any design process affected by uncertainties. However, in this work we focus on application to the design of electrotechnical devices. We evaluate the approaches on two benchmark problems, a rectangular waveguide and a permanent magnet synchronous machine (PMSM). Significant savings of computing effort can be observed in yield estimation, and single- and multi-objective yield optimization. This allows the application of design optimization under uncertainty in industry

Convergence Analysis and Improvements for Projection Algorithms and Splitting Methods

Non-smooth convex optimization problems occur in all fields of engineering. A common approach to solving this class of problems is proximal algorithms, or splitting methods. These first-order optimization algorithms are often simple, well suited to solve large-scale problems and have a low computational cost per iteration. Essentially, they encode the solution to an optimization problem as a fixed point of some operator, and iterating this operator eventually results in convergence to an optimal point. However, as for other first order methods, the convergence rate is heavily dependent on the conditioning of the problem. Even though the per-iteration cost is usually low, the number of iterations can become prohibitively large for ill-conditioned problems, especially if a high accuracy solution is sought.In this thesis, a few methods for alleviating this slow convergence are studied, which can be divided into two main approaches. The first are heuristic methods that can be applied to a range of fixed-point algorithms. They are based on understanding typical behavior of these algorithms. While these methods are shown to converge, they come with no guarantees on improved convergence rates.The other approach studies the theoretical rates of a class of projection methods that are used to solve convex feasibility problems. These are problems where the goal is to find a point in the intersection of two, or possibly more, convex sets. A study of how the parameters in the algorithm affect the theoretical convergence rate is presented, as well as how they can be chosen to optimize this rate

Algorithmic and Technical Improvements for Next Generation Drug Design Software Tools

[eng] The pharmaceutical industry is actively looking for new ways of boosting the efficiency and effectiveness of their R&D programmes. The extensive use of computational modeling tools in the drug discovery pipeline (DDP) is having a positive impact on research performance, since in silico experiments are usually faster and cheaper that their real counterparts. The lead identification step is a very sensitive point in the DDP. In this context, Virtual high-throughput screening techniques (VHTS) work as a filtering mecha-nism that benefits the following stages by reducing the number of compounds to be tested experimentally. Unfortunately the simplifications applied in the VHTS docking software make them prone generate false positives and negatives. These errors spread across the rest of the DDP stages, and have a negative impact in terms of financial and time costs. In the Electronic and Atomic Protein Modelling group (Barcelona Supercomputing Center, Life Sciences department), we have developed the Protein Energy Landscape Exploration (PELE) software. PELE has demonstrated to be a good alternative to explore the conformational space of proteins and perform ligand-protein docking simulations. In this thesis we discuss how to turn PELE into a faster and more efficient tool by improving its technical and algorithmic features, so that it can be eventually used in VHTS protocols. Besides, we have addressed the difficulties of analyzing extensive data associated with massive simulation production. First, we have rewritten the software using C++ and modern software engineering techniques. As a consequence, our code base is now well organized and tested. PELE has become a piece of software which is easier to modify, understand, and extend. It is also more robust and reliable. The rewriting the code has helped us to overcome some of its previous technical limitations, such as the restrictions on the size of the systems. Also, it has allowed us to extend PELE with new solvent models, force fields, and types of biomolecules. Moreover, the rewriting has make it possible to adapt the code in order to take advantage of new parallel architectures and accelerators obtaining promising speedup results. Second, we have improved the way PELE handles protein flexibility by im-plemented and internal coordinate Normal Mode Analysis (icNMA) method. This method is able to produce more energy favorable perturbations than the current Anisotropic Network Model (ANM) based strategy. This has allowed us to eliminate the unneeded relaxation phase of PELE. As a consequence, the overall computational performance of the sampling is significantly improved (-5-7x). The new internal coordinates-based methodology is able to capture the flexibility of the backbone better than the old method and is in closer agreement to molecular dynamics than the ANM-based method

System- and Data-Driven Methods and Algorithms

An increasing complexity of models used to predict real-world systems leads to the need for algorithms to replace complex models with far simpler ones, while preserving the accuracy of the predictions. This two-volume handbook covers methods as well as applications. This first volume focuses on real-time control theory, data assimilation, real-time visualization, high-dimensional state spaces and interaction of different reduction techniques

Efficient Optimization Algorithms for Nonlinear Data Analysis

Identification of low-dimensional structures and main sources of variation from multivariate data are fundamental tasks in data analysis. Many methods aimed at these tasks involve solution of an optimization problem. Thus, the objective of this thesis is to develop computationally efficient and theoretically justified methods for solving such problems. Most of the thesis is based on a statistical model, where ridges of the density estimated from the data are considered as relevant features. Finding ridges, that are generalized maxima, necessitates development of advanced optimization methods. An efficient and convergent trust region Newton method for projecting a point onto a ridge of the underlying density is developed for this purpose. The method is utilized in a differential equation-based approach for tracing ridges and computing projection coordinates along them. The density estimation is done nonparametrically by using Gaussian kernels. This allows application of ridge-based methods with only mild assumptions on the underlying structure of the data. The statistical model and the ridge finding methods are adapted to two different applications. The first one is extraction of curvilinear structures from noisy data mixed with background clutter. The second one is a novel nonlinear generalization of principal component analysis (PCA) and its extension to time series data. The methods have a wide range of potential applications, where most of the earlier approaches are inadequate. Examples include identification of faults from seismic data and identification of filaments from cosmological data. Applicability of the nonlinear PCA to climate analysis and reconstruction of periodic patterns from noisy time series data are also demonstrated. Other contributions of the thesis include development of an efficient semidefinite optimization method for embedding graphs into the Euclidean space. The method produces structure-preserving embeddings that maximize interpoint distances. It is primarily developed for dimensionality reduction, but has also potential applications in graph theory and various areas of physics, chemistry and engineering. Asymptotic behaviour of ridges and maxima of Gaussian kernel densities is also investigated when the kernel bandwidth approaches infinity. The results are applied to the nonlinear PCA and to finding significant maxima of such densities, which is a typical problem in visual object tracking.Siirretty Doriast