Evolutionary algorithms for hyperparameter optimization in machine learning for application in high energy physics

The analysis of vast amounts of data constitutes a major challenge in modern high energy physics experiments. Machine learning (ML) methods, typically trained on simulated data, are often employed to facilitate this task. Several choices need to be made by the user when training the ML algorithm. In addition to deciding which ML algorithm to use and choosing suitable observables as inputs, users typically need to choose among a plethora of algorithm-specific parameters. We refer to parameters that need to be chosen by the user as hyperparameters. These are to be distinguished from parameters that the ML algorithm learns autonomously during the training, without intervention by the user. The choice of hyperparameters is conventionally done manually by the user and often has a significant impact on the performance of the ML algorithm. In this paper, we explore two evolutionary algorithms: particle swarm optimization (PSO) and genetic algorithm (GA), for the purposes of performing the choice of optimal hyperparameter values in an autonomous manner. Both of these algorithms will be tested on different datasets and compared to alternative methods.Comment: Corrected typos. Removed a remark on page 2 regarding the similarity of minimization and maximization problem. Removed a remark on page 9 (Summary) regarding thee ANN, since this was not studied in the pape

Design Space Covering for Uncertainty: Exploration of a New Methodology for Decision Making in Early Stage Design

Decisions made in early-stage design are of vital importance as they significantly impact the quality of the final design. Despite recent developments in design theory for early-stage design, designers of large complex systems still lack sufficient tools to make robust and reliable preliminary design decisions that do not have a lasting negative impact on the final design. Much of the struggle stems from uncertainty in early-stage design due to loosely defined problems and unknown parameters. Existing methods to handle this uncertainty in point-based design provide feasible, but often suboptimal, solutions that cover the range of uncertainty. Robust Optimization and Reliability Based Design Optimization are examples of point-based design methods that handle uncertainty. To maintain feasibility over the range of uncertainty, these methods accept suboptimal designs resulting in a design margin. In set-based design, design decisions are delayed preventing suboptimal final designs but at the expense of computational efficiency. This work proposes a method that evaluates a compromise between these two methodologies by evaluating the trade off of the induced regret and computational cost of keeping a larger design space. The design space covering for uncertainty (DSC-U) problem defines the metrics regret, which measures suboptimality, and space remaining, which quantifies the design space size after it is reduced. Solution methods for the DSC-U problem explore the trade space between these two metrics. When there is uncertainty in a problem, and the design space is reduced, there is the possibility that the optimal solution for the realized values of the uncertainty parameters has been eliminated; but without performing the design space reduction, it is computationally expensive to properly explore the original design space. Because of this, smart design space reductions need to be made to avoid the elimination of the optimal solution. To make smart design space reductions, designers need information regarding the design space and the trade-offs between the computational efficiency of a smaller subspace and the expected regret, or suboptimality, of the final design. As part of the DSC-U defitition, two separate spaces for the design variables and the uncertain parameters are defined. Two algorithms are presented here that solve the DSC-U problem as it is defined. A nested optimizer algorithm using a single objective optimization problem, nested in a multi-objective optimization problem is capable of finding the Pareto front in the regret-space remaining trade space for small problems. The nested optimizer algorithm is used to solve a box girder design and a cantilever tube design problems. The level set covering (LSC) algorithm solves for the Pareto front by solving the set covering problem with level sets corresponding to allowable regret levels. The LSC is used to solve a 7-variable Rosenbrock problem and a midship design problem. The presented solutions show that the DSC-U problem is a valid approach for handling uncertainty in early-stage design.PHDNaval Architecture & Marine EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/149877/1/clausl_1.pd

Nonlinear Filtering based on Log-homotopy Particle Flow : Methodological Clarification and Numerical Evaluation

The state estimation of dynamical systems based on measurements is an ubiquitous problem. This is relevant in applications like robotics, industrial manufacturing, computer vision, target tracking etc. Recursive Bayesian methodology can then be used to estimate the hidden states of a dynamical system. The procedure consists of two steps: a process update based on solving the equations modelling the state evolution, and a measurement update in which the prior knowledge about the system is improved based on the measurements. For most real world systems, both the evolution and the measurement models are nonlinear functions of the system states. Additionally, both models can also be perturbed by random noise sources, which could be non-Gaussian in their nature. Unlike linear Gaussian models, there does not exist any optimal estimation scheme for nonlinear/non-Gaussian scenarios. This thesis investigates a particular method for nonlinear and non-Gaussian data assimilation, termed as the log-homotopy based particle flow. Practical filters based on such flows have been known in the literature as Daum Huang filters (DHF), named after the developers. The key concept behind such filters is the gradual inclusion of measurements to counter a major drawback of single step update schemes like the particle filters i.e. namely the degeneracy. This could refer to a situation where the likelihood function has its probability mass well seperated from the prior density, and/or is peaked in comparison. Conventional sampling or grid based techniques do not perform well under such circumstances and in order to achieve a reasonable accuracy, could incur a high processing cost. DHF is a sampling based scheme, which provides a unique way to tackle this challenge thereby lowering the processing cost. This is achieved by dividing the single measurement update step into multiple sub steps, such that particles originating from their prior locations are graduated incrementally until they reach their final locations. The motion is controlled by a differential equation, which is numerically solved to yield the updated states. DH filters, even though not new in the literature, have not been fully explored in the detail yet. They lack the in-depth analysis that the other contemporary filters have gone through. Especially, the implementation details for the DHF are very application specific. In this work, we have pursued four main objectives. The first objective is the exploration of theoretical concepts behind DHF. Secondly, we build an understanding of the existing implementation framework and highlight its potential shortcomings. As a sub task to this, we carry out a detailed study of important factors that affect the performance of a DHF, and suggest possible improvements for each of those factors. The third objective is to use the improved implementation to derive new filtering algorithms. Finally, we have extended the DHF theory and derived new flow equations and filters to cater for more general scenarios. Improvements in the implementation architecture of a standard DHF is one of the key contributions of this thesis. The scope of the applicability of DHF is expanded by combining it with other schemes like the Sequential Markov chain Monte Carlo and the tensor decomposition based solution of the Fokker Planck equation, resulting in the development of new nonlinear filtering algorithms. The standard DHF, using improved implementation and the newly derived algorithms are tested in challenging simulated test scenarios. Detailed analysis have been carried out, together with the comparison against more established filtering schemes. Estimation error and the processing time are used as important performance parameters. We show that our new filtering algorithms exhibit marked performance improvements over the traditional schemes

Convergence of derivative-free nonmonotone Direct Search Methods for unconstrained and box-constrained mixed-integer optimization

This paper presents a class of nonmonotone Direct Search Methods that converge to stationary points of unconstrained and boxed constrained mixed-integer optimization problems. A new concept is introduced: the quasi-descent direction. A point x is stationary on a set of search directions if there exists no feasible qdd on that set. The method does not require the computation of derivatives nor the explicit manipulation of asymptotically dense matrices. Preliminary numerical experiments carried out on small to medium problems are encouraging.Universidade de Vigo/CISU

On the Use of Surrogate Functions for Mixed Variable Optimization of Simulated Systems

This research considers the efficient numerical solution of linearly constrained mixed variable programming (MVP) problems, in which the objective function is a black-box stochastic simulation, function evaluations may be computationally expensive, and derivative information is typically not available. MVP problems are those with a mixture of continuous, integer, and categorical variables, the latter of which may take on values only from a predefined list and may even be non-numeric. Mixed Variable Generalized Pattern Search with Ranking and Selection (MGPS-RS) is the only existing, provably convergent algorithm that can be applied to this class of problems. Present in this algorithm is an optional framework for constructing and managing less expensive surrogate functions as a means to reduce the number of true function evaluations that are required to find approximate solutions. In this research, the NOMADm software package, an implementation of pattern search for deterministic MVP problems, is modified to incorporate a sequential selection with memory (SSM) ranking and selection procedure for handling stochastic problems. In doing so, the underlying algorithm is modified to make the application of surrogates more efficient. A second class of surrogates based on the Nadaraya-Watson kernel regression estimator is also added to the software. Preliminary computational testing of the modified software is performed to characterize the relative efficiency of selected surrogate functions for mixed variable optimization in simulated systems

Massively parallel split-step Fourier techniques for simulating quantum systems on graphics processing units

The split-step Fourier method is a powerful technique for solving partial differential equations and simulating ultracold atomic systems of various forms. In this body of work, we focus on several variations of this method to allow for simulations of one, two, and three-dimensional quantum systems, along with several notable methods for controlling these systems. In particular, we use quantum optimal control and shortcuts to adiabaticity to study the non-adiabatic generation of superposition states in strongly correlated one-dimensional systems, analyze chaotic vortex trajectories in two dimensions by using rotation and phase imprinting methods, and create stable, threedimensional vortex structures in Bose–Einstein condensates through artificial magnetic fields generated by the evanescent field of an optical nanofiber. We also discuss algorithmic optimizations for implementing the split-step Fourier method on graphics processing units. All computational methods present in this work are demonstrated on physical systems and have been incorporated into a state-of-the-art and open-source software suite known as GPUE, which is currently the fastest quantum simulator of its kind.Okinawa Institute of Science and Technology Graduate Universit

Adaptive construction of surrogate functions for various computational mechanics models

In most science and engineering fields, numerical simulation models are often used to replicate physical systems. An attempt to imitate the true behavior of complex systems results in computationally expensive simulation models. The models are more often than not associated with a number of parameters that may be uncertain or variable. Propagation of variability from the input parameters in a simulation model to the output quantities is important for better understanding the system behavior. Variability propagation of complex systems requires repeated runs of costly simulation models with different inputs, which can be prohibitively expensive. Thus for efficient propagation, the total number of model evaluations needs to be as few as possible. An efficient way to account for the variations in the output of interest with respect to these parameters in such situations is to develop black-box surrogates. It involves replacing the expensive high-fidelity simulation model by a much cheaper model (surrogate) using a limited number of the high-fidelity simulations on a set of points called the design of experiments (DoE). The obvious challenge in surrogate modeling is to efficiently deal with simulation models that are expensive and contains a large number of uncertain parameters. Also, replication of different types of physical systems results in simulation models that vary based on the type of output (discrete or continuous models), extent of model output information (knowledge of output or output gradients or both), and whether the model is stochastic or deterministic in nature. All these variations in information from one model to the other demand development of different surrogate modeling algorithms for maximum efficiency. In this dissertation, simulation models related to application problems in the field of solid mechanics are considered that belong to each one of the above-mentioned classes of models. Different surrogate modeling strategies are proposed to deal with these models and their performance is demonstrated and compared with existing surrogate modeling algorithms. The developed algorithms, because of their non-intrusive nature, can be easily extended to simulation models of similar classes, pertaining to any other field of application

Differential Models, Numerical Simulations and Applications

This Special Issue includes 12 high-quality articles containing original research findings in the fields of differential and integro-differential models, numerical methods and efficient algorithms for parameter estimation in inverse problems, with applications to biology, biomedicine, land degradation, traffic flows problems, and manufacturing systems

Data-driven parameter and model order reduction for industrial optimisation problems with applications in naval engineering

In this work we study data-driven reduced order models with a specific focus on reduction in parameter space to fight the curse of dimensionality, especially for functions with low-intrinsic structure, in the context of digital twins. To this end we proposed two different methods to improve the accuracy of responce surfaces built using the Active Subspaces (AS): a kernel-based approach which maps the inputs onto an higher dimensional space before applying AS, and a local approach in which a clustering induced by the presence of a global active subspace is exploited to construct localized regressors. We also used AS within a multi-fidelity nonlinear autoregressive scheme to reduced the approximation error of high-dimensional scalar function using only high-fidelity data. This multi-fidelity approach has also been integrated within a non-intrusive Proper Oorthogonal Decomposition (POD) based framework in which every modal coefficient is reconstructed with a greater precision. Moving to optimization algorithms we devised an extension of the classical genetic algorithm exploiting AS to accelerate the convergence, especially for highdimensional optimization problems. We applied different combinations of such methods in a diverse range of engineering problems such as structural optimization of cruise ships, shape optimization of a combatant hull and a NACA airfoil profile, and the prediction of hydroacoustic noises. A specific attention has been devoted to the naval engineering applications and many of the methodological advances in this work have been inspired by them. This work has been conducted within the framework of the IRONTH project, an industrial Ph.D. grant financed by Fincantieri S.p.A