Simulation-based Methods for Stochastic Control and Global Optimization

Ideas of stochastic control have found applications in a variety of areas. A subclass of the problems with parameterized policies (including some stochastic impulse control problems) has received significant attention recently because of emerging applications in the areas of engineering, management, and mathematical finance. However, explicit solutions for this type of stochastic control problems only exist for some special cases, and effective numerical methods are relatively rare. Deriving efficient stochastic derivative estimators for payoff functions with discontinuities arising in many problems of practical interest is very challenging. Global optimization problems are extremely hard to solve due to the typical multimodal properties of objective functions. With the increasing availability of computing power and memory, there is a rapid development in the merging of simulation and optimization techniques. Developing new and efficient simulation-based optimization algorithms for solving stochastic control and global optimization problems is the primary goal of this thesis. First we develop a new simulation-based optimization algorithm to solve a stochastic control problem with a parameterized policy that arises in the setting of dynamic pricing and inventory control. We consider a joint dynamic pricing and inventory control problem with continuous stochastic demand and model the problem as a stochastic control problem. An explicit solution is given when a special demand model is considered. For general demand models with a parameterized policy, we develop a new simulation-based method to solve this stochastic control problem. We prove the convergence of the algorithm and show the effectiveness of the algorithm by numerical experiments. In the second part of this thesis, we focus on the problem of estimating the derivatives for a class of discontinuous payoff functions, for which existing methods are either not valid or not efficient. We derive a new unbiased stochastic derivative estimator for performance functions containing indicator functions. One important feature of this new estimator is that it can be computed from a single sample path or simulation, whereas existing estimators in the literature require additional simulations. Finally we propose a new framework for solving global optimization problems by establishing a connection with evolutionary games, and show that a particular equilibrium set of the evolutionary game is asymptotically stable. Based on this connection, we propose a Model-based Evolutionary Optimization (MEO) algorithm, which uses probabilistic models to generate new candidate solutions and uses dynamics from evolutionary game theory to govern the evolution of the probabilistic models. MEO gives new insight into the mechanism of model updating in model-based global optimization algorithms from the perspective of evolutionary game theory. Furthermore, it opens the door to developing new algorithms by using various learning algorithms and analysis techniques from evolutionary game theory

Sensitivity analysis and evolutionary optimization for building design

In order to achieve global carbon reduction targets, buildings must be designed to be energy efficient. Building performance simulation methods, together with sensitivity analysis and evolutionary optimization methods, can be used to generate design solution and performance information that can be used in identifying energy and cost efficient design solutions. Sensitivity analysis is used to identify the design variables that have the greatest impacts on the design objectives and constraints. Multi-objective evolutionary optimization is used to find a Pareto set of design solutions that optimize the conflicting design objectives while satisfying the design constraints; building design being an inherently multi-objective process. For instance, there is commonly a desire to minimise both the building energy demand and capital cost while maintaining thermal comfort. Sensitivity analysis has previously been coupled with a model-based optimization in order to reduce the computational effort of running a robust optimization and in order to provide an insight into the solution sensitivities in the neighbourhood of each optimum solution. However, there has been little research conducted to explore the extent to which the solutions found from a building design optimization can be used for a global or local sensitivity analysis, or the extent to which the local sensitivities differ from the global sensitivities. It has also been common for the sensitivity analysis to be conducted using continuous variables, whereas building optimization problems are more typically formulated using a mixture of discretized-continuous variables (with physical meaning) and categorical variables (without physical meaning). This thesis investigates three main questions; the form of global sensitivity analysis most appropriate for use with problems having mixed discretised-continuous and categorical variables; the extent to which samples taken from an optimization run can be used in a global sensitivity analysis, the optimization process causing these solutions to be biased; and the extent to which global and local sensitivities are different. The experiments conducted in this research are based on the mid-floor of a commercial office building having 5 zones, and which is located in Birmingham, UK. The optimization and sensitivity analysis problems are formulated with 16 design variables, including orientation, heating and cooling setpoints, window-to-wall ratios, start and stop time, and construction types. The design objectives are the minimisation of both energy demand and capital cost, with solution infeasibility being a function of occupant thermal comfort. It is concluded that a robust global sensitivity analysis can be achieved using stepwise regression with the use of bidirectional elimination, rank transformation of the variables and BIC (Bayesian information criterion). It is concluded that, when the optimization is based on a genetic algorithm, that solutions taken from the start of the optimization process can be reliably used in a global sensitivity analysis, and therefore, there is no need to generate a separate set of random samples for use in the sensitivity analysis. The extent to which the convergence of the variables during the optimization can be used as a proxy for the variable sensitivities has also been investigated. It is concluded that it is not possible to identify the relative importance of variables through the optimization, even though the most important variable exhibited fast and stable convergence. Finally, it is concluded that differences exist in the variable rankings resulting from the global and local sensitivity methods, although the top-ranked solutions from each approach tend to be the same. It also concluded that the sensitivity of the objectives and constraints to all variables is obtainable through a local sensitivity analysis, but that a global sensitivity analysis is only likely to identify the most important variables. The repeatability of these conclusions has been investigated and confirmed by applying the methods to the example design problem with the building being located in four different climates (Birmingham, UK; San Francisco, US; and Chicago, US)

Smart grid initialization reduces the computational complexity of multi-objective image registration based on a dual-dynamic transformation model to account for large anatomical differences

We recently demonstrated the strong potential of using dual-dynamic transformation models when tackling deformable image registration problems involving large anatomical differences. Dual-dynamic transformation models employ two moving grids instead of the common single moving grid for the target image (and single fixed grid for the source image). We previously employed powerful optimization algorithms to make use of the additional flexibility offered by a dual-dynamic transformation model with good results, directly obtaining insight into the trade-off between important registration objectives as a result of taking a multi-objective approach to optimization. However, optimization has so far been initialized using two regular grids, which still leaves a great potential of dual-dynamic transformation models untapped: a-priori grid alignment with image structures/areas that are expected to deform more. This allows (far) less grid points to be used, compared to using a sufficiently refined regular grid, leading to (far) more efficient optimization, or, equivalently, more accurate results using the same number of grid points. We study the implications of exploiting this potential by experimenting with two new smart grid initialization procedures: one manual expert-based and one automated image-feature-based. We consider a CT test case with large differences in bladder volume with and without a multi-resolution scheme and find a substantial benefit of using smart grid initialization

Phi-array: A novel method for fitness visualization and decision making in evolutionary design optimization

This article was published in the journal, Advanced Engineering Informatics [© Elsevier] and the definitive version is available at: http://dx.doi.org/10.1016/j.aei.2011.07.005There is a growing interest in integrating model based evolutionary optimization in engineering design decision making for effective search of the solution space. Most applications of evolutionary optimization are concerned with the search for optimal solutions satisfying pre-defined constraints while minimizing or maximizing desired goals. A few have explored post-optimization decision making using concepts such as Pareto optimality, but mostly in multi-objective problems. Sub-optimal solutions are usually discarded and do not contribute to decision making after optimization runs. However, the discarded ‘inferior’ solutions and their fitness contain useful information about underlying sensitivities of the system and can play an important role in creative decision making. The need for understanding the underlying system behavior is more pronounced in cases where variations in the genotype space can cause non-deterministic changes in either the fitness or phenotype space and where fitness evaluations are computationally expensive. The optimized design of an artificial lighting environment of a senior living room is used as a test case to demonstrate the need for and application of fitness visualization in genotype and phenotype spaces for effective decision making. Sub-optimal solutions are retained during optimization and visualized along with the optimum solution in a fitness array visualization system called phi-array, developed as part of this research. The optimization environment is based on genetic algorithm (GA) in which a compute-intensive raytracing rendering engine, RADIANCE, is used to evaluate the fitness of prospective design solutions. Apart from describing the development of the optimization system and demonstrating the utility of phi-array in effective decision making, this article explores optimization parameters and their effectiveness for artificial lighting design problems and the nature of their rugged fitness and constraint landscapes

Controlled particle systems for nonlinear filtering and global optimization

This thesis is concerned with the development and applications of controlled interacting particle systems for nonlinear filtering and global optimization problems. These problems are important in a number of engineering domains. In nonlinear filtering, there is a growing interest to develop geometric approaches for systems that evolve on matrix Lie groups. Examples include the problem of attitude estimation and motion tracking in aerospace engineering, robotics and computer vision. In global optimization, the challenges typically arise from the presence of a large number of local minimizers as well as the computational scalability of the solution. Gradient-free algorithms are attractive because in many practical situations, evaluating the gradient of the objective function may be computationally prohibitive. The thesis comprises two parts that are devoted to theory and applications, respectively. The theoretical part consists of three chapters that describe methods and algorithms for nonlinear filtering, global optimization, and numerical solutions of the Poisson equation that arise in both filtering and optimization. For the nonlinear filtering problem, the main contribution is to extend the feedback particle filter (FPF) algorithm to connected matrix Lie groups. In its general form, the FPF is shown to provide an intrinsic coordinate-free description of the filter that automatically satisfies the manifold constraint. The properties of the original (Euclidean) FPF, especially the gain-times-error feedback structure, are preserved in the generalization. For the global optimization problem, a controlled particle filter algorithm is introduced to numerically approximate a solution of the global optimization problem. The theoretical significance of this work comes from its variational aspects: (i) the proposed particle filter is a controlled interacting particle system where the control input represents the solution of a mean-field type optimal control problem; and (ii) the associated density transport is shown to be a gradient flow (steepest descent) for the optimal value function, with respect to the Kullback--Leibler divergence. For both the nonlinear filtering and optimization problems, the numerical implementation of the proposed algorithms require a solution of a Poisson equation. Two numerical algorithms are described for this purpose. In the Galerkin scheme, the gain function is approximated using a set of pre-defined basis functions; In the kernel-based scheme, a numerical solution is obtained by solving a certain fixed-point equation. Well-posedness results for the Poisson equation are also discussed. The second part of the thesis contains applications of the proposed algorithms to specific nonlinear filtering and optimization problems. The FPF is applied to the problem of attitude estimation - a nonlinear filtering problem on the Lie group SO(3). The formulae of the filter are described using both the rotation matrix and the quaternion coordinates. A comparison is provided between FPF and the several popular attitude filters including the multiplicative EKF, the invariant EKF, the unscented Kalman filter, the invariant ensemble Kalman filter and the bootstrap particle filter. Numerical simulations are presented to illustrate the comparison. As a practical application, experimental results for a motion tracking problem are presented. The objective is to estimate the attitude of a wrist-worn motion sensor based on the motion of the arm. In the presence of motion, considered here as the swinging motion of the arm, the observability of the sensor attitude is shown to improve. The estimation problem is mathematically formulated as a nonlinear filtering problem on the product Lie group SO(3)XSO(2), and experimental results are described using data from the gyroscope and the accelerometer installed on the sensor. For the global optimization problem, the proposed controlled particle filter is compared with several model-based algorithms that also employ probabilistic models to inform the search of the global minimizer. Examples of the model-based algorithms include the model reference adaptive search, the cross entropy, the model-based evolutionary optimization, and two algorithms based on bootstrap particle filtering. Performance comparisons are provided between the control-based and the sampling-based implementation. Results of Monte-Carlo simulations are described for several benchmark optimization problems

A nodal model for shape optimization of offset strip fin heat exchanger

Paper presented to the 10th International Conference on Heat Transfer, Fluid Mechanics and Thermodynamics, Florida, 14-16 July 2014.A methodology of a nodal model implementation for heat exchanger optimization is presented as an alternative to direct CFD-based optimization. The accuracy and the interest of the use of a nodal model in the frame of optimization process for heat exchanger geometries composed of a base pattern such as offset strip fins (OSF) geometries is detailed. Then an example of optimization of OSF profiles using the nodal model and a genetic algorithm is performed.cf201

Performance prediction and Bayesian optimization of screw compressors using Gaussian Process Regression

Optimizing the performance of screw compressors is critical for achieving high efficiency and reducing costs in various industrial and engineering applications. Often, the design and optimization processes are time-consuming owing to the underlying iterative complex analyses. In this context, the present research investigates the potential of Gaussian Process Regression (GPR) and Bayesian optimization for the prediction and optimization of the performance of an oil-flooded screw compressor. Specifically, the GPR-based surrogate model is developed to predict the compressor performance characteristics based on its four main geometrical design parameters such as wrap angle, relative length, tip speed of the male rotor and built-in volume ratio. The model is trained using a dataset comprising 19,200 data points relating the input design parameters with the compressor performance, obtained using physics-based multi-chamber thermodynamic models. While four different learning algorithms such as Support Vector Machine (SVM), Artificial Neural Network (ANN), Polynomial regression and GPR are explored, the GPR performed the best resulting in an R2 value of 0.99 for the test dataset after hyperparameter tuning. Further, the model is also experimentally validated on a completely unseen dataset, showing very good predictions with a maximum error of 5%. The resulting surrogate model is then used to optimize the compressor design parameters using Bayesian optimization. The results are compared with optimization using Genetic Algorithm (GA) and physics-based multi-chamber thermodynamic model. It was shown the proposed approach results in similar optimal design parameters but with a significantly less optimization time by a factor of 7. The study highlight the potential of machine learning-based prediction and optimization of screw compressors in engineering applications

Stochastic Systems with Cumulative Prospect Theory

Stochastic control problems arise in many fields. Traditionally, the most widely used class of performance criteria in stochastic control problems is risk-neutral. More recent attempts at introducing risk-sensitivity into stochastic control problems include the application of utility functions. The decision theory community has long debated the merits of using expected utility for modeling human behaviors, as exemplified by the Allais paradox. Substantiated by strong experimental evidence, Cumulative Prospect Theory (CPT) based performance measures have been proposed as alternatives to expected utility based performance measures for evaluating human-centric systems. Our goal is to study stochastic control problems using performance measures derived from the cumulative prospect theory. The first part of this thesis solves the problem of evaluating Markov decision processes (MDPs) using CPT-based performance measures. A well-known method of solving MDPs is dynamic programming, which has traditionally been applied with an expected utility criterion. When the performance measure is CPT-inspired, several complications arise. Firstly, when solving a problem via dynamic programming, it is important that the performance criterion has a recursive structure, which is not true for all CPT-based criteria. Secondly, we need to prove the traditional optimality criteria for the updated problems (i.e., MDPs with CPT-based performance criteria). The theorems stated in this part of the thesis answer the question: what are the conditions required on a CPT-inspired criterion such that the corresponding MDP is solvable via dynamic programming? The second part of this thesis deals with stochastic global optimization problems. Using ideas from the cumulative prospect theory, we are able to introduce a novel model-based randomized optimization algorithm: Cumulative Weighting Optimization (CWO). The key contributions of our research are: 1) proving the convergence of the algorithm to an optimal solution given a mild assumption on the initial condition; 2) showing that the well-known cross-entropy optimization algorithm is a special case of CWO-based algorithms. To the best knowledge of the author, there is no previous convergence proof for the cross-entropy method. In practice, numerical experiments have demonstrated that a CWO-based algorithm can find a better solution than the cross-entropy method. Finally, in the future, we would like to apply some of the ideas from cumulative prospect theory to games. In this thesis, we present a numerical example where cumulative prospect theory has an unexpected effect on the equilibrium points of the classic prisoner's dilemma game

Adaptive swarm optimisation assisted surrogate model for pipeline leak detection and characterisation.

Pipelines are often subject to leakage due to ageing, corrosion and weld defects. It is difficult to avoid pipeline leakage as the sources of leaks are diverse. Various pipeline leakage detection methods, including fibre optic, pressure point analysis and numerical modelling, have been proposed during the last decades. One major issue of these methods is distinguishing the leak signal without giving false alarms. Considering that the data obtained by these traditional methods are digital in nature, the machine learning model has been adopted to improve the accuracy of pipeline leakage detection. However, most of these methods rely on a large training dataset for accurate training models. It is difficult to obtain experimental data for accurate model training. Some of the reasons include the huge cost of an experimental setup for data collection to cover all possible scenarios, poor accessibility to the remote pipeline, and labour-intensive experiments. Moreover, datasets constructed from data acquired in laboratory or field tests are usually imbalanced, as leakage data samples are generated from artificial leaks. Computational fluid dynamics (CFD) offers the benefits of providing detailed and accurate pipeline leakage modelling, which may be difficult to obtain experimentally or with the aid of analytical approach. However, CFD simulation is typically time-consuming and computationally expensive, limiting its pertinence in real-time applications. In order to alleviate the high computational cost of CFD modelling, this study proposed a novel data sampling optimisation algorithm, called Adaptive Particle Swarm Optimisation Assisted Surrogate Model (PSOASM), to systematically select simulation scenarios for simulation in an adaptive and optimised manner. The algorithm was designed to place a new sample in a poorly sampled region or regions in parameter space of parametrised leakage scenarios, which the uniform sampling methods may easily miss. This was achieved using two criteria: population density of the training dataset and model prediction fitness value. The model prediction fitness value was used to enhance the global exploration capability of the surrogate model, while the population density of training data samples is beneficial to the local accuracy of the surrogate model. The proposed PSOASM was compared with four conventional sequential sampling approaches and tested on six commonly used benchmark functions in the literature. Different machine learning algorithms are explored with the developed model. The effect of the initial sample size on surrogate model performance was evaluated. Next, pipeline leakage detection analysis - with much emphasis on a multiphase flow system - was investigated in order to find the flow field parameters that provide pertinent indicators in pipeline leakage detection and characterisation. Plausible leak scenarios which may occur in the field were performed for the gas-liquid pipeline using a three-dimensional RANS CFD model. The perturbation of the pertinent flow field indicators for different leak scenarios is reported, which is expected to help in improving the understanding of multiphase flow behaviour induced by leaks. The results of the simulations were validated against the latest experimental and numerical data reported in the literature. The proposed surrogate model was later applied to pipeline leak detection and characterisation. The CFD modelling results showed that fluid flow parameters are pertinent indicators in pipeline leak detection. It was observed that upstream pipeline pressure could serve as a critical indicator for detecting leakage, even if the leak size is small. In contrast, the downstream flow rate is a dominant leakage indicator if the flow rate monitoring is chosen for leak detection. The results also reveal that when two leaks of different sizes co-occur in a single pipe, detecting the small leak becomes difficult if its size is below 25% of the large leak size. However, in the event of a double leak with equal dimensions, the leak closer to the pipe upstream is easier to detect. The results from all the analyses demonstrate the PSOASM algorithm's superiority over the well-known sequential sampling schemes employed for evaluation. The test results show that the PSOASM algorithm can be applied for pipeline leak detection with limited training datasets and provides a general framework for improving computational efficiency using adaptive surrogate modelling in various real-life applications