Extending model-based optimization with resource-aware parallelization and for dynamic optimization problems

This thesis contains two works on the topic of sequential model-based optimization (MBO). In the first part an extension of MBO towards resource-aware parallelization is presented and in the second part MBO is adapted to optimize dynamic optimization problems. Before the newly developed methods are introduced the reader is given a detailed introduction into various aspects of MBO and related work. This covers thoughts on the choice of the initial design, the surrogate model, the acquisition functions, and the final optimization result. As most methods in this thesis rely on the Gaussian process regression it is covered in detail as well. The chapter on “Parallel MBO” dives into the topic of making use of multiple workers that can evaluate the black-box and especially focuses on the problem of heterogeneous runtimes. Strategies that tackle this problem can be divided into synchronous and asynchronous methods. Instead of proposing one configuration in an iterative fashion, as done by ordinary MBO, synchronous methods usually propose as many configurations as there are workers available. Previously proposed synchronous methods neglect the problem of heterogeneous runtimes which causes idling, when evaluations end at different times. This work presents current methods for parallel MBO that cover synchronous and asynchronous methods and presents the newly proposed Resource-Aware Model-based Optimization (RAMBO) Framework. This work shows that synchronous and asynchronous methods each have their advantages and disadvantages and that RAMBO can outperform common synchronous MBO methods if the runtime is predictable but still obtains comparable results in the worst case. The chapter on “MBO with Concept Drift” (MBO-CD) explains the adaptions that have been developed to allow optimization of black-box functions that change systematically over time. Two approaches are explained on how MBO can be taught to handle black-box functions where the relation between input and output changes over time, i.e. where a concept drift occurs. The window approach trains the surrogate only on the most recent observations. The time-as-covariate approach includes the time as an additional input variable in the surrogate, giving it the ability to learn the effect of the time. For the latter, a special acquisition function, the temporal expected improvement, is proposed

Improving adaptive seamless designs through Bayesian optimization

We propose to use Bayesian optimization (BO) to improve the efficiency of the design selection process in clinical trials. BO is a method to optimize expensive black-box functions, by using a regression as a surrogate to guide the search. In clinical trials, planning test procedures and sample sizes is a crucial task. A common goal is to maximize the test power, given a set of treatments, corresponding effect sizes, and a total number of samples. From a wide range of possible designs, we aim to select the best one in a short time to allow quick decisions. The standard approach to simulate the power for each single design can become too time consuming. When the number of possible designs becomes very large, either large computational resources are required or an exhaustive exploration of all possible designs takes too long. Here, we propose to use BO to quickly find a clinical trial design with high power from a large number of candidate designs. We demonstrate the effectiveness of our approach by optimizing the power of adaptive seamless designs for different sets of treatment effect sizes. Comparing BO with an exhaustive evaluation of all candidate designs shows that BO finds competitive designs in a fraction of the time

Performance evaluation and hyperparameter tuning of statistical and machine-learning models using spatial data

Machine-learning algorithms have gained popularity in recent years in the field of ecological modeling due to their promising results in predictive performance of classification problems. While the application of such algorithms has been highly simplified in the last years due to their well-documented integration in commonly used statistical programming languages such as R, there are several practical challenges in the field of ecological modeling related to unbiased performance estimation, optimization of algorithms using hyperparameter tuning and spatial autocorrelation. We address these issues in the comparison of several widely used machine-learning algorithms such as Boosted Regression Trees (BRT), k-Nearest Neighbor (WKNN), Random Forest (RF) and Support Vector Machine (SVM) to traditional parametric algorithms such as logistic regression (GLM) and semi-parametric ones like generalized additive models (GAM). Different nested cross-validation methods including hyperparameter tuning methods are used to evaluate model performances with the aim to receive bias-reduced performance estimates. As a case study the spatial distribution of forest disease Diplodia sapinea in the Basque Country in Spain is investigated using common environmental variables such as temperature, precipitation, soil or lithology as predictors. Results show that GAM and RF (mean AUROC estimates 0.708 and 0.699) outperform all other methods in predictive accuracy. The effect of hyperparameter tuning saturates at around 50 iterations for this data set. The AUROC differences between the bias-reduced (spatial cross-validation) and overoptimistic (non-spatial cross-validation) performance estimates of the GAM and RF are 0.167 (24%) and 0.213 (30%), respectively. It is recommended to also use spatial partitioning for cross-validation hyperparameter tuning of spatial data

In situ temperature determination using magnetic resonance spectroscopy thermometry for noninvasive postmortem examinations

Magnetic resonance spectroscopy (MRS) thermometry offers a noninvasive, localized method for estimating temperature by leveraging the temperature‐dependent chemical shift of water relative to a temperature‐stable reference metabolite under suitable calibration. Consequentially, this technique has significant potential as a tool for postmortem MR examinations in forensic medicine and pathology. In these examinations, the deceased are examined at a wide range of body temperatures, and MRS thermometry may be used for the temperature adjustment of magnetic resonance imaging (MRI) protocols or for corrections in the analysis of MRI or MRS data. However, it is not yet clear to what extent postmortem changes may influence temperature estimation with MRS thermometry. In addition, N‐acetylaspartate, which is commonly used as an in vivo reference metabolite, is known to decrease with increasing postmortem interval (PMI). This study shows that lactate, which is not only present in significant amounts postmortem but also has a temperature‐stable chemical shift, can serve as a suitable reference metabolite for postmortem MRS thermometry. Using lactate, temperature estimation in postmortem brain tissue of severed sheep heads was accurate up to 60 h after death, with a mean absolute error of less than 0.5°C. For this purpose, published calibrations intended for in vivo measurements were used. Although postmortem decomposition resulted in severe metabolic changes, no consistent deviations were observed between measurements with an MR‐compatible temperature probe and MRS thermometry with lactate as a reference metabolite. In addition, MRS thermometry was applied to 84 deceased who underwent a MR examination as part of the legal examination. MRS thermometry provided plausible results of brain temperature in comparison with rectal temperature. Even for deceased with a PMI well above 60 h, MRS thermometry still provided reliable readings. The results show a good suitability of MRS thermometry for postmortem examinations in forensic medicine

Two-dimensional electron gas formation in undoped In[0.75]Ga[0.25]As/In[0.75]Al[0.25]As quantum wells

We report on the achievement of a two-dimensional electron gas in completely undoped In[0.75]Al[0.25]As/In[0.75]Ga[0.25]As metamorphic quantum wells. Using these structures we were able to reduce the carrier density, with respect to reported values in similar modulation-doped structures. We found experimentally that the electronic charge in the quantum well is likely due to a deep-level donor state in the In[0.75]Al[0.25]As barrier band gap, whose energy lies within the In[0.75]Ga[0.25]As/In[0.75]Al[0.25]As conduction band discontinuity. This result is further confirmed through a Poisson-Schroedinger simulation of the two-dimensional electron gas structure.Comment: 17 pages, 6 figures, to be published in J. Vac. Sci. Technol.

MODES: model-based optimization on distributed embedded systems

The predictive performance of a machine learning model highly depends on the corresponding hyper-parameter setting. Hence, hyper-parameter tuning is often indispensable. Normally such tuning requires the dedicated machine learning model to be trained and evaluated on centralized data to obtain a performance estimate. However, in a distributed machine learning scenario, it is not always possible to collect all the data from all nodes due to privacy concerns or storage limitations. Moreover, if data has to be transferred through low bandwidth connections it reduces the time available for tuning. Model-Based Optimization (MBO) is one state-of-the-art method for tuning hyper-parameters but the application on distributed machine learning models or federated learning lacks research. This work proposes a framework MODES that allows to deploy MBO on resource-constrained distributed embedded systems. Each node trains an individual model based on its local data. The goal is to optimize the combined prediction accuracy. The presented framework offers two optimization modes: (1) MODES-B considers the whole ensemble as a single black box and optimizes the hyper-parameters of each individual model jointly, and (2) MODES-I considers all models as clones of the same black box which allows it to efficiently parallelize the optimization in a distributed setting. We evaluate MODES by conducting experiments on the optimization for the hyper-parameters of a random forest and a multi-layer perceptron. The experimental results demonstrate that, with an improvement in terms of mean accuracy (MODES-B), run-time efficiency (MODES-I), and statistical stability for both modes, MODES outperforms the baseline, i.e., carry out tuning with MBO on each node individually with its local sub-data set