Sequential vs. Integrated Algorithm Selection and Configuration: A Case Study for the Modular CMA-ES

When faced with a specific optimization problem, choosing which algorithm to use is always a tough task. Not only is there a vast variety of algorithms to select from, but these algorithms often are controlled by many hyperparameters, which need to be tuned in order to achieve the best performance possible. Usually, this problem is separated into two parts: algorithm selection and algorithm configuration. With the significant advances made in Machine Learning, however, these problems can be integrated into a combined algorithm selection and hyperparameter optimization task, commonly known as the CASH problem. In this work we compare sequential and integrated algorithm selection and configuration approaches for the case of selecting and tuning the best out of 4608 variants of the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) tested on the Black Box Optimization Benchmark (BBOB) suite. We first show that the ranking of the modular CMA-ES variants depends to a large extent on the quality of the hyperparameters. This implies that even a sequential approach based on complete enumeration of the algorithm space will likely result in sub-optimal solutions. In fact, we show that the integrated approach manages to provide competitive results at a much smaller computational cost. We also compare two different mixed-integer algorithm configuration techniques, called irace and Mixed-Integer Parallel Efficient Global Optimization (MIP-EGO). While we show that the two methods differ significantly in their treatment of the exploration-exploitation balance, their overall performances are very similar

Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges

Most machine learning algorithms are configured by a set of hyperparameters whose values must be carefully chosen and which often considerably impact performance. To avoid a time-consuming and irreproducible manual process of trial-and-error to find well-performing hyperparameter configurations, various automatic hyperparameter optimization (HPO) methods—for example, based on resampling error estimation for supervised machine learning—can be employed. After introducing HPO from a general perspective, this paper reviews important HPO methods, from simple techniques such as grid or random search to more advanced methods like evolution strategies, Bayesian optimization, Hyperband, and racing. This work gives practical recommendations regarding important choices to be made when conducting HPO, including the HPO algorithms themselves, performance evaluation, how to combine HPO with machine learning pipelines, runtime improvements, and parallelization. This article is categorized under: Algorithmic Development > Statistics Technologies > Machine Learning Technologies > Prediction

Feasibility of improving risk stratification in the inherited cardiac conditions

Fatal ventricular arrhythmias can occur in patients with Hypertrophic Cardiomyopathy, Brugada Syndrome and rarely in patients with normal cardiac investigations. Despite very different pathogeneses, we hypothesised that a common electrophysiological substrate precipitates these arrhythmias and could be used as a marker for risk stratification. In Chapter 3 of this thesis, we found that fewer than half the cardiac arrest survivors with Brugada Syndrome would have been offered prophylactic defibrillators based on current risk scoring, highlighting the need for better risk stratification. Our group previously used a commercially available 252-electrode vest which constructs ventricular electrograms onto a CT image of the heart to show exercise related differences in high-risk patients. In Chapter 4, we applied this method to Brugada patients, but could not reproduce prior results. Further investigation revealed periodic changes in activation patterns after exercise that could explain this discrepancy. An alternative matrix approach was developed to overcome this problem. Exercise induced conduction heterogeneity differentiated Brugada patients from unaffected controls, but not those surviving cardiac arrest. However, if considered alongside spontaneous type 1 ECG and syncope, inducible conduction heterogeneity markedly improved identification of Brugada cardiac arrest survivors. In Chapter 5 the method was shown to differentiate idiopathic ventricular fibrillation patients from those fully recovered from acute ischaemic cardiac arrest, implying a permanent electrophysiological abnormality. In Chapter 8, we showed prolonged mean local activation times and activation-recovery intervals in hypertrophic cardiomyopathy cardiac arrest survivors compared to those without previous ventricular arrhythmia. These metrics were combined into both logistic regression and support vector machine models to strongly differentiate the groups. We concluded that electrophysiological changes could identify cardiac arrest survivors in various cardiac conditions, but a single factor common pathway was not established. Prospective studies are required to determine if using these parameters could enhance current risk stratification for sudden death.Open Acces