Designing and comparing multiple portfolios of parameter configurations for online algorithm selection

National Research Foundation (NRF) Singapore under its International Research Centres in Singapore Funding Initiativ

Designing a portfolio of parameter configurations for online algorithm selection

National Research Foundation (NRF) Singapore under International Research Centres in Singapore Funding Initiativ

Auto-Sklearn 2.0: The Next Generation

Automated Machine Learning, which supports practitioners and researchers with the tedious task of manually designing machine learning pipelines, has recently achieved substantial success. In this paper we introduce new Automated Machine Learning (AutoML) techniques motivated by our winning submission to the second ChaLearn AutoML challenge, PoSH Auto-sklearn. For this, we extend Auto-sklearn with a new, simpler meta-learning technique, improve its way of handling iterative algorithms and enhance it with a successful bandit strategy for budget allocation. Furthermore, we go one step further and study the design space of AutoML itself and propose a solution towards truly hand-free AutoML. Together, these changes give rise to the next generation of our AutoML system, Auto-sklearn (2.0). We verify the improvement by these additions in a large experimental study on 39 AutoML benchmark datasets and conclude the paper by comparing to Auto-sklearn (1.0), reducing the regret by up to a factor of five

Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning

Automated Machine Learning (AutoML) supports practitioners and researchers with the tedious task of designing machine learning pipelines and has recently achieved substantial success. In this paper, we introduce new AutoML approaches motivated by our winning submission to the second ChaLearn AutoML challenge. We develop PoSH Auto-sklearn, which enables AutoML systems to work well on large datasets under rigid time limits by using a new, simple and meta-feature-free meta-learning technique and by employing a successful bandit strategy for budget allocation. However, PoSH Auto-sklearn introduces even more ways of running AutoML and might make it harder for users to set it up correctly. Therefore, we also go one step further and study the design space of AutoML itself, proposing a solution towards truly hands-free AutoML. Together, these changes give rise to the next generation of our AutoML system, Auto-sklearn 2.0. We verify the improvements by these additions in an extensive experimental study on 39 AutoML benchmark datasets. We conclude the paper by comparing to other popular AutoML frameworks and Auto-sklearn 1.0, reducing the relative error by up to a factor of 4.5, and yielding a performance in 10 minutes that is substantially better than what Auto-sklearn 1.0 achieves within an hour

Automatic Algorithm Selection for Complex Simulation Problems

To select the most suitable simulation algorithm for a given task is often difficult. This is due to intricate interactions between model features, implementation details, and runtime environment, which may strongly affect the overall performance. The thesis consists of three parts. The first part surveys existing approaches to solve the algorithm selection problem and discusses techniques to analyze simulation algorithm performance.The second part introduces a software framework for automatic simulation algorithm selection, which is evaluated in the third part.Die Auswahl des passendsten Simulationsalgorithmus für eine bestimmte Aufgabe ist oftmals schwierig. Dies liegt an der komplexen Interaktion zwischen Modelleigenschaften, Implementierungsdetails und Laufzeitumgebung. Die Arbeit ist in drei Teile gegliedert. Der erste Teil befasst sich eingehend mit Vorarbeiten zur automatischen Algorithmenauswahl, sowie mit der Leistungsanalyse von Simulationsalgorithmen. Der zweite Teil der Arbeit stellt ein Rahmenwerk zur automatischen Auswahl von Simulationsalgorithmen vor, welches dann im dritten Teil evaluiert wird

Toward guiding simulation experiments

To face the variety of simulation experiment methods, tools are needed that allow their seamless integration, guide the user through the steps of an experiment, and support him in selecting the most suitable method for the task at hand. This work presents techniques for facing such challenges. To guide users through the experiment process, six typical tasks have been identified for structuring the experiment workflow. The M&S framework JAMES II and its plug-in system is exploited to integrate various methods. Finally, an approach for automatic selection and use of such methods is realized

Conservation Return on Investment Analysis: A Review of Results, Methods, and New Directions

Conservation investments are increasingly evaluated on the basis of their return on investment (ROI). Conservation ROI analysis quantitatively measures the costs, benefits, and risks of investments so conservancies can rank or prioritize them. This paper surveys the existing conservation ROI and related literatures. We organize our synthesis around the way studies treat recurring, core elements of ROI, as a guide for practitioners and consumers of future ROI analyses. ROI analyses involve quantification of a consistent set of elements, including the definition and measurement of the conservation objective as well as identification of the relevant baselines, the type of conservation investments evaluated, and investment costs. We document the state of the art, note some open questions, and provide suggestions for future improvements in data and methods. We also describe ways ROI analysis can be extended to a broader suite of conservation outcomes than biodiversity conservation, which is the typical focus.return on investment, conservation planning, reserve site selection

Fusion Product Planning: A Market Offering Perspective

Devices that integrate multiple functions together are popular in consumer electronic markets. We describe these multifunction devices as fusion products as they fuse together products that traditionally stand alone in the marketplace. In this article, we investigate the manufacturer\u27s fusion product planning decision, adopting a market offering perspective that allows us to address the design and product portfolio decisions simultaneously. The general approach adopted is to develop and analyze a profit-maximizing model for a single firm that integrates product substitution effects in identifying an optimal market offering. In the general model, we demonstrate that the product design and portfolio decisions are analytically difficult to characterize because the number of possible portfolios can be extremely large. The managerial insight from a stylized all-in-one model and numerical analysis is that the manufacturer should, in most cases, select only a subset of fusion and single-function products to satisfy the market\u27s multidimension needs. This may explain why the function compositions available in certain product markets are limited. In particular, one of the key factors driving the product portfolio decision is the margin associated with the fusion products. If a single all-in-one fusion product has relatively high margins, then this product likely dominates the product portfolio. Also, the congruency of the constituent single-function products is an important factor. When substitution effects are relatively high (i.e., the product set is more congruent), a portfolio containing a smaller number of products is more likely to be optimal