Candidate selection and instance ordering for realtime algorithm configuration

Many modern combinatorial solvers have a variety of parameters through which a user can customise their behaviour. Algorithm configuration is the process of selecting good values for these parameters in order to improve performance. Time and again algorithm configuration has been shown to significantly improve the performance of many algorithms for solving challenging computational problems. Automated systems for tuning parameters regularly out-perform human experts, sometimes but orders of magnitude. Online algorithm configurators, such as ReACTR, are able to tune a solver online without incurring costly offline training. As such ReACTR’s main focus is on runtime minimisation while solving combinatorial problems. To do this ReACTR adopts a one-pass methodology where each instance in a stream of instances to be solved is considered only as it arrives. As such ReACTR’s performance is sensitive to the order in which instances arrive. It is still not understood which instance orderings positively or negatively effect the performance of ReACTR. This paper investigates the effect of instance ordering and grouping by empirically evaluating different instance orderings based on difficulty and feature values. Though the end use is generally unable to control the order in which instances arrive it is important to understand which orderings impact Re- ACTR’s performance and to what extent. This study also has practical benefit as such orderings can occur organically. For example as business grows the problems it may encounter, such as routing or scheduling, often grow in size and difficulty. ReACTR’s performance also depends strongly configuration selection procedure used. This component controls which configurations are selected to run in parallel from the internal configuration pool. This paper evaluates various ranking mechanisms and different ways of combining them to better understand how the candidate selection procedure affects realtime algorithm configuration. We show that certain selection procedures are superior to others and that the order which instances arrive in determines which selection procedure performs best. We find that both instance order and grouping can significantly affect the overall solving time of the online automatic algorithm configurator ReACTR. One of the more surprising discoveries is that having groupings of similar instances can actually negatively impact on the overall performance of the configurator. In particular we show that orderings based on nearly any instance feature values can lead to significant reductions in total runtime over random instance orderings. In addition, certain candidate selection procedures are more suited to certain orderings than others and selecting the correct one can show a marked improvement in solving times

Capping methods for the automatic configuration of optimization algorithms

Automatic configuration techniques are widely and successfully used to find good parameter settings for optimization algorithms. Configuration is costly, because it is necessary to evaluate many configurations on different instances. For decision problems, when the objective is to minimize the running time of the algorithm, many configurators implement capping methods to discard poor configurations early. Such methods are not directly applicable to optimization problems, when the objective is to optimize the cost of the best solution found, given a predefined running time limit. We propose new capping methods for the automatic configuration of optimization algorithms. They use the previous executions to determine a performance envelope, which is used to evaluate new executions and cap those that do not satisfy the envelope conditions. We integrate the capping methods into the irace configurator and evaluate them on different optimization scenarios. Our results show that the proposed methods can save from about 5% to 78% of the configuration effort, while finding configurations of the same quality. Based on the computational analysis, we identify two conservative and two aggressive methods, that save an average of about 20% and 45% of the configuration effort, respectively. We also provide evidence that capping can help to better use the available budget in scenarios with a configuration time limit.This research has been supported by Coordenação de Aperfeiçoa-mento de Pessoal de Nível Superior – Brasil (CAPES) – Finance Code001. M. de Souza acknowledges the support of the Santa Catarina State University, Brasil. M. Ritt acknowledges the support of CNPq, Brasil (grant 437859/2018-5) and Google Research Latin America (grant25111). M. López-Ibáñez is a ‘‘Beatriz Galindo’’ Senior Distinguished Researcher (BEAGAL 18/00053) funded by the Spanish Ministry of Science and Innovation (MICINN). This research is partially funded by TAILOR ICT-48 Network (No 952215) funded by EU Horizon 2020 research and innovation programme. Funding for open access charge: Universidad de Málaga / CBU

Real-time algorithm configuration

This dissertation presents a number of contributions to the field of algorithm configur- ation. In particular, we present an extension to the algorithm configuration problem, real-time algorithm configuration, where configuration occurs online on a stream of instances, without the need for prior training, and problem solutions are returned in the shortest time possible. We propose a framework for solving the real-time algorithm configuration problem, ReACT. With ReACT we demonstrate that by using the parallel computing architectures, commonplace in many systems today, and a robust aggregate ranking system, configuration can occur without any impact on performance from the perspective of the user. This is achieved by means of a racing procedure. We show two concrete instantiations of the framework, and show them to be on a par with or even exceed the state-of-the-art in offline algorithm configuration using empirical evaluations on a range of combinatorial problems from the literature. We discuss, assess, and provide justification for each of the components used in our framework instantiations. Specifically, we show that the TrueSkill ranking system commonly used to rank players’ skill in multiplayer games can be used to accurately es- timate the quality of an algorithm’s configuration using only censored results from races between algorithm configurations. We confirm that the order that problem instances arrive in influences the configuration performance and that the optimal selection of configurations to participate in races is dependent on the distribution of the incoming in- stance stream. We outline how to maintain a pool of quality configurations by removing underperforming configurations, and techniques to generate replacement configurations with minimal computational overhead. Finally, we show that the configuration space can be reduced using feature selection techniques from the machine learning literature, and that doing so can provide a boost in configuration performance

Blueswitch: Enabling provably consistent configuration of network switches

Previous research on consistent updates for distributed network configurations has focused on solutions for centralized networkconfiguration controllers. However, such work does not address the complexity of modern switch datapaths. Modern commodity switches expose opaque configuration mechanisms, with minimal guarantees for datapath consistency and with unclear configuration semantics. Furthermore, would-be solutions for distributed consistent updates must take into account the configuration guarantees provided by each individual switch – plus the compositional problems of distributed control and multi-switch configurations that considerably transcend the single-switch problems. In this paper, we focus on the behavior of individual switches, and demonstrate that even simple rule updates result in inconsistent packet switching in multi-table datapaths. We demonstrate that consistent configuration updates require guarantees of strong switch-level atomicity from both hardware and software layers of switches – even in a single switch. In short, the multiple-switch problems cannot be reasonably approached until single-switch consistency can be resolved. We present a hardware design that supports a transactional configuration mechanism, and provides packet-consistent configuration: all packets traversing the datapath will encounter either the old configuration or the new one, and never an inconsistent mix of the two. Unlike previous work, our design does not require modifications to network packets. We precisely specify the hardwaresoftware protocol for switch configuration; this enables us to prove the correctness of the design, and to provide well-specified invariants that the software driver must maintain for correctness. We implement our prototype switch design using the NetFPGA-10G hardware platform, and evaluate our prototype against commercial off-the-shelf switches.This work was jointly supported by the Defense Advanced Research Projects Agency (DARPA) and the Air Force Research Laboratory (AFRL), under contract FA8750-11-C-0249. The views, opinions, and/or findings contained in this article/presentation are those of the author/ presenter and should not be interpreted as representing the official views or policies, either expressed or implied, of the Department of Defense or the U.S. Government. We also acknowledge the support of the UK EPSRC for contributing to parts of our work, through grant EP/H040536/1. Additional data related to this publication is available at the http://www.cl.cam.ac. uk/research/srg/netfpga/blueswitch/ data repository.This is the author accepted manuscript. The final version is available from IEEE via http://dx.doi.org/10.1109/ANCS.2015.711011

Working Notes from the 1992 AAAI Workshop on Automating Software Design. Theme: Domain Specific Software Design

The goal of this workshop is to identify different architectural approaches to building domain-specific software design systems and to explore issues unique to domain-specific (vs. general-purpose) software design. Some general issues that cut across the particular software design domain include: (1) knowledge representation, acquisition, and maintenance; (2) specialized software design techniques; and (3) user interaction and user interface

Réagir et s’adapter à son environnement: Concevoir des méthodes autonomes pour l’optimisation combinatoire à plusieurs objectifs

Large-scale optimisation problems are usually hard to solve optimally. Approximation algorithms such as metaheuristics, able to quickly find sub-optimal solutions, are often preferred. This thesis focuses on multi-objective local search (MOLS) algorithms, metaheuristics able to deal with the simultaneous optimisation of multiple criteria. As many algorithms, metaheuristics expose many parameters that significantly impact their performance. These parameters can be either predicted and set before the execution of the algorithm, or dynamically modified during the execution itself.While in the last decade many advances have been made on the automatic design of algorithms, the great majority of them only deal with single-objective algorithms and the optimisation of a single performance indicator such as the algorithm running time or the final solution quality. In this thesis, we investigate the relations between automatic algorithm design and multi-objective optimisation, with an application on MOLS algorithms.We first review possible MOLS strategies ans parameters and present a general, highly configurable, MOLS framework. We also propose MO-ParamILS, an automatic configurator specifically designed to deal with multiple performance indicators. Then, we conduct several studies on the automatic offline design of MOLS algorithms on multiple combinatorial bi-objective problems. Finally, we discuss two online extensions of classical algorithm configuration: first the integration of parameter control mechanisms, to benefit from having multiple configuration predictions; then the use of configuration schedules, to sequentially use multiple configurations.Les problèmes d’optimisation à grande échelle sont généralement difficiles à résoudre de façon optimale. Des algorithmes d’approximation tels que les métaheuristiques, capables de trouver rapidement des solutions sous-optimales, sont souvent préférés. Cette thèse porte sur les algorithmes de recherche locale multi-objectif (MOLS), des métaheuristiques capables de traiter l’optimisation simultanée de plusieurs critères. Comme de nombreux algorithmes, les MOLS exposent de nombreux paramètres qui ont un impact important sur leurs performances. Ces paramètres peuvent être soit prédits et définis avant l’exécution de l’algorithme, soit ensuite modifiés dynamiquement.Alors que de nombreux progrès ont récemment été réalisés pour la conception automatique d’algorithmes, la grande majorité d’entre eux ne traitent que d’algorithmes mono-objectif et l’optimisation d’un unique indicateur de performance. Dans cette thèse, nous étudions les relations entre la conception automatique d’algorithmes et l’optimisation multi-objective.Nous passons d’abord en revue les stratégies MOLS possibles et présentons un framework MOLS général et hautement configurable. Nous proposons également MO-ParamILS, un configurateur automatique spécialement conçu pour gérer plusieurs indicateurs de performance. Nous menons ensuite plusieurs études sur la conception automatique de MOLS sur de multiples problèmes combinatoires bi-objectifs. Enfin, nous discutons deux extensions de la configuration d’algorithme classique : d’abord l’intégration des mécanismes de contrôle de paramètres, pour bénéficier de multiples prédictions de configuration; puis l’utilisation séquentielle de plusieurs configurations

Providing machine level data for cloud based analytics

The target of this thesis was to investigate the problems, possibilities and background of transporting data collected in a manufacturing process to a cloud environment for analytics. The underlying reason for this was to spot themes and topics that need to be addressed when planning a real-life project aiming to improve manufacturing with cloud manufacturing. To achieve this, first the underlying theories were studied to understand why we actually need cloud analytics and what kind of paradigms could help in achieving it. The reasoning was mostly based on themes revolving around Germany’s Industry 4.0 initiative. After that, the state of the art was researched to understand what has been already done and how some fundamental problems have been solved. From this base, requirements for a solution model were made. After requirements for a good system were formulated, some of the most promising technologies were researched and after that evaluated according to the requirements. This thesis was able to highlight many core problems that modern cloud analytic systems for manufacturing will face. The most central technological findings revolve around a vital need for industry wide standardization, which is a theme that requires both customers and vendors: customers need to demand vendors to embrace common open standards and vendors need to respond to this need. Another core problem discussed in this thesis was the usage of different cloud services. Many manufacturing giants offer their own platforms for providing cloud analytics, but also bring some fundamental problems to play. On the other hand open cloud platform providers like Microsoft Azure and Amazon Web Services offer a very large offering of components, services and applications with competitive pricing. Altogether this thesis produced some topics that require a thorough analysis when planning a system like this and some insight into the core problems. In addition, some recommendations were produced on how to tackle problems in ways that transition well in to the future as the field is evolving rapidly and future compatibility is important.Tämän diplomityön tarkoituksena oli tutkia datan keräystä tuotantoprosessista pilviympäristöön analytiikkaa varten, sekä siihen liittyviä ilmiöitä, ongelmia sekä mahdollisuuksia. Motivaatioina tähän kaikkeen oli havaita mahdollisia aiheita ja aihepiirejä, joita tulisi tutkia ja arvioida reaalimaailman projektia suunnitellessa. Tavoitteisiin pääsemiseksi aluksi tutkittiin teorioita, joiden avulla haettiin ymmärrystä miksi oikeastaan pilvianalytiikkaa tarvitaan ja minkälaiset ratkaisumallit voisivat auttaa sen toteuttamisessa. Käsitellyt teemat liittyivät hyvin vahvasti Saksan Industry 4.0 hankkeeseen ja siihen liittyvään tutkimustyöhön. Tämän jälkeen tarkasteltiin nykytilaa ja minkälaisilla ratkaisuilla ongelmaa yleensä on ratkaistu. Tämän pohjalta muotoiltiin vaatimuksia ratkaisumallille. Vaatimusmäärittelyiden pohjalta alettiin tutkia lupaavimpia teknologioita ja tekniikoita vaatimusten pohjalta. Työ onnistui nostamaan esille useita keskeisiä ongelmia aiheuttavia teemoja, mitä modernit pilvianalytiikkaratkaisut joutuvat kohtaamaan. Keskeisimmät tekniikkaan liittyvät löydökset ovat vahvasti kytköksissä toimialojen laajuisen standardoinnin tarpeeseen. Tämä on teema, joka vaatii toimenpiteitä niin asiakkailta kuin toimittajilta: asiakkaiden tulee vaatia toimittajia perustamaan ratkaisunsa avoimille ja yhteisille standardeille ja toimittajien tulee vastata tähän tarpeeseen. Toinen keskeinen käsitelty ongelma oli erilaisten pilvipalveluiden vertailu ja niistä oikean valinta. Useat valmistavan teollisuuden toimittajajätit tarjoavat omia alustoja pilvianalytiikalle, jotka tarjoavat valmiita ratkaisuja ainakin joihinkin ongelmiin. Toisaalta avoimet pilvitoimittajat kuten Microsoft (Azure) ja Amazon (AWS) tarjoavat erittäin laajalla skaalalla erilaisia komponentteja, palveluita ja sovelluksia kilpailukykyisellä hinnoittelulla. Yhteenvetona tämä diplomityö nosti ja tuotti pohdintaa useasta eri aihealueesta, mitkä vaativat huolellista analyysiä reaalimaailman ratkaisua suunnitellessa. Lisäksi pohdinnan perusteella pystyttiin muotoilemaan joitakin ratkaisuehdotuksia ongelmien välttämiseksi tavalla, jotka tulevaisuudessa tukevat käsitellyn alan nopeasti kehittyvää luonnetta ja korkeita vaatimuksia yhteensopivuudelle

Scalable and Reliable Middlebox Deployment

Middleboxes are pervasive in modern computer networks providing functionalities beyond mere packet forwarding. Load balancers, intrusion detection systems, and network address translators are typical examples of middleboxes. Despite their benefits, middleboxes come with several challenges with respect to their scalability and reliability. The goal of this thesis is to devise middlebox deployment solutions that are cost effective, scalable, and fault tolerant. The thesis includes three main contributions: First, distributed service function chaining with multiple instances of a middlebox deployed on different physical servers to optimize resource usage; Second, Constellation, a geo-distributed middlebox framework enabling a middlebox application to operate with high performance across wide area networks; Third, a fault tolerant service function chaining system