Use of static surrogates in hyperparameter optimization

Optimizing the hyperparameters and architecture of a neural network is a long yet necessary phase in the development of any new application. This consuming process can benefit from the elaboration of strategies designed to quickly discard low quality configurations and focus on more promising candidates. This work aims at enhancing HyperNOMAD, a library that adapts a direct search derivative-free optimization algorithm to tune both the architecture and the training of a neural network simultaneously, by targeting two keys steps of its execution and exploiting cheap approximations in the form of static surrogates to trigger the early stopping of the evaluation of a configuration and the ranking of pools of candidates. These additions to HyperNOMAD are shown to improve on its resources consumption without harming the quality of the proposed solutions.Comment: http://www.optimization-online.org/DB_HTML/2021/03/8296.htm

Un environnement pour l’optimisation sans dérivées

RÉSUMÉ : L’optimisation sans dérivées (DFO) est une branche particulière de l’optimisation qui étudie les problèmes pour lesquels les dérivées de l’objectif et/ou des contraintes ne sont pas disponibles. Généralement issues des simulations, les fonctions traitées peuvent être coûteuses à évaluer que ce soit en temps d’exécution ou en mémoire, bruitées, non dérivables ou simplement pas accessibles pour des raisons de confidentialité. La DFO spécifie des algorithmes qui se basent sur divers concepts dont certains ont été spécialement conçus pour traiter ce type de fonctions. On parle aussi parfois de boîtes grises ou noires pour souligner le peu d’information disponible sur la fonction objectif et/ou les contraintes. Il existe plusieurs solveurs et boîtes à outils qui permettent de traiter les problèmes de DFO. Le but de ce mémoire est de présenter un environnement entièrement développé en Python qui regroupe quelques outils et modules utiles dans un contexte de DFO, ainsi que quelques solveurs. Cet environnement a la particularité d’être écrit de façon modulaire. Cela permet une liberté à l’utilisateur en termes de manipulation, personnalisation et développement d’algorithmes de DFO. Dans ce mémoire, on présente la structure générale de la bibliothèque fournie, nommée DFO.py, ainsi que les détails des solveurs implémentés, en précisant les parties qui peuvent être modifiées et les différentes options disponibles. Une étude comparative est aussi présentée, le cas échéant, afin de mettre en évidence l’effet des choix des options utilisées sur l’efficacité de chaque solveur. Ces comparaisons sont visualisées à l’aide de profils de performance et de données que nous avons aussi implémentés dans un module indépendant nommé Profiles.py. Mots clés : Optimisation sans dérivées, optimisation de boîte noire, Python, profils de performance, profils de données.----------ABSTRACT : Derivative free optimization (DFO) is a branch of optimization that aims to study problems for which the derivatives of the objective function and/or constraints are not available. These functions generally come from simulation problems, therefore calling a function to evaluate a certain point can be expensive in terms of execution time or memory. The functions can be noisy, non diffrentiable or simply not accessible. DFO algorithms use different concepts and tools to adjust to these special circumstances in order to provide the best solution possible. Sometimes, the terms grey box or black box optimisation can also be used to emphasize the lack of information given about the objective function. There is a rich literature of solvers and toolboxes specialized in DFO problems. Our goal is to provide a Python environment called DFO.py, that regroups certain tools and modules that can be used in a DFO framework. The modular implementation of this environment is meant to allow a certain freedom in terms of modifying and customizing solvers. In this document, we present the general structure of DFO.py as well as the implementation details of each solver provided. These solvers can be modified by changing certain options that are mentioned in their respective sections. We also provide a benchmark study to compare the results obtained with each version of the same solver. This benchmark is done using performance and data profiles, which are part of an independent module called Profiles.py also presented in this document. Key words: Derivative free optimization, black box optimization, performance profile, data profile, Python

Tuning a variational autoencoder for data accountability problem in the Mars Science Laboratory ground data system

The Mars Curiosity rover is frequently sending back engineering and science data that goes through a pipeline of systems before reaching its final destination at the mission operations center making it prone to volume loss and data corruption. A ground data system analysis (GDSA) team is charged with the monitoring of this flow of information and the detection of anomalies in that data in order to request a re-transmission when necessary. This work presents $\Delta$-MADS, a derivative-free optimization method applied for tuning the architecture and hyperparameters of a variational autoencoder trained to detect the data with missing patches in order to assist the GDSA team in their mission

Efficient Training Under Limited Resources

Training time budget and size of the dataset are among the factors affecting the performance of a Deep Neural Network (DNN). This paper shows that Neural Architecture Search (NAS), Hyper Parameters Optimization (HPO), and Data Augmentation help DNNs perform much better while these two factors are limited. However, searching for an optimal architecture and the best hyperparameter values besides a good combination of data augmentation techniques under low resources requires many experiments. We present our approach to achieving such a goal in three steps: reducing training epoch time by compressing the model while maintaining the performance compared to the original model, preventing model overfitting when the dataset is small, and performing the hyperparameter tuning. We used NOMAD, which is a blackbox optimization software based on a derivative-free algorithm to do NAS and HPO. Our work achieved an accuracy of 86.0 % on a tiny subset of Mini-ImageNet at the ICLR 2021 Hardware Aware Efficient Training (HAET) Challenge and won second place in the competition. The competition results can be found at haet2021.github.io/challenge and our source code can be found at github.com/DouniaLakhmiri/ICLR\_HAET2021

Removal of Reactive Yellow 160 from Industrial Wastewater onto Modified Sand (Sand of Larache city beach. Morocco)

The aim of this research paper is to investigate the removal of Reactive Yellow 160 (RY160) from industrial wastewater onto Modified Sand (MS). The adsorbent was characterized by infrared spectroscopy (FT-IR), X-ray diffraction (XRD), scanning electron microscope (SEM) and energy dispersive X-ray analysis (EDXA). The effects of significant parameters such as adsorbent dose, pH, initial dye concentration, contact time, temperature were examined. It was revealed that the removal rate percentage was equal to 92.6%, the maximum adsorption capacity appeared at pH 1, and the optimal contact time for the removal of RY160 onto MS was 120min. Adsorption kinetics, isotherms and thermodynamic parameters were studied. The finding shows that the Langmuir isotherm and the Pseudo-second order kinetic model described well the adsorption process. The thermodynamic study disclosed that the adsorption of RY160 onto MS is exothermic and spontaneous with a physisorption nature. Keywords: Modified Sand, Reactive Yellow 160 (RY160), Adsorption, wastewater treatment, Industrial dyes

Quantum chemical approach (DFT) of the binary complexation of Hg(II) with L-canavanine and L-arginine.

The experimental study of the complexation of the two amino acids, L-canavanine, and L-arginine, with the mercuric ion Hg(II), was completed by the characterization by a quantum calculation based on the DFT method. This study covers electronic, energetic, and structural aspects in the neutral, deprotonated, and complexed states. The atomic net charges show that the active sites of the carboxyl, guanidyl, and amino groups are the oxygen and nitrogen atoms. In fact, the L-canavanine (Can) gave stable mercuric bidentate chelates via the amino and guanidyl groups. Hg(Can)(H2O)2, Hg(Can)2 and Hg(OH)(H2O)(Can), while the L-arginine (Arg) resulted in engagement of carboxyl and amino groups to bidentate complexes: Hg(Arg)(H2O)2, Hg(Arg)2, Hg(Arg)(OH)(H2O). The metal-ligand coordination bond is more rigid with the guanidyl and carboxyl groups than with the amino group; and the bond formed with the amino group is more rigid in the L-canavanine than L-arginine

Removal of RR-23 dye from industrial textile wastewater by adsorption on cistus ladaniferus seeds and their biochar

The use of low-cost, easily obtained and eco-friendly adsorbents has been employed as an ideal alternative for the methods of removing dyes from wastewater. Cistus ladaniferus seeds (CLS) and their biochar (BCCLS) are the biomaterials used as a bio-adsorbent for removing of Reactive red 23 (RR-23). The bio-char of cistus seed is prepared by a thermo-chemical route known as pyrolysis under optimum conditions, temperature equal to 450 °C, heating rate 21 °C.min-1 and particle sizes of 0.3 to 0.6 mm after the BCCLS is grinded with a ceramic grinder until the particle size is between 0.1 and 0.2 mm. The kinetics adsorption of dye by CLS and BCCLS are correctly described by the pseudo-2nd-order model with a correlation factor (R2 = 0.997) and (R2 = 0.998) respectively. As for the modeling of the adsorption isotherm, among the four models tested, Lungmuir type II and type I is most appropriate with a correlation factor equal to 0.999 and 0.98 for the BCCLS and the CLS respectively. On the other hand, the ability to remove the dye by the BCCLS is advantageous and the elimination efficiency reaches a maximum value of 99.237% for the BCCLS and 82% for the CLS. Keywords: Biochar, Isotherm, Adsorption, Cistus Seed, pyrolysis, Technical analysis

Removal of Reactive Yellow 135 from Wastewaterof Textile Industry onto Chitosan Extracted by Hydrothermo-Chemical Method.

The capabilities of chitosan for removing anionic dyes as reactive yellow 135 from wastewater of textile industry were examined. The chitosan was extracted from shrimp co-products "PandalusBorrealis" by the hydrothermo-chimical method in two steps. The bio-adsorbent obtained was characterized by infrared spectroscopy (FT-IR), X-ray diffraction (XRD), scanning electron microscope (SEM) and energy dispersive X-ray analysis (EDXA). Operational parameters studied were pH, contact time, adsorbate and adsorbent concentrations with a removal rate percentage of 98% and a maximum adsorption capacity of 69.244 mg/g at pH 1.9. Adsorption kinetics for the removal of reactive yellow 135 onto chitosan followed pseudo-second-order kinetics model. The examination of the isotherm data showed that the Freudlich isotherm model is the best fitting model. Keywords : Chitosan, Reactive Yellow 135 (RY135), Adsorption, Wastewater treatment, Industrial dyes, Textile industry

Modified Chitosan Immobilized on Modified Sand for Industrial Wastewater Treatment in Multicomponent Sorption: Shrimp Biowaste Processing

In this paper, modified chitosan immobilized on modified sand (MCs/MS) was synthesized and characterized by infrared spectroscopy (FT-IR), X-ray diffraction (XRD), scanning electron microscope (SEM) and energy dispersive X-ray analysis (EDXA). MCs/MS composite was used to remove Reactive Red 23 (RR23), Reactive Blue 19 (RB19) and Iron III (Fe3+) in three single-component and three binary, RR23+RB19, RR23+Fe3+ and RB19+Fe3+. Batch experiments were carried out for adsorption kinetics, isotherms and thermodynamics. Operational parameters studied were pH, contact time, temperature, adsorbate and adsorbent concentrations. Adsorption kinetics in single and binary systems of components followed pseudo- second-order kinetics model. The isotherm data in single and binary systems followed Freundlich isotherm model. Thermodynamic parameters have disclosed that the adsorption is exothermic and not spontaneous with a physical adsorption for both single and binary systems. The results showed that MCs/MS composite was an effective adsorbent to remove hazardous pollutants with a removal rate between 80% and 99.6%, the optimal contact time was between 120 and 180 min for all components in single and multicomponent system. Keywords : Modified chitosan immobilized on modified sand, Multicomponent system, Reactive Red 23, Reactive Blue 19, Iron III, Hydrothermo-Chemical method