Improving the Flexibility and Robustness of Model-Based Derivative-Free Optimization Solvers

We present DFO-LS, a software package for derivative-free optimization (DFO) for nonlinear Least-Squares (LS) problems, with optional bound constraints. Inspired by the Gauss-Newton method, DFO-LS constructs simplified linear regression models for the residuals. DFO-LS allows flexible initialization for expensive problems, whereby it can begin making progress from as few as two objective evaluations. Numerical results show DFO-LS can gain reasonable progress on some medium-scale problems with fewer objective evaluations than is needed for one gradient evaluation. DFO-LS has improved robustness to noise, allowing sample averaging, the construction of regression-based models, and multiple restart strategies together with an auto-detection mechanism. Our extensive numerical experimentation shows that restarting the solver when stagnation is detected is a cheap and effective mechanism for achieving robustness, with superior performance over both sampling and regression techniques. We also present our package Py-BOBYQA, a Python implementation of BOBYQA (Powell, 2009), which also implements robustness to noise strategies. Our numerical experiments show that Py-BOBYQA is comparable to or better than existing general DFO solvers for noisy problems. In our comparisons, we introduce a new adaptive measure of accuracy for the data profiles of noisy functions that strikes a balance between measuring the true and the noisy objective improvement.Comment: Minor edits to wordin

Efficient Hyperparameter Tuning with Dynamic Accuracy Derivative-Free Optimization

Many machine learning solutions are framed as optimization problems which rely on good hyperparameters. Algorithms for tuning these hyperparameters usually assume access to exact solutions to the underlying learning problem, which is typically not practical. Here, we apply a recent dynamic accuracy derivative-free optimization method to hyperparameter tuning, which allows inexact evaluations of the learning problem while retaining convergence guarantees. We test the method on the problem of learning elastic net weights for a logistic classifier, and demonstrate its robustness and efficiency compared to a fixed accuracy approach. This demonstrates a promising approach for hyperparameter tuning, with both convergence guarantees and practical performance.Comment: Accepted to the 12th OPT Workshop on Optimization for Machine Learning at NeurIPS 202

Modelando la evolución del SARS-COV-2 usando una aproximación fraccionaria

To show the potential of non-commensurable fractional-order dynamical systems in modeling epidemiological phenomena, we will adjust the parameters of a fractional generalization of the SIR model to describe the population distributions generated by SARS-CoV-2 in France and Colombia. Despite the completely different contexts of both countries, we will see how the system presented here manages to adequately model them thanks to the flexibility provided by the fractional-order differential equations. The data for Colombia were obtained from the records published by the Colombian Ministry of Information Technology and Communications from March 24 to July 10, 2020. Those for France were taken from the information published by the Ministry of Solidarity and Health from May 1 to September 6, 2020. As for the methodology implemented in this study, we conducted an exploratory analysis focused on solving the fractional SIR model by means of the fractional transformation method. In addition, the model parameters were adjusted using a sophisticated optimization method known as the Bound Optimization BY Quadratic Approximation (BOBYQA) algorithm. According to the results, the maximum error percentage for the evolution of the susceptible, infected, and recovered populations in France was 0.05%, 19%, and 6%, respectively, while that for the evolution of the susceptible, infected, and recovered populations in Colombia was 0.003%, 19%, and 38%, respectively. This was considered for data in which the disease began to spread and human intervention did not imply a substantial change in the community.Con el objetivo de exponer el potencial de los sistemas dinámicos de orden fraccionario, inconmensurables para la modelación de fenómenos epidemiológicos, en este artículo se ajustarán los parámetros de una generalización fraccionaria del modelo SIR (susceptibles, infectados y recuperados) para describir las distribuciones poblacionales generadas por el SARS-CoV-2 en Francia y Colombia, dos países cuyos contextos son totalmente diferentes. Asimismo, se mostrará cómo el sistema presentado logra describir adecuadamente los dos contextos debido a la flexibilidad proporcionada por las ecuaciones diferenciales de orden fraccionario. Los datos, para Colombia, fueron obtenidos del registro hecho por el Ministerio de Tecnologías de la Información y las Comunicaciones, considerándose las fechas del 24 de marzo del 2020 hasta el 10 de julio del mismo año. Por su parte, para Francia, los datos fueron tomados del monitoreo hecho por el Ministerio de Solidaridad y Salud, en un periodo comprendido desde el 1 de mayo de 2020 hasta el 6 de septiembre del mismo año. La metodología seguida es un análisis exploratorio centrado en la solución del modelo SIR fraccionario a partir del método de la transformación fraccionaria, ajustado mediante un plan sofisticado de optimización llamado algoritmo BOBYQA. Los resultados presentados muestran que el porcentaje de error máximo para la evolución de la población susceptible, infectada y recuperada en Francia es de 0.05 %, 19 % y 6 %, respectivamente. Mientras tanto, en Colombia se tiene un valor correspondiente de 0.003 %, 19 %, 38 %, esto para datos en los que se inició la dispersión de la enfermedad, donde la intervención humana no tuvo un cambio contundente en la comunidad. &nbsp

An Empirical Study of Derivative-Free-Optimization Algorithms for Targeted Black-Box Attacks in Deep Neural Networks

We perform a comprehensive study on the performance of derivative free optimization (DFO) algorithms for the generation of targeted black-box adversarial attacks on Deep Neural Network (DNN) classifiers assuming the perturbation energy is bounded by an $\ell_\infty$ constraint and the number of queries to the network is limited. This paper considers four pre-existing state-of-the-art DFO-based algorithms along with the introduction of a new algorithm built on BOBYQA, a model-based DFO method. We compare these algorithms in a variety of settings according to the fraction of images that they successfully misclassify given a maximum number of queries to the DNN. The experiments disclose how the likelihood of finding an adversarial example depends on both the algorithm used and the setting of the attack; algorithms limiting the search of adversarial example to the vertices of the $\ell^\infty$ constraint work particularly well without structural defenses, while the presented BOBYQA based algorithm works better for especially small perturbation energies. This variance in performance highlights the importance of new algorithms being compared to the state-of-the-art in a variety of settings, and the effectiveness of adversarial defenses being tested using as wide a range of algorithms as possible.Comment: arXiv admin note: text overlap with arXiv:2002.1034