International audienceBesides the conventional and established applications of sensitivity analysis in the field of optimization, sensitivity analysis-based approaches can also be used in research on systematic performance analysis of meta-heuristic optimization algorithms, and the construction of synthetic industrially relevant benchmark problems. By means of academic examples as well as industrial case studies, we demonstrate innovative approaches which set steps towards systematic and representative benchmarking and empirical optimization algorithm performance analysis

Baldanzini, Niccolò

Müller, Ralf

Pierini, Marco

Sala, Ramses

English

Archive Ouverte en Sciences de l'Information et de la Communication

HAL Id: hal-02308998
https://hal.archives-ouvertes.fr/hal-02308998
Submitted on 8 Oct 2019
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
Copyright
Sensitivity Analysis in Systematic and Representative
Benchmarking of Optimization Algorithm Performance
Ramses Sala, Ralf Müller, Niccolò Baldanzini, Marco Pierini
To cite this version:
Ramses Sala, Ralf Müller, Niccolò Baldanzini, Marco Pierini. Sensitivity Analysis in Systematic and
Representative Benchmarking of Optimization Algorithm Performance. Ninth International Confer-
ence on Sensitivity Analysis of Model Output, Oct 2019, Barcelona, Spain. ￿hal-02308998￿
 
Ninth International Conference on Sensitivity Analysis of Model Output, SAMO, Barcelona, 2019 
29 Oktober, 2019 1 38815 
 
Sensitivity Analysis in Systematic and Representative Benchmarking of 
Optimization Algorithm Performance  
 
Ramses Sala1, Ralf Müller1, Niccolò Baldanzini2 and Marco Pierini2 
 
1
Institute of Applied Mechanics, Department of Mechanical and Process Engineering, TU Kaiserslautern, Germany 
2
Department of Industrial Engineering, University of Florence, Italy 
 
Introduction 
Besides the conventional and established applications of sensitivity analysis in the field of 
optimization [1], sensitivity analysis-based approaches can also be used in research on systematic 
performance analysis of meta-heuristic optimization algorithms, and the construction of synthetic 
industrially relevant benchmark problems. 
The No Free Lunch Theorems for search and optimization [2] imply that there is no universal best 
algorithm for all problem types. The challenges in the field of non-convex optimization are thus 
moving towards the quests to identify and develop efficient algorithms for particular problem types. 
Industrially relevant product and process engineering problems often involve computationally 
expensive, highly non-linear, non-convex simulations responses. The industrial problems for which 
performance matters the most, are also those for which it is the most difficult to identify and develop 
efficient optimization approaches. Therefore often artificial test functions and benchmark problems 
are used in the optimization community. For most of the commonly used artificial benchmark 
problems for meta-heuristic optimization algorithms, it is, however, difficult to relate relative 
algorithm performance on one artificial problem, to any real-world problems, simulation-based 
problems or other artificial benchmark problems. In order to increase the value of such benchmarks 
from a set of many isolated statistical tests to scientific tests with some generalization value, 
benchmark functions with a more systematic structure are required. Various forms of variable 
screening and sensitivity analysis [3] can aid the construction of such improved benchmark problems 
[4]. Besides more systematic comparative assessments of optimization algorithms, it is also important 
to find ways to relate algorithm performance on artificial benchmark tests to real-world and industrial 
optimization problems [5]. Also, here, sensitivity analysis could aid to characterize the response 
behavior of real-world and simulation-based problems in order to match the problem features with 
particular synthetic benchmarks. By means of academic examples as well as industrial case studies, 
we demonstrate innovative approaches which set steps towards systematic and representative 
benchmarking and empirical optimization algorithm performance analysis. 
 
Characterization of Optimization Problems (Summary) 
A first step towards obtaining generalizable information from optimization benchmarks is to answer 
questions such as: What will happen to the optimization algorithm performance when a characteristic 
feature of the benchmark problem would be changed? Problem changes of practical interest could for 
example involve modifications of the problem dimension, the distribution of the first and higher order 
sensitivity indices of the model output w.r.t the model input, the smoothness of the objective function, 
or its multi-modality. 
To investigate the answers to questions of the previously stated kind, benchmark functions 
parameterized w.r.t several function features were developed. By means of such parameterized test 
problems, the performance of meta-heuristic algorithms is systematically investigated, showing clear 
trends and generalizable quantitative performance characteristics of optimization algorithms w.r.t. 
particular problem features. 
 
Most multidisciplinary engineering optimization problems are not only composed of an objective 
function but also contain one or more constraints. The objective and all the constraint functions can 
be dependent on the results of several computationally expensive simulations with non-linear 
responses. Not only the structure of the individual simulations responses is important for the 
performance of optimization algorithms, but also the structure and dependencies between the 
 
Ninth International Conference on Sensitivity Analysis of Model Output, SAMO, Barcelona, 2019 
29 Oktober, 2019 2 38815 
 
simulation responses. Design variables which are important for the objective function, are not 
necessarily also those variables which are important for those of the non-linear constraints. In order 
to develop synthetic benchmark problems which are representative for particular real-world or 
industrial problems, it is necessary to be able to construct benchmark problems with similar response 
structures, and similar relations between the objective and constraints. 
To address this challenge, various forms of sensitivity analysis could be used to characterize the 
structure of the real-world and industrially relevant optimization problems. These characteristics 
could then be used to construct computationally affordable benchmark functions, on which the 
performance of different algorithms and algorithm settings can be investigated. Such an approach 
could be used to select efficient algorithms for particular optimization problems, and to tune or 
optimize the optimization algorithm parameters to problems with particular features. An approach 
along the lines of these ideas has been developed and tested on a relatively complex industrial 
engineering problem, namely the optimization of car body structures involving crashworthiness, 
vibrational comfort, and weight requirements. 
 
Discussion and Outlook 
Although the results of the presented ideas and approaches are promising, the area of research of 
using sensitivity analysis methods for systematic insight into optimization algorithm performance is 
still very young, and not widespread. There is much potential for improvement, and there are still 
many open challenges: Which variable screening and or sensitivity analysis methods are the most 
suitable for such applications? How to effectively balance the trade-off between function evaluation 
cost and the nature and quality of the obtained sensitivity measures? Also, from the computational 
simulation perspective, various approaches to increase the efficiency of the response characterization 
need to be investigated [6]. Sensitivity analysis-based approaches can contribute towards systematic 
optimization algorithm assessments, with more generalizable results. They also seems an essential 
ingredient to extricate the problem of matching and developing efficient algorithms for complex 
computationally expensive industrial optimization problems. 
 
References 
 
[1] Castillo, E., Mínguez, R., & Castillo, C. (2008). Sensitivity analysis in optimization and 
reliability problems. Reliability Engineering & System Safety, 93(12), 1788-1800. 
 
[2] Wolpert, D. H., & Macready, W. G. (1997). No free lunch theorems for optimization. IEEE 
transactions on evolutionary computation, 1(1), 67-82. 
 
[3] Iooss, B., & Lemaître, P. (2015). A review on global sensitivity analysis methods. In 
Uncertainty management in simulation-optimization of complex systems (pp. 101-122). Springer, 
Boston, MA. 
 
[4] Sala, R., Baldanzini, N., & Pierini, M. (2017). Global optimization test problems based on 
random field composition. Optimization Letters, 11(4), 699-713. 
 
[5] Sala, R., Baldanzini, N., & Pierini, M. (2016). Representative surrogate problems as test 
functions for expensive simulators in multidisciplinary design optimization of vehicle structures. 
Structural and Multidisciplinary Optimization, 54(3), 449-468. 
 
[6] Biehler, J., Gee, M. W., & Wall, W. A. (2015). Towards efficient uncertainty quantification in 
complex and large-scale biomechanical problems based on a Bayesian multi-fidelity scheme. 
Biomechanics and modeling in mechanobiology, 14(3), 489-513. 


Sensitivity Analysis in Systematic and Representative Benchmarking of Optimization Algorithm Performance

International audienceBesides the conventional and established applications of sensitivity analysis in the field of optimization, global sensitivity analysis-based approaches can also be used for systematic analysis of optimization problems and algorithm performance

Hal-Diderot

HAL Descartes

https://hal.archives-ouvertes.fr/hal-02308998/file/SAMO_Abstract2019_Final_ModOnline.pdf

Sensitivity Analysis in Systematic and Representative Benchmarking of Optimization Algorithm Performance

Abstract

Similar works

Full text

Available Versions

Archive Ouverte en Sciences de l'Information et de la Communication

Hal-Diderot

HAL Descartes