Reliable error estimation for Sobol' indices

International audienceIn the field of sensitivity analysis, Sobol’ indices are sensitivity measures widely used to assess the importance of inputs of a model to its output. The estimation of these indices is often performed trough Monte Carlo or quasi-Monte Carlo methods. A notable method is the replication procedure that estimates first-order indices at a reduced cost in terms of number of model evaluations. An inherent practical problem of this estimation is how to quantify the number of model evaluations needed to ensure that estimates satisfy a desired error tolerance. This paper addresses this challenge by proposing a reliable error bound for first-order and total effect Sobol’ indices. Starting from the integral formula of the indices, the error bound is defined in terms of the discrete Walsh coefficients of the different integrands.We propose a sequential estimation procedure of Sobol’ indices using the error bound as a stopping criterion. The sequential procedure combines Sobol’ sequences with either Saltelli’s strategy to estimate both first-order and total effect indices, or the replication procedure to estimate only first-order indices

Challenges in Developing Great Quasi-Monte Carlo Software

Quasi-Monte Carlo (QMC) methods have developed over several decades. With the explosion in computational science, there is a need for great software that implements QMC algorithms. We summarize the QMC software that has been developed to date, propose some criteria for developing great QMC software, and suggest some steps toward achieving great software. We illustrate these criteria and steps with the Quasi-Monte Carlo Python library (QMCPy), an open-source community software framework, extensible by design with common programming interfaces to an increasing number of existing or emerging QMC libraries developed by the greater community of QMC researchers

On Bounding and Approximating Functions of Multiple Expectations using Quasi-Monte Carlo

Monte Carlo and Quasi-Monte Carlo methods present a convenient approach for approximating the expected value of a random variable. Algorithms exist to adaptively sample the random variable until a user defined absolute error tolerance is satisfied with high probability. This work describes an extension of such methods which supports adaptive sampling to satisfy general error criteria for functions of a common array of expectations. Although several functions involving multiple expectations are being evaluated, only one random sequence is required, albeit sometimes of larger dimension than the underlying randomness. These enhanced Monte Carlo and Quasi-Monte Carlo algorithms are implemented in the QMCPy Python package with support for economic and parallel function evaluation. We exemplify these capabilities on problems from machine learning and global sensitivity analysis

Computing Shapley Effects for Sensitivity Analysis

Shapley effects are attracting increasing attention as sensitivity measures. When the value function is the conditional variance, they account for the individual and higher order effects of a model input. They are also well defined under model input dependence. However, one of the issues associated with their use is computational cost. We present a new algorithm that offers major improvements for the computation of Shapley effects, reducing computational burden by several orders of magnitude (from $k!\cdot k$ to $2^k$, where $k$ is the number of inputs) with respect to currently available implementations. The algorithm works in the presence of input dependencies. The algorithm also makes it possible to estimate all generalized (Shapley-Owen) effects for interactions.Comment: 16 pages, 5 figures, 3 tables, 2 algorithm

The Future of Sensitivity Analysis: An essential discipline for systems modeling and policy support

Sensitivity analysis (SA) is en route to becoming an integral part of mathematical modeling. The tremendous potential benefits of SA are, however, yet to be fully realized, both for advancing mechanistic and data-driven modeling of human and natural systems, and in support of decision making. In this perspective paper, a multidisciplinary group of researchers and practitioners revisit the current status of SA, and outline research challenges in regard to both theoretical frameworks and their applications to solve real-world problems. Six areas are discussed that warrant further attention, including (1) structuring and standardizing SA as a discipline, (2) realizing the untapped potential of SA for systems modeling, (3) addressing the computational burden of SA, (4) progressing SA in the context of machine learning, (5) clarifying the relationship and role of SA to uncertainty quantification, and (6) evolving the use of SA in support of decision making. An outlook for the future of SA is provided that underlines how SA must underpin a wide variety of activities to better serve science and society.John Jakeman’s work was supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Scientific Discovery through Advanced Computing (SciDAC) program. Joseph Guillaume received funding from an Australian Research Council Discovery Early Career Award (project no. DE190100317). Arnald Puy worked on this paper on a Marie Sklodowska-Curie Global Fellowship, grant number 792178. Takuya Iwanaga is supported through an Australian Government Research Training Program (AGRTP) Scholarship and the ANU Hilda-John Endowment Fun

The future of sensitivity analysis: an essential discipline for systems modeling and policy support

Sensitivity analysis (SA) is en route to becoming an integral part of mathematical modeling. The tremendous potential benefits of SA are, however, yet to be fully realized, both for advancing mechanistic and data-driven modeling of human and natural systems, and in support of decision making. In this perspective paper, a multidisciplinary group of researchers and practitioners revisit the current status of SA, and outline research challenges in regard to both theoretical frameworks and their applications to solve real-world problems. Six areas are discussed that warrant further attention, including (1) structuring and standardizing SA as a discipline, (2) realizing the untapped potential of SA for systems modeling, (3) addressing the computational burden of SA, (4) progressing SA in the context of machine learning, (5) clarifying the relationship and role of SA to uncertainty quantification, and (6) evolving the use of SA in support of decision making. An outlook for the future of SA is provided that underlines how SA must underpin a wide variety of activities to better serve science and society

Sensitivity Analysis

Sensitivity analysis (SA), in particular global sensitivity analysis (GSA), is now regarded as a discipline coming of age, primarily for understanding and quantifying how model results and associated inferences depend on its parameters and assumptions. Indeed, GSA is seen as a key part of good modelling practice. However, inappropriate SA, such as insufficient convergence of sensitivity metrics, can lead to untrustworthy results and associated inferences. Good practice SA should also consider the robustness of results and inferences to choices in methods and assumptions relating to the procedure. Moreover, computationally expensive models are common in various fields including environmental domains, where model runtimes are long due to the nature of the model itself, and/or software platform and legacy issues. To extract using GSA the most accurate information from a computationally expensive model, there may be a need for increased computational efficiency. Primary considerations here are sampling methods that provide efficient but adequate coverage of parameter space and estimation algorithms for sensitivity indices that are computationally efficient. An essential aspect in the procedure is adopting methods that monitor and assess the convergence of sensitivity metrics. The thesis reviews the different categories of GSA methods, and then it lays out the various factors and choices therein that can impact the robustness of a GSA exercise. It argues that the overall level of assurance, or practical trustworthiness, of results obtained is engendered from consideration of robustness with respect to the individual choices made for each impact factor. Such consideration would minimally involve transparent justification of individual choices made in the GSA exercise but, wherever feasible, include assessment of the impacts on results of plausible alternative choices. Satisfactory convergence plays a key role in contributing to the level of assurance, and hence the ultimate effectiveness of the GSA can be enhanced if choices are made to achieve that convergence. The thesis examines several of these impact factors, primary ones being the GSA method/estimator, the sampling method, and the convergence monitoring method, the latter being essential for ensuring robustness. The motivation of the thesis is to gain a further understanding and quantitative appreciation of elements that shape and guide the results and computational efficiency of a GSA exercise. This is undertaken through comparative analysis of estimators of GSA sensitivity measures, sampling methods and error estimation of sensitivity metrics in various settings using well-established test functions. Although quasi-Monte Carlo Sobol' sampling can be a good choice computationally, it has error spike issues which are addressed here through a new Column Shift resampling method. We also explore an Active Subspace based GSA method, which is demonstrated to be more informative and computationally efficient than those based on the variance-based Sobol' method. Given that GSA can be computationally demanding, the thesis aims to explore ways that GSA can be more computationally efficient by: addressing how convergence can be monitored and assessed; analysing and improving sampling methods that provide a high convergence rate with low error in sensitivity measures; and analysing and comparing GSA methods, including their algorithm settings