A sensitivity analysis of the PAWN sensitivity index

The PAWN index is gaining traction among the modelling community as a sensitivity measure. However, the robustness to its design parameters has not yet been scrutinized: the size (N) and sampling (ε) of the model output, the number of conditioning intervals (n) or the summary statistic (θ). Here we fill this gap by running a sensitivity analysis of a PAWN-based sensitivity analysis. We compare the results with the design uncertainties of the Sobol’ total-order index (S*Ti). Unlike in S*Ti, the design uncertainties in PAWN create non-negligible chances of producing biased results when ranking or screening inputs. The dependence of PAWN upon (N, n, ε, θ) is difficult to tame, as these parameters interact with one another. Even in an ideal setting in which the optimum choice for (N, n, ε, θ) is known in advance, PAWN might not allow to distinguish an influential, non-additive model input from a truly non-influential model input

Quantitative storytelling in the making of a composite indicator

The reasons for and against composite indicators are briefly reviewed, as well as the available theories for their construction. After noting the strong normative dimension of these measures—which ultimately aim to ‘tell a story’, e.g. to promote the social discovery of a particular phenomenon, we inquire whether a less partisan use of a composite indicator can be proposed by allowing more latitude in the framing of its construction. We thus explore whether a composite indicator can be built to tell ‘more than one story’ and test this in practical contexts. These include measures used in convergence analysis in the field of cohesion policies and a recent case involving the World Bank’s Doing Business Index. Our experiments are built to imagine different constituencies and stakeholders who agree on the use of evidence and of statistical information while differing on the interpretation of what is relevant and vital

Silver as a constraint for a large-scale development of solar photovoltaics? Scenario-making to the year 2050 supported by expert engagement and global sensitivity analysis

In this study we assess whether availability of silver could constrain a large-scale deployment of solar photovoltaics (PV). While silver-paste use in photovoltaics cell metallization is becoming more efficient, solar photovoltaics power capacity installation is growing at an exponential pace. Along photovoltaics, silver is also employed in an array of industrial and non-industrial applications. The trends of these uses are examined up to the year 2050. The technical coefficients for the expansion in photovoltaics power capacity and contraction in silver paste use have been assessed through an expert-consultation process. The trend of use in the non-PV sectors has been estimated through an ARIMA (auto-regressive integrated moving average) model. The yearly and cumulative silver demand are evaluated against the technological potential for increasing silver mining and the estimates of its global natural availability, respectively. The model implemented is tested with a quasi-random Monte Carlo variance-based global sensitivity analysis. The result of our inquiry is that silver may not represent a constraint for a very-large-scale deployment of photovoltaics (up to tens TW in installed power capacity) provided the present decreasing trend in the use of silver paste in the photovoltaics sector continues at an adequate pace. Silver use in non-photovoltaic sectors plays also a tangible influence on potential constraints. In terms of natural constraints, most of the uncertainty is dependent on the actual estimates of silver natural budget

Is VARS more intuitive and efficient than Sobol' indices?

The Variogram Analysis of Response Surfaces (VARS) has been proposed by Razavi and Gupta as a new comprehensive framework in sensitivity analysis. According to these authors, VARS provides a more intuitive notion of sensitivity and it is much more computationally efficient than Sobol' indices. Here we review these arguments and critically compare the performance of VARS-TO, for total-order index, against the total-order Jansen estimator. We argue that, unlike classic variance-based methods, VARS lacks a clear definition of what an "important" factor is, and show that the alleged computational superiority of VARS does not withstand scrutiny. We conclude that while VARS enriches the spectrum of existing methods for sensitivity analysis, especially for a diagnostic use of mathematical models, it complements rather than substitutes classic estimators used in variance-based sensitivity analysis.Comment: Currently under review in Environmental Modelling & Softwar

Are the results of the groundwater model robust?

De Graaf et al. (2019) suggest that groundwater pumping will bring 42--79\% of worldwide watersheds close to environmental exhaustion by 2050. We are skeptical of these figures due to several non-unique assumptions behind the calculation of irrigation water demands and the perfunctory exploration of the model's uncertainty space. Their sensitivity analysis reveals a widespread lack of elementary concepts of design of experiments among modellers, and can not be taken as a proof that their conclusions are robust.Comment: Comment on the paper by De Graaf et al. 2019. Environmental flow limits to global groundwater pumping. Nature 574 (7776), 90-9

A comprehensive comparison of total-order estimators for global sensitivity analysis

Sensitivity analysis helps identify which model inputs convey the most uncertainty to the model output. One of the most authoritative measures in global sensitivity analysis is the Sobol' total-order index, which can be computed with several different estimators. Although previous comparisons exist, it is hard to know which estimator performs best since the results are contingent on the benchmark setting defined by the analyst (the sampling method, the distribution of the model inputs, the number of model runs, the test function or model and its dimensionality, the weight of higher order effects, or the performance measure selected). Here we compare several total-order estimators in an eight-dimension hypercube, where these benchmark parameters are treated as random parameters. This arrangement significantly relaxes the dependency of the results on the benchmark design. We observe that the most accurate estimators are from Razavi and Gupta, Jansen, or Janon/Monod for factor prioritization, and from Jansen, Janon/Monod, or Azzini and Rosatifor approaching the "true" total-order indices. The rest lag considerably behind. Our work helps analysts navigate myriad total-order formulae by reducing the uncertainty in the selection of the most appropriate estimator