Brown, James D.

Mysiak, Jaroslav

Norton, John P.

English

SFU Library Digital Publishing (Simon Fraser University, Vancouver)

IAJ The Integrated Assessment Journal            B r i d g i n g  S c i e n c e s  &  P o l i c yVol. 6, Iss. 1 (2006), Pp. 83–88To what extent, and how, might uncertainty bedefined?Comments engendered by “Defining uncertainty:a conceptual basis for uncertainty management inmodel-based decision support”: Walker et al.,Integrated Assessment 4: 1, 2003John P. NortonDepartment of Electronic, Electrical and Computer EngineeringThe University of BirminghamEdgbaston, Birmingham B15 2TT, United KingdomandDepartment of Mathematics andIntegrated Catchment Assessment & Management CentreThe Australian National UniversityACT 0200, Australia ∗James D. BrownInstitute for Biodiversity and Ecosystem DynamicsUniversiteit van AmsterdamNiewe Achtergracht 166, 1018 WV Amsterdam, The Netherlands †Jaroslav MysiakUFZ Centre for Environmental ResearchLeipzig-Halle, Permoserstraße 15D-04318, Leipzig, Germany ‡Decisions about the exploitation and management of social and natural re-sources are frequently informed by predictions from models. In order to managethe contribution of models to decision making, it is important to understandthe uncertainties associated with these predictions. In practice, this is notstraightforward, for several reasons: models are structurally diverse, they are∗E-mail: j.p.norton@bham.ac.uk†E-mail:brown@science.uva.nl‡Corresponding Author; E-mail:jaroslav.mysiak@ufz.de, Phone: ++49 +341 2353199, Fax:++49 +341 2352825IAJNorton et al.: Defining uncertaintyused in a wide range of contexts and for many different reasons (Beck, 2002),and the quality of a model’s predictions may be highly case-dependent. Centralto coping with this complexity is the need for new concepts that link classes ofuncertainty to the types of situation in which models are used and the methodsavailable for evaluating uncertainty. Walker et al. (2003) propose a scheme forclassifying uncertainties in models intended for decision support. The purposeof this note is to examine 1) the extent to which the aims of Walker et al. areachievable and 2) whether the proposed classification meets these aims.Walker et al. aim to provide an interdisciplinary framework for assessing un-certainties in models used for decision support. We agree that such a frameworkis needed and should be developed from a classification of types of uncertaintyand situations in which they arise. Numerous taxonomies of imperfect knowl-edge have been proposed in recent years, including schemes that focus on generaltypes of imperfect knowledge (Suter et al., 1987; Smithson, 1989; Faber et al.,1992, 1996; Wynne, 1992; Dovers et al., 2001) and taxonomies that focus on spe-cific sources of imperfect knowledge (Wätzold, 2000). However, an importantbarrier to achieving a common understanding or interdisciplinary framework isthe diversity of meanings associated with terms such as ‘uncertainty’ and ‘igno-rance’, both within and between disciplines. A further impediment is the depthwith which different meanings are embedded in particular disciplines; commonunderstanding is not simply a matter of clarity. For example, some authorsequate uncertainty with ‘doubt’ or ‘unsureness’ (Brown, In Press), while othersinclude in their definitions sources of uncertainty, such as imprecision and ran-domness, or even view ignorance as an extreme form of scientific uncertainty(Walker et al., 2003; Smithson, 1989).Also, uncertainties may be framed by the presentation, sources, and socialconstruction of information (a social science perspective), as well as the degreesand perceived quality of information available (a Bayesian, physical science per-spective). Indeed, from a social science perspective, the apparently uncontro-versial assertion of Walker et al. that “completely deterministic knowledge ofthe relevant system” is “ideal” can be viewed as deeply problematic, becausethe pursuit of this “ideal” would discourage consideration of the disposition ofpeople to exaggerate, suppress or complicate expressions of uncertainty, inten-tionally or unintentionally. Thus, while classification is an important source ofmeaning, some discussion on the scope of uncertainty is a prerequisite to un-derstanding terms such as ‘location’, ‘level’ and ‘nature’ of uncertainty, as usedby Walker et al.. In the absence of a discussion about the different meaningsof uncertainty and the situations in which they arise, a classification cannot beexpected to provide a common understanding or interdisciplinary framework forassessing uncertainties in models used for decision support.Although a common framework is difficult to envisage in practice (and im-possible without very careful treatment of definitions), a systematic treatmentof uncertainty might nevertheless be possible. The Walker et al.. scheme specif-ically aims “to provide a systematic treatment of uncertainty in decision sup-port in order to improve the management of uncertainty in decision makingprocesses”. Given the applied nature of this aim, it follows that the scheme forIAJ, Vol. 6, Iss. 1 (2006), Pg. 84IAJclassifying uncertainty should (eventually) link to the application of models asdecision-support tools, and to the methods available for assessing their uncer-tainties. In this context, the scheme of Walker et al.. is limited in two importantways. First, by adopting a modellers’ view of uncertainty rather than a deci-sion makers’ perspective, the classification scheme omits some relevant sourcesof uncertainty (perhaps even the most important ones) that arise before andafter scientific models are applied. For example, it fails to consider the differentways in which goals (concepts) are translated into decision criteria (entities)and then into observable quantities (data and models), or how decision maker’spreferences based on model predictions are aggregated by decision-support toolssuch as cost-benefit analysis or multi-criteria analysis. Secondly, by not relatingclasses of uncertainty to the types of situation in which models are used or themethods available for estimating uncertainty, Walker et al. do not achieve theircentral aim of providing an operational scheme. If, having identified a specificclass of uncertainty, the classification is not accompanied by some means, how-ever crude, to follow up its implications for model-based decision support, it isinherently limited as an operational tool. Of course, it may lead to a deeperunderstanding of the fundamental components of uncertainty, but many classifi-cations have been proposed for this purpose (see above), and the current schemeis no more successful in this respect than the previous attempts, because 1) itdoes not address the diversity of meanings associated with terms such as ‘uncer-tainty’ and ‘ignorance’ in the context of model-based decision support, and 2) itdoes not explore how these concepts are assessed and used by different groups ofmodellers, including model developers and model users (e.g. ‘academics’ versus‘practitioners’).In the scheme proposed by Walker et al., uncertainty is classified accordingto its location, level and nature, each with a small number of classes. Thisresembles a taxonomy. Taxonomies usually have one or more orderings, of-ten hierarchical. Examples are the periodic table and the standard zoologicalclassification scheme. The attribute “location” has classes (context, model,inputs, parameters, outputs) and subclasses, and “nature” includes a binarysplit between epistemic uncertainty and ontological uncertainty (ignoring lin-guistic imprecision). While these do not impose any natural ordering, the at-tribute “level” introduces an order from ‘certainty’ through ‘statistical uncer-tainty’ and ‘scenario uncertainty’ to ‘recognised ignorance’ and ‘unrecognisedignorance’. This ordering is problematic if uncertainty is viewed as a stateof confidence, because certainty does not imply that a judgement is correct;rather, ignorance or ‘unrecognised ignorance’ in the Walker et al. scheme isfundamentally different from a state of confidence. Furthermore, if ‘statisticaluncertainty’ is assumed to refer to quantifiable uncertainty, Walker et al. fail toacknowledge the distinct spectrum of well-established methods, not all statisti-cal, for characterising degrees of credibility, ranging from bounds (binary clas-sification as possible/impossible) through rough sets (ternary classification aspossible/doubtful/impossible), fuzzy sets (graded from possible to impossible)and histograms (graded in relative frequency with a probabilistic interpretationavailable) to probability density functions (taking a Bayesian view). However,IAJ, Vol. 6, Iss. 1 (2006), Pg. 85IAJNorton et al.: Defining uncertaintythis one-dimensional spectrum does not cover all ways of describing uncertainty.For example, the characterisation of a group of uncertain items by their vectormean and covariance matrix has a non-probabilistic parallel in the specificationof ellipsoid bounds, characterised by the center and defining matrix. The latteris algebraically identical to the former, quite different in its interpretation andimplications, yet asymmetrically linked: the describing matrix of an ellipsoidalbound on a vector is an upper bound on its covariance, but a given covariancedoes not, of course, imply bounds on the vector. Similarly, pdfs are superfi-cially complete descriptors of uncertainty, well suited to assessing uncertaintypropagation through a model, whereas qualitative expressions of uncertainty aremore difficult to define unequivocally and to use in an uncertainty propagationanalysis, but are potentially more informative than pdfs, as they can contain anarbitrarily large number of types of attributes. As another example, informationabout the number of samples is typically lost in fitting a pdf to a probabilitytest sample, but may be highly relevant if the number is small.Although the scheme of Walker et al. is insufficiently precise about the at-tribute “level” and does not consider sufficiently the relationship between statesof knowledge and methods for describing uncertainty, it has the advantage ofbeing specific about the location of uncertainty in models. For example, itdistinguishes between model-structure uncertainty and input uncertainty, andwithin the latter between uncertainty in external forcing and in measured data.Uncertainties in calibrated parameters and model solution (numerical approx-imation) might be included here. However, in non-linear systems the effectsof inputs and parameters on outputs, or even stability, may be inextricablylinked, so that the contribution of input uncertainty to the overall uncertaintyin model prediction cannot be separated from those of uncertain model param-eters, structure and solution. Thus, an uncertainty-propagation analysis shouldbe performed for specific cases, and the roles of specific sources of uncertaintymay be highly case-dependent. Walker et al. also distinguish between con-trollable and uncontrollable inputs, but if the aim is to explore the effects ofuncertainty on model predictions it is necessary to consider controllability (ofthe state of a model, and thence of its outputs, from its inputs) as a prop-erty of the model and its inputs jointly, rather than its forcing inputs alone.Here the complementary properties of observability and identifiability shouldbe considered alongside controllability, and the possibility acknowledged thatover-parameterisation could accentuate uncertainties in model predictions. Ofcourse, amplification of uncertainty by ill-posedness may originate from a poorchoice of origin, scaling of variables or order of computation, as well as fromover-parameterisation.A further potentially important source of uncertainty in models originatesfrom the extrapolation of model predictions into regions of behaviour that arepoorly covered, or not covered at all, during model calibration. Thus, it is usefulto distinguish between the quality of a model in approximating some aspectsof ‘reality’ and the uncertainties associated with extrapolation of model pre-dictions. The former may depend on a combination of insufficiently detailedknowledge about the system or its inputs, and deliberate reduction of modelIAJ, Vol. 6, Iss. 1 (2006), Pg. 86IAJcomplexity for reasons of comprehensibility, computational load or analyticalconvenience. Approximation errors may be complex even in linear models. Forexample, aliasing or rounding errors, caused by an unsuitable sampling rate orquantisation interval for a variable in a model, may be entirely predictable fromthe characteristics of the variable being approximated, but may be very diffi-cult to describe in an uncertainty analysis. The uncertainties associated withextrapolation are essentially case-dependent, and their evaluation is impossi-ble without either extending experiments into a larger region of state space ormaking untested assumptions about regularity of behaviour over that space.Extended experimentation may be impossible in practice when resources arelimited or the state space is difficult to observe (e.g., in groundwater hydrol-ogy), or if the model has to predict in scenarios that are not currently realisable(e.g., in changed climate).Finally, in evaluating ‘structural uncertainty’ it is important to distinguishbetween the predictive performance of a model and its ability to explain realpatterns and processes. The former is fully reflected in the quantifiable differencebetween model predictions and independent observations, whereas the latterdepends also on collateral knowledge (e.g., recognising the possible contributionof snowmelt in calibrating a hydrological model relating rainfall to runoff). Inthis context, observations are not inherently more certain than models (althoughsuch an assumption is often implicit, especially for measured inputs) and theiruncertainties may be poorly reflected in stationary statistics.Underlying the classification of uncertainty in general, and undermining theclassification of Walker et al. specifically, is the possibility that interactionsbetween different sources of uncertainty are obscured or overlooked. Crucially,therefore, classifications of uncertainty should not sustain the generally unwar-ranted assumption, often made unconsciously, that the effects of different uncer-tainties are additive. Rather, once these uncertainties are identified (classified)they must be analysed and their implications for decision-making evaluated asa whole. For the same reason, an uncertainty analysis is always required in spe-cific cases, and the quality of a classification should be judged by its ability toguide the application of an uncertainty analysis in specific cases. Since Walkeret al. do not relate classes of uncertainty to methods of analysis or propagation,they cannot provide an operational scheme for addressing uncertainties and donot fulfil their central aim of providing “a systematic treatment of uncertainty indecision support in order to improve the management of uncertainty in decisionmaking processes”.To summarise, the scheme proposed by Walker et al. is valuable in stim-ulating further thought about the role of uncertainty in model-aided decision-making, but incomplete in its discussion of concepts (e.g., the meaning of un-certainty) and hence difficult to apply within an interdisciplinary framework,and insufficiently grounded in methodology and thus difficult to apply in anoperational content. Furthermore, it overlooks interactions between uncertain-ties that can severely limit a simple classification from both a modellers’ and adecision makers’ perspective.IAJ, Vol. 6, Iss. 1 (2006), Pg. 87IAJNorton et al.: Defining uncertainty1 BibliographyBeck, M. B. (2002), Environmental Foresight and Models: A Manifesto, Elsevier,Oxford. 84Brown, J. (In Press), ‘Knowledge, uncertainty and physical geography: Towardsthe development of methodologies for questioning belief’, Transactions of theInstitute of British Geographers. 84Dovers, S., Norton, T. & Handmer, J. (2001), Ignorance, uncertainty and ecol-ogy: Key themes, in J. Handmer, T. Norton & S. Dovers, eds, ‘Ecology,Uncertainty, and Policy: Managing Ecosystems for Sustainability’, PearsonEducation Limited, Harlow, pp. 1–25. 84Faber, M., Manstetten, R. & Proops, J. (1992), Towards an open future: Ig-norance, novelty and evolution, in R. Costanza, B. Norton & B. Haskell,eds, ‘Ecosystem Health: New Goals for Environmental Management’, IslandPress, Washington, D.C., pp. 72–96. 84Faber, M., Manstetten, R. & Proops, J. (1996), Ecological Economics, EdwardElgar, Cheltenham. 84Smithson, M. (1989), Ignorance and Uncertainty: Emerging Paradigms,Springer-Verlag, New York. 84Suter, G., Barnthouse, L. & O’Neill, R. (1987), ‘Treatment of risk in environ-mental impact assessment’, Environmental Management 11, 295–303. 84Walker, W., Harremoës, P., Rotmans, J., van der Sluijs, J., van Asselt, M.,Janssen, P. & Krayer von Krauss, M. (2003), ‘Defining uncertainty: A con-ceptual basis for uncertainty management in model-based decision support’,Integrated Assessment 4(1), 5–18. 84, 85, 86, 87Wätzold, F. (2000), ‘Efficiency and applicability of economic concepts dealingwith environmental risk and ignorance’, Ecological Economics 2(33), 299–311.84Wynne, B. (1992), ‘Uncertainty and environmental learning: Reconceiving sci-ence and policy in the preventative paradigm’, Global Environmental Change2, 111–127. 84IAJ, Vol. 6, Iss. 1 (2006), Pg. 88

To what extent, and how, might uncertainty be defined?

https://journals.lib.sfu.ca/index.php/iaj/article/download/2702/1988

To what extent, and how, might uncertainty be defined?

Abstract

Similar works

Full text

Available Versions

SFU Library Digital Publishing (Simon Fraser University, Vancouver)