In a commentary paper, Bamber et al. (Nat Clim Change 3:424–427, 2016) respond to our recent assessment (De Vries and Van de Wal Clim Change 1–14, 2015) of their expert judgment based study on projections of future sea level rise due to the melting of the large ice sheets (Bamber and Aspinall Nat Clim Chang 3:424–427, 2013). In this response we comment on their remarks

de Vries, Hylke

van de Wal, Roderik S W

Utrecht University Repository

Climatic Change (2016) 137:329–332DOI 10.1007/s10584-016-1712-3RESPONSEResponse to commentary by J. L. Bamber, W. P. Aspinalland R. M. Cooke (2016)Hylke de Vries1 ·Roderik S. W. van de Wal2Received: 13 May 2016 / Accepted: 23 May 2016 / Published online: 7 June 2016© Springer Science+Business Media Dordrecht 2016Abstract In a commentary paper, Bamber et al. (Nat Clim Change 3:424–427, 2016)respond to our recent assessment (De Vries and Van de Wal Clim Change 1–14, 2015) oftheir expert judgment based study on projections of future sea level rise due to the meltingof the large ice sheets (Bamber and Aspinall Nat Clim Change 3:424–427, 2013). In thisresponse we comment on their remarks.We thank Bamber et al. (2016) [BAC16] for discussing our assessment (De Vries andVan de Wal 2015) [VW15] of their study (Bamber and Aspinall 2013) [BA13]. BAC16present additional information in support of the approach taken in BA13, but also criticiseour re-interpretation of the expert judgment data. Here we reflect on this criticism. We startby restating two principal reasons for publishing VW15.1 Asking the right questionIt all starts with asking the right question. A central goal of BA13 is to derive an expert-based distribution for the cumulative sea level rise arising from the ice-sheets over the 21stcentury. In VW15 we argue that this distribution can not be inferred straightforwardly fromtheir raw expert data, the main reason being that the experts were not asked that question.Instead, they were questioned about the individual sea-level trend contributions in the year2100 from the largest three ice-sheets (Greenland, West- and East Antarctica). As a con-sequence of this mismatch between question and goal, BA13 had to invoke a number of“post-processing” steps to allow the construction of that distribution. We regret that thisThis reply refers to the comment available at: http://dx.doi.org/10.1007/s10584-016-1672-7. Hylke de Vrieshylke.de.vries@knmi.nl1 Royal Netherlands Meteorological Institute (KNMI), De Bilt, The Netherlands2 Institute for Marine and Atmospheric Research Utrecht, Utrecht, The Netherlands330 Climatic Change (2016) 137:329–332point has barely received attention in BA13 and neither in their commentary BAC16. Itleaves the reader of BA13 with the believe that what is shown in BA13 is a true reflectionof the opinion of “the experts”, whereas in practice it is not.In VW15 we show that the required post-processing steps, made by the authors and there-fore at best guided by the experts’ answers, have a strong imprint on the final result. It couldhave been an excellent validation question if BA13 had also questioned the experts abouttheir estimates of cumulative sea level rise in 2100. In fact, it would have been the ultimatetest of their own underlying distribution reconstruction assumptions as well as providinga basic internal consistency check of the answers of the experts. The expert evaluation byHorton et al. (2014) is for this reason more straightforward to interpret as for that question-naire the experts were asked to provide a number for the cumulative sea level. This point isalso discussed in the recent paper by Oppenheimer et al. (2016).2 There is no consensus among expertsA second key message in VW15 is that neither the general public is unanimous about sea-level rise in 2100 nor the community of climate scientists. In fact, not even those climatescientists who consider themselves sea level rise experts or are considered as such by peersor by otherwise defined metrics of their expertise-level. These experts have widely differingviews on the future sea level contributions from the large ice sheets, especially the tailsof the probability distribution. The origin of this lack of consensus is clear, as some ofthe fundamental ice-dynamical processes (e.g., calving, hydrofracturing, ice cliff stability,marine ice-sheet instability) are not fully understood. The wide range of expert viewpointsis probably the most notable characteristic of the raw data presented in the supplementarymaterial of BA13. Yet they are not discussed in full depth in BA13.In VW15 we argue that this lack of consensus needs to be addressed in the interpretationof post-processing of the expert answers. How can one aggregate widely differing expertviewpoints in meaningful numbers? BA13 used the Classical Model approach for that andBAC16 provide ample references to show it is a robust methodology, which we do not aimto question. BA13 and also BAC16 seem however confident that once this “hurdle” of howto weight the different expert opinions has been taken, one can thereafter safely speak aboutwhat “The experts think about sea level change in 2100”. Given the huge spread in the rawdata we like to bring in some nuance.3 DiscussionIn the previous sections we explained two motivations for publishing VW15. Here we givefurther remarks and reflect on the criticism presented in BAC16.The “effective” number of experts Upon closer inspection of BA13 the following inter-esting statement can be found (p10 of their supplementary data): “And, equally clearly,self-assessed Expertise Level scores on these seed items are completely uncorrelated withthe corresponding performance-based weights determined empirically by the Classicalmodel”. Thus, although experts may well be very convinced they are a true expert in theirfield, their confidence does not relate to how well they score on the seed questions.Climatic Change (2016) 137:329–332 331If we study the normalised Cooke weights in the Supplementary Note 2 of BA13, onecan see that from the 13 experts, only 5 get a non-negligible weight factor (some even getweight 0.0). Therefore the initial pool of 13 experts decreases drastically after the objectivevalidation of the experts’ appropriateness to answer the question of sea level rise. BA13 donot discuss this strong reduction and apparently have no problem with the small effectivesize of the pool of experts. It is to be questioned where one puts the lower limit for therepresentativeness of “The Expert”. Obviously, had only one expert ‘survived’ the validation(in the sense of his/her weighting factor being much larger than all the others), the paperwould essentially represent the view of this one expert. We believe 5 is a small number andsome discussion on this is warranted.Reproducing pooled averages Unfortunately we were not able to reconstruct the “Perfwts” averages using the raw data and the normalised weight factors given in the Supple-mentary data of BA13. Either the data has been shuffled across different tables or the Cookeweighting factors differ substantially between 2010 and 2012 such that reconstruction is notpossible. Note that the latter would imply that experts with high weights (“true” experts)in the 2010 study can suddenly become “non-experts” in the follow up questionnaire in2012, or vice versa. Reproducing the “equal wts” results should be straightforward as itis a simple arithmetic mean of the contributing experts. Unfortunately, again we did notmanage to reconstruct these numbers from the raw data listed in the tables (we tested thelow, central and high estimates of EAIS, WAIS and GRIS). Maybe we have overlookedsomething, but our conclusion was that either the pooled averages are ok but the raw datais not listed correctly in the published tables, or vice versa. In VW15 we considered bothpossibilities.Comments to specific remarks of BAC16 BAC16 come up with a list of criticism toVW15. We are thankful for locations where this critique is constructive such as for examplein Section 1 in BAC16. VW15 has never had the purpose to come up with “the” (ulti-mate) solution. At first we aimed to just crosscheck the BA13 results, and asked the authorsfurther information regarding details by direct personal communication. When this failedwe proceeded with our attempts for a reconstruction and detected room for an alternativeapproach where at least the widely varying opinions of the experts are taken into account asis reflected in VW15.In their Section 2 (”Consensus distribution”) BAC16 provide a theoretical proof of whyour methodology will result in quantiles that are essentially averages of the individual expertquantile estimates. Of course; if one follows our line of medians, then by definition 50 %of the experts will have a more conservative estimate, 50 % a more extreme view. We admitthat our terminology of a “consensus distribution” is perhaps not the appropriate naming.It is a means of showing as directly as possible (and thereby necessarily as 2-dimensionalplots) the impact of the experts differing opinions. We are not convinced that the long tableslisted in BAC16 add anything valuable in this respect. In their Section 3 BAC16 argue thatwe have not explained our definition of “level of consensus” in VW15. What we had inmind is just to show the range of expert opinions at a given percentile estimate, but againwe regret any misunderstanding of our terminology.We appreciate the additional information presented in BAC16 Section 4 (”Lognormalfitting”) regarding details of the log-normal distribution, especially about the way the lowerand upper bounds are determined by the default intrinsic range. However, it is still unclear332 Climatic Change (2016) 137:329–332to us how their software package deals with the negative values in conjunction with a log-normal distribution as the latter is principally defined for positive values only (the log ofthe values is normally distributed). This plays an important role in aggregating the answersfor the different ice sheets to a cumulative contribution, which is at the heart of our crit-ical remarks. We therefore completely support the remark in BAC16 that the choice ofunderlying distribution warrants further study.4 ConclusionIn their conclusions BAC16 return to their primary argument that using pooled averagesbased on “validated” experts opinions is superior to using those based on “unvalidated”experts. They conclude with “the goal of science-based uncertainty quantification is notserved by neglecting validation”. We cannot agree more. VW15 has been intended to opendiscussion on the inconclusiveness of the experts’ opinions, and on the subjectivity intro-duced by the post-processing of their answers. We believe that BA13 has merit as a studyand opened a new but narrow avenue in glaciology, but that it also highlights only one sideof the coin. The side that remains mostly hidden contains a discussion on the large dis-crepancies between the experts, the small effective size of the sample after validation and,finally, the influences of the post-questionnaire decisions needed to come to the cumulativesea level distributions.ReferencesBamber JL, Aspinall WP (2013) An expert judgment assessment of future sea level rise from the ice sheets.Nat Clim Change 3:424–427. doi:10.1038/NCLIMATE1778Bamber JL, Aspinall WL, Cooke RM (2016) A commentary on “how to interpret expert judgmentassessments of 21st century sea-level rise. Climatic ChangeDe Vries H, Van de Wal RSW (2015) How to interpret expert judgement assessments of 21st century sealevel rise. Clim Change:1–14. doi:10.1007/s10584-015-1346-xHorton BJ, Rahmstorf S, Engelhart SE, Kemp AC (2014) Expert assessment of sea-level rise by AD 2100and AD 2300. Quat Sci Rev 84:1–6Oppenheimer M, Little CM, Cooke RM (2016) Expert judgement and uncertainty quantification for climatechange. Nat Clim Change 6(5):445–451

Response to commentary by J. L. Bamber, W. P. Aspinall and R. M. Cooke (2016)

https://dspace.library.uu.nl/bitstream/1874/432761/1/s10584-016-1712-3.pdf

Response to commentary by J. L. Bamber, W. P. Aspinall and R. M. Cooke (2016)

Abstract

Similar works

Full text

Available Versions

Utrecht University Repository