How can indicator weights for multi-criteria sustainability assessments be determined based on experts' opinions? How do different opinions affect the results of sustainability assessments

Baumgart, Lukas

Blockeel, Johan

Heidenreich, Anja

Landert, Jan

Marton, Silvia

Moakes, Simon

Schader, Christian

Ssebunya, Brian

Organic Eprints

Uncertainty Assessment in Multi-Criteria  Sustainability AssessmentsThe Case of the SMART-Farm ToolDownload a PDF of this poster:http://orgprints.org/32058© 2017 FIBLChristian Schader, Anja Heidenreich, Jan Landert, Johan Blockeel, Lukas Baumgart, Brian Ssebunya, Simon Moakes, Silvia MartonResearch Institute of Organic Agriculture FiBL, SwitzerlandResearch QuestionsHow can indicator weights for multi-criteria sustainability assessments be determined based on experts' opinions? How do different opinions affect the results of sustainability assessments?SMART-Farm ToolA globally applicable and comparable method for analysing farms of different types• Based on SAFA Guidelines (FAO, 2014)• Set of 327 indicators to assess the degree of goal achievement of 58 SAFA sustainability subthemesOverall ScoreCorporateEthicsAccountabilityParticipationRule of LawHolisticManagementAtmosphereWaterLandBiodiversityMaterials &EnergyAnimalWelfareInvestmentVulnerabilityProduct Quality &InformationLocalEconomyDecentLivelihoodsFair TradingPracticesLabourRightsEquityHuman Health &SafetyCulturalDiversity0%20%40%60%80%100%Figure 3: Illustrative results of SMART-Farm Tool sustaina-bility assessments of SAFA Subthemes• Each indicator can impact one or more SAFA subthemes (in total more than 1700 linkages between SAFA subthemes and SMART-Farm Tool indicators)• The importance of an indicator in different SAFA subtheme might be stronger or weaker, thus for each subtheme different weights were defined to specify its impact• These weights were defined by >60 international experts in developed and developing countries in temperate and tropical environments; following a 3-step nominal group technique (NGT) approach (Figure 4)Figure 4: Overview of steps of the NGT0%20%40%60%0 0.2 0.4 0.6 0.8 1Proportion of experts giving a certain indicator weightIndicator weight on subtheme"Stability of Markets"Length of customer relationshipExperts for developedcountriesExperts for developingcountries0%50%100%0 0.2 0.4 0.6 0.8 1Proportion of experts giving a certain indicator weightIndicator weight on subtheme"Stability of Markets"Agroforestry systemsExperts for developedcountriesExperts for developingcountries0%50%100%0 0.2 0.4 0.6 0.8 1Proportion of experts giving a certain indicator weightIndicator weight on subtheme"Stability of Markets"Agroforestry systemsExperts for developedcountriesExperts for developingcountriesFactor 1: Indicator weightings• Step 1  (0.5-1.5h)•I ndividual expert ratings for eachindicatorAnalysis of results of step 1• Step 2 (0.5-1h)•A daptation ofratings for eachindicatorAnalysis of results of step 2 • Step 3  (1-2.5h)•O nline discussionconcerningcontroversial indicatorsSimultaneousadaptation of ratings based onexchangesNo consensusFigure 1: Examples of weights specified by experts for two (of 22) indicators that contribute to the SAFA subtheme: “Stability of Markets” Indicator weighting:0 - indicator is irrelevant1 - indicator is the critical  indicator for determining the degree of goal achieve-ment of SAFA subthemeConclusion• Consideration of uncertainty is important for interpreting results from  multi-criteria sustainability assessments for decision making• The approach used for this study can be adapted to other tools• More time and interaction needed to discuss single indicators• Formation of an expert panel planned to improve weightingsFactor 2: Indicator ratingsTable 1: SMART-Farm Tool indicator ratings within “Stability of Markets” SAFA subtheme for four example farms in developed and developing countries Result: Subtheme scoreFigure 2: Exemplary distribu-tion of subtheme scores which are calculated based on the farms’ indicator ratings (Table 1) and the differing indicator weights resulting from the NGT process (Figure 1) after Monte Carlo SimulationMonte Carlo SimulationDegree of goal achievement for  subtheme “Stability of Markets”Farm AFarm BFarm CFarm DFrequency of degree of  goal achievementFigure 3: Examples of weights specifi d by experts for two (of 22) indicator  that contribute to the SAFA subtheme: “Stability of Markets” Indicator weighting:0 - indicator is irrelevant1 - indicator is the critical indicator for determining the degree of goal achievement of SAFA subthemeFigure 2: Overview of steps of the NGTRESEARCH QUESTIONCONCLUSIONSMART-FarmTool• A globally applicable andcomparable method for analysingfarms of different types• Based on SAFA Guidelines (FAO,2014)• Set of 327 indicators to assess thedegree of goal achievement of 58SAFA sustainability subthemes• Each indicator can impact one ormore SAFA subthemes (in totalmore than 1’700 linkages betweenSAFA subthemes and SMART-FarmTool indicators)• The importance of an indicator indifferent SAFA subtheme might bestronger or weaker, thus for eachsubtheme different weights weredefined to specify its impact• These weights were defined by >60international experts in developedand developing countries intemperate and tropicalenvironments; following a 3-stepnominal group technique (NGT)approach (Figure 2)Figure 1: Illustrative results of SMART-Farm Tool sustainability assessments of SAFA SubthemesHow can indicator weights for multi-criteria sustainabilityassessments be determined based on experts' opinions?How do different opinions affect the results of sustainabilityassessments?Figure 4: Exemplary distribution of subtheme scores which are calculated based on the farms’ indicator ratings (Table 1) and the differing indicator weights resulting from the NGT process (Figure 3) after Monte Carlo Simulation• C nsideration of uncertainty is important for interpreting results from multi-criteria sustainability assessments for decision making• The approach used for this study can be adapted to other tools• More time and interaction needed to discuss single indicators• Formation of an expert panel planned to improve weig tingsTable 1: SMART-Far  Tool indicator ratings within “Stability of Markets” SAFA subtheme for four example farms in developed and developing countriesFarm A Farm B Farm C Farm DLength of customer relationship 25% 100% 0% 0%Agroforestry systems 10% 20% 100% 50%Rare or endangered agricultural crops 100% n/a* 100% 0%Access to advisory services 100% 100% 0% 0%… (in total 22 indicators in subtheme) x% x% x% x%* No cropping on farm B, indicator was excluded Uncertainty Assessment in Multi-Criteria Sustainability AssessmentsThe Case of the SMART-Farm ToolChristian Schader, Anja Heidenreich, Jan Landert, J han Blockeel, Lukas Baumgart, Brian Ssebunya, Simon Moakes, Silvia MartonMonte Carlo SimulationFactor 1:Indicator weightingsFactor 2: Indicator ratingsResult:Subtheme scoreFarm AFarm BFarm CFarm DDegree of goal achievement for subtheme “Stab lity of Markets”Frequency of degree of goal achievement• Step 1  (0.5-1.5h)• Individual expert ratings for each indicatorAnalysis of results of step 1• Step 2  (0.5-1h)• Adaptation of ratings for each indicatorAnalysis of results of step 2 • Step 3  (1-2.5h)• Online discussion concerning controversial indicatorsSimultaneous adaptation of ratings based on exchangesNo consensus

Uncertainty Assessment in Multi-Criteria Sustainability Assessments

Uncertainty Assessment in Multi-Criteria Sustainability Assessments

Abstract

Similar works

Full text

Available Versions

Organic Eprints