Al Waer, H.

Barnes, J.

Berne, S.

Chouaib, R.

Harris, N.

Lord, J.

MacPherson, I.

Murray, K.

Webb, B.

Online Research @ Cardiff

[Type here]    RTPI Research Paper    MEASURING WHAT MATTERS  Planning Outcomes Research Handbook   November 2020   The Royal Town Planning Institute (RTPI) RTPI champions the power of planning in creating prosperous places and vibrant communities. We have over 25,000 members in the private, public, academic and voluntary sectors.  Using our expertise and research we bring evidence and thought leadership to shape planning policies and thinking, putting the profession at the heart of society's big debates. We set the standards of planning education and professional behaviour that give our members, wherever they work in the world, a unique ability to meet complex economic, social and environmental challenges. We are the only body in the United Kingdom that confers Chartered status to planners, the highest professional qualification sought after by employers in both private and public sectors. Report authors Kevin Murray Associates, University of Dundee, McCabe Durney Barnes, Cardiff University, Yellow Book About this paper This Handbook is part of a suite of documents forming the Measuring What Matters Planning Outcomes Research. This Handbook offers practical guidance alongside the research toolkits that provide the practical framework for implementing this new process of measurement. Other documents forming part of this research are:   Measuring What Matters: Planning Outcomes Research Report and Toolkit (this toolkit can be adapted and applied to any nation)  Scottish Report and Toolkit   Irish Report and Toolkit  Wales Report and Toolkit  Front and back cover image Photo credit: Alistair Battles Measuring What Matters: Planning Outcomes Handbook 3 Handbook contents  1. The handbook introduction ................................................................... 4 2. Measuring What Matters: the process .................................................. 5 3. The toolkit framework ............................................................................ 6 4. The toolkit steps..................................................................................... 8 5. Scoring ................................................................................................. 10 6. Visualisation options ........................................................................... 11 7. Graphic equaliser variant .................................................................... 13 8. Review and feedback loop ................................................................... 14       Measuring What Matters: Planning Outcomes Handbook 4 1. The handbook introduction This document is a supplement to Measuring What Matters: Planning Outcomes Research Report and sets out the process for use of the toolkit. The toolkit is a way of bringing all the necessary information and data into a single location, to be ‘diagnosed’ and reviewed as part of the process in shifting output monitoring towards tracking impact. Four versions of the toolkit exist - for Scotland, Ireland, Wales and a common toolkit that can be applied in other planning jurisdictions. Use this handbook, alongside any of the toolkits, as a guide through the process and to help make the most of your use of the toolkit.    Measuring What Matters: Planning Outcomes Handbook 5 2. Measuring What Matters: the process The research highlighted that there is a need to develop a toolkit that enables a better understanding of the impact of planning. The toolkit is built around addressing the question – what difference is planning making? To understand the impact of planning, the immediate activities and direct outputs of planning need to be understood, alongside the wider outcomes from the planning process.  The three levels below provide a guide, showing a progression from understanding policy effectiveness in application determinations (Level 1),  to direct physical development outcomes (Level 2), to wider cumulative direct and indirect impacts of planning (Level 3).     Level 3: Evaluating impact Measuring wider impacts – understanding the difference planning makes, directly and indirectly, to what matters      Level 2: Development outcomes Monitoring direct change to environment through development or protection – and whether in line with policy/objectives      Level 1: Inputs, Activities, Outputs Monitoring how policy and objectives influence planning decisions that follow      Measuring What Matters: Planning Outcomes Handbook 6 3. The toolkit framework The Toolkit is a flexible tracking framework that can operate at multiple spatial scales. It has been tested in a range pilots, ranging from regional plans to urban development frameworks. This framework is used to pull together all the information needed to undertake the assessment for any level. By drawing plan information together with monitoring information, it acts as a diagnostic tool, moving beyond simply monitoring to aiding an assessment of progress towards objectives/targets. A full-size version of the framework below, with examples, is provided in the Measuring What Matters: Planning Outcomes Research Toolkits.  Theme Ref No. Policy/Objective Objectives/Targets Monitoring Indicators Data Achievement Score Place, Design & People       Health & Wellbeing       Environment, Conservation & Improvement       Climate Change       Homes & Community       Economy & Town Centres       Movement       Process & Engagement         Measuring What Matters: Planning Outcomes Handbook 7 Toolkit Input Columns (see columns on page 6)  Themes The eight themes reflect the sectoral areas over which spatial planning has an influencing a role and impact Ref No. A numbering system for users to refer and cross-refer with a data library. This is important for tracking Policy/Objective Identify theme-related policies or objectives to be monitored. This material may be drawn from approved statutory plans and their constituent policies or from other strategies/masterplans at a variety of scales (e.g. city, regional) Objectives/Targets Clearly identify the purpose of the plan or policy, including any specific objectives and/or targets Monitoring Indicators From available monitoring material, identify appropriate performance measurement indicators that are relevant to the policy/objective being tracked Data Insert data associated with the indicator, beginning with a ‘baseline’ position. Note likely tracking intervals (e.g. 1, 2, 5 years)                                                     Achievement Score Assessment of the data against objective/target. An aggregate score for the theme can be presented using diagrams. More detail is provided on Scoring at page 9 Note: The data source tab in the toolkit can be used to help record available data sources   Measuring What Matters: Planning Outcomes Handbook 8 4. The toolkit steps Central to the process is the working through a series of steps over time to measure progress. The five steps below form a full evaluation cycle: Step 1 is preparation, Steps 2 to 4 use the toolkit (from collecting data to diagnostic review and assessing impact) while Step 5 guides the future actions arising from evaluation. These steps can be applied at each of the three levels (shown on page 5). Step 1 Scoping exercise  Identify clearly what plan and policy is to be tracked and its performance monitored, including reasons  Define the spatial scope and timeframe of assessment and note what stage the plan/policy is at in its current cycle  Explain the purpose and stage of assessment (e.g. mid-term review of policy, thematic review, baseline, 5-year review)  Identify any current monitoring processes and evaluation feedback to benchmark policy effectiveness and data  Reflect on how toolkit will be used to identify activities and select indicators to measure outputs and outcomes  State any working assumptions underpinning the use of the toolkit  Establish project team and agreed procedures (e.g. for sharing and tracking data across organisations/teams)   Step 2 Collation of data  Confirm selection of plan and policies to be tracked - these may be drawn from a mix of site/locational/area specific, to strategic/sectoral and indirect impact (e.g. health and wellbeing)  Assign selected policies to one of the eight framework themes  Identify tracking indicators from available monitoring material  Publicly available data connected to impact, either directly or indirectly attributable to planning, is required  Draw together all data sources relating to monitoring of plans/ policies and their objectives/targets      Measuring What Matters: Planning Outcomes Handbook 9 Step 3 Diagnostic review  Analyse trends in measurable data to gain picture of performance over timeframe  Support measurement with qualitative assessment - include via group review - to reflect on context of application and limitations  Identify and understand effectiveness of the policy in operational decision-making - e.g. applications and appeals  Identify any gaps in data and monitoring knowledge. This can influence future data and performance enhancement   Step 4 Assessing impact  Evaluation and scoring of performance to be undertaken as either internal, external (or combined) team assessment  Assign a score of between 1 and 5 reflect performance against target, policy effectiveness, achievement of objective  Aggregate scoring allows for visualisation and composite presentation of findings for multiple policy strands in same theme. (See pages 9 - 11 for advice on the visualisation of scores)  Consideration may be given to weighting of different policies depending upon their strategic importance before producing finalised aggregate score   Step 5 Review, resolve, implement  Review the strengths and weaknesses in performance of plan/strategy/policy in the round  Consider how policies align with higher level strategy and objectives, such as national or regional  Review of effectiveness and outcomes of the policy, and if the policy is working  Review and amend strategy/policy/plan objectives, if required from analysis  Update indicators and monitoring process including additional data gathering requirements  Identify and share all lessons learnt for the assessment process  Implement the lessons learnt the next cycle of performance measurement in the chosen timeframe   Measuring What Matters: Planning Outcomes Handbook 10 5. Scoring A method of comparative scoring is needed to give an indication of achievement against objectives that can be read across policies, plans and the eight themes.  To provide such comparability, the scoring is based on a five-level ranking, each with performance benchmarks and proposed action.  To be rigorous and fair in measuring impact, the scoring needs to be an assessment of data against targets. This ensures that the measurement is a real gauge of progress over time, not an arbitrary ranking. To influence future performance the achievement score needs to be associated with some action in the next steps of the process.  Score Performance benchmark Action 5 Excellent progress towards outcomes Share learning with others 4 Good progress towards outcomes Consider increasing ambition in policy objectives/targets 3 Fair progress towards outcomes Review and aim to improve implementation and raise ambition in policy 2 Poor progress towards outcomes Review policy and understand implementation 1 Not achieving progress towards outcomes Immediate review and revisit of policy and its delivery  The first cycle in the application of the toolkit will establish baseline scores. It may be that, in the course of this initial cycle, accurate evaluation is not possible due to a lack of clear goals or reliable indicator data.  Because the process relates to sharing lessons and guiding future action, this should aid not only better performance, but also better policy and future measurement too.  Measuring What Matters: Planning Outcomes Handbook 11 6. Visualisation options   Visualisation of Scores Communication of the impact of planning by scoring is a key visual product of the toolkit. We have created two options for illustrating this. By aggregating the scoring within in each of the eight themes, and visually representing on a radar diagram (opposite) or graphic equaliser matrix (below on page 12) we can provide quick ‘easyread’ feedback on the impact planning  for a specified plan, area, policy.  As both forms show, performance can be linked to the UN SDGs and also to a range of national objectives or outcomes. The circular ‘radar’ diagram scores performance from the inside (low, under-achievement) outwards to the outside (excellent). It is useful for communicating online, in presentations or other instances where early visual impact is important. The linear ‘graphic equaliser’ diagram can be more nuanced for formal reporting and other uses where space and visual impact are not at a premium. Measuring What Matters: Planning Outcomes Handbook 12   This shows an example of a radar diagram with aggregate scores from each theme inserted.  The themes with no scores showing reflect that either (1) no specific policies were identified under these themes for measurement, or (2) no data was able to be identified. The expectation is that, over time, all theme categories will be able to be address at a local authority area plan level by at least some trackable policies, ideally several. However, with some specialist plans or policies, such as economic, housing or health, they may not trigger policy objectives or responses across all eight themes. Measuring What Matters: Planning Outcomes Handbook 13 7. Graphic equaliser variant This version below shows the matrix/graphic equaliser method of representing the performance within each theme. The right choice will depend upon the occasion and audience. Typical audiences for communicating the analysis might include internal team members, elected members, funding agencies and the wider public, such as civic groups.    Measuring What Matters: Planning Outcomes Handbook 14 8. Review and feedback loop The toolkit is intended to be used in more than one full evaluation cycle, embracing monitor, review and application of the learning. Only by cyclical repetition of the process with a critical review at Stage 5, can improvement be tracked and delivered.  The process is set up to ensure that the basic building blocks are in place to assess impacts, address issues and improve performance.  The key elements of the Stage 5 review stage are: Review the assessment  Are inputs being monitored?  Are activities and outputs being monitored?   Are development outcomes being monitored?  Are place impacts (quality and value) being monitored?  Does the scoring indicate strong alignment with local/regional/national plans and National outcomes?  Are Key Performance Indicators, SMART targets or other markers being established where needed?      Figure 1 Handbook: Aligning and Setting Targets Measuring What Matters: Planning Outcomes Handbook 15 Resolve issues arising  Inputs1 – what data needs to be captured to monitor effectively?  Activities and outputs – what data needs to be captured to monitor effectively?  Outcomes – what data needs to be captured to monitor effectively?  Impacts – what data needs to be captured to monitor effectively?  What policy areas need to be developed to create stronger alignment with local/regional/national plan and national outcomes? Implement actions  Develop policy to align with local/regional/national plans and national outcomes  Develop objectives and targets that can be measured  Identify existing data to begin monitoring  Identify future data requirements This completes a feedback loop of any one cycle and feeds into the next cycle of monitoring. Evidence for how to take forward to higher levels of performance, including national and international, are provided in Appendix B of the research report. Congratulations on completing your first cycle on the process towards measuring the direct and indirect impacts of planning.                                                    1 Inputs refer to the human, financial (including fees from planning income) and other resources deployed by planning policymakers and Planning Authorities. The other resources may include legal powers, statutory plans, planning policies and published guidance. The inputs are deployed to deliver planning activities. Royal Town Planning Institute,  41 Botolph Lane, London EC3R 8DL. Registered Charity in England (262865) & Scotland (SC037841)                                    For more information about this paper, visit:  www.rtpi.org.uk RTPI - Royal Town Planning Institute policy@rtpi.org.uk OR research@rtpi.org.uk Tel: 020 7929 9494 Report contact Rhian Brimble   

RTPI measuring what matters: planning outcomes research handbook

https://orca.cardiff.ac.uk/id/eprint/138225/1/N%20Harris%202020%20RTPI%20measuring%20what%20matters%20handbook.pdf

RTPI measuring what matters: planning outcomes research handbook

Abstract

Similar works

Full text

Available Versions

Online Research @ Cardiff