Dykes, J.

Lloyd, D.

Radburn, R.

English

City Research Online

Dykes, J., Lloyd, D. & Radburn, R. (2009). Using the Analytic Hierarchy Process to prioritise candidate improvements to a geovisualization application. Paper presented at the GIS Research UK 17th Annual Conference (GISRUK 2009), 1 - 3 Apr 2009, University of Durham, Durham, UK.City Research OnlineOriginal citation: Dykes, J., Lloyd, D. & Radburn, R. (2009). Using the Analytic Hierarchy Process to prioritise candidate improvements to a geovisualization application. Paper presented at the GIS Research UK 17th Annual Conference (GISRUK 2009), 1 - 3 Apr 2009, University of Durham, Durham, UK.Permanent City Research Online URL: http://openaccess.city.ac.uk/412/ Copyright & reuseCity  University  London has developed City  Research Online  so that  its  users  may access the research outputs of City University London's staff. Copyright © and Moral Rights for this paper are retained by the individual author(s) and/ or other copyright holders. Users may download and/ or print one  copy  of  any  article(s)  in  City  Research  Online  to  facilitate  their  private  study  or  for  non-commercial research. Users may not engage in further distribution of the material or use it for any profit-making activities or any commercial gain. All material in City Research Online is checked for eligibility for copyright before being made available in the live archive. URLs from City Research Online may be freely distributed and linked to from other web pages. Versions of researchThe version in City Research Online may differ from the final published version. Users are advised to check the Permanent City Research Online URL above for the status of the paper.EnquiriesIf you have any enquiries about any aspect of City Research Online, or if you wish to make contact  with the author(s) of this paper, please email the team at publications@city.ac.uk.Using the Analytic Hierarchy Process to prioritise candidate improvements to a geovisualization application  David Lloyd, Jason Dykes, Robert Radburn giCentre, City University London, EC1V 0HB  Telephone: +44 (0)20 7040 0212, Fax: +44 (0)20 7040 8584 Email: {d.lloyd | jad7 | sbbd476}@soi.city.ac.uk, Web: www.soi.city.ac.uk/~at775  KEYWORDS: geovisualization, analytic hierarchy process, crime and disorder reduction  1. Introduction   Crime and disorder reduction (CDR) research analysts („analysts‟) in a UK local authority have generated candidate improvements for enhancing geovisualization prototypes designed using human-centred methods (Lloyd et al., 2007; Lloyd et al., 2008). Prioritising these is an important process and may require modification to established decision support techniques due to the nature of geovisualization. We explore this issue through the Analytic Hierarchy Process (AHP) (Saaty 1977) examining both the resulting priorities and their consistency.   2. Approach  Three crime analysts suggested ~350 explicit and implicit improvements to prototypes in seven experiments that enable analysts to explore crime attributes (absolute and relative numbers) spatially (using choropleth shading) and temporally (using glyphs). When coded and grouped the ~120k transcribed words yield 35 possible improvements. A clear task is to prioritise these possible improvements, initially unconstrained, and then in the context of limited development resource in order to direct development. Approaches to the first of these include multi-criteria decision analysis (MCDA) (Dodgson et al., 2000), GIS-based MCDA (Malczewski 2006), and the well established (Wasil and Golden 2003) Analytic Hierarchy Process which has been widely used in prioritising software development (Karlsson and Ryan 1997). AHP participants prioritise from a list by relating every possible pair of combinations. An overall score and ranking are produced for each item, along with a consistency ratio for each user.   Our 35 possible improvements would need too many pairwise comparisons for completion in a reasonable time. We reduce this number by grouping (Karlsson et al.,1997) and use pairwise group comparison to subsequently relate the group results. Analysts consider improvement groups in turn: „data-related' (dealing with aggregation, filtering, context); „interface-related‟ (system behaviour, complexity, speed); „interaction-related‟ (readability, orientation, scale, legend) and „new‟ (novel visualization tools and displays). Two analysts score preferences on each pairwise comparison within each of the four groups and then for the four groups themselves using an integer divergent scale (Karlsson and Ryan 1997). Comments made during the test are noted, and analysts asked about the process retrospectively. The perspective of „geovisualization expert‟ („expert‟) was provided by Dykes who had participated in the human-centered development process and undertook the AHP.  3. Findings  3.1 Quantitative findings Marked similarities are noted in the rankings of the desirability of the 35 possible improvements prioritised by the two CDR analysts (Pearson coefficient 0.50, significant at 0.01 level; 2 tailed, n=35). This is not the case with expert‟s rankings, which are significantly different from both analysts‟. Figure 1 shows analysts‟ and expert‟s rankings as parallel plot small multiples, conditioned by improvement group. Analysts‟ priorities are skewed towards „data-related‟ improvements and against „new‟ items. The expert‟s priorities are more evenly distributed, and incline towards „interaction related‟ and against „interface related‟ choices.    Figure 1. Parallel plots showing the rank of candidate improvements from 1 (top) to 35 (bottom), for each group, The same plot is shown four times with each group highlighted in turn: „data‟ (10 lines highlighted corresponding to 10 'data' improvements), „interface‟ (6), „interactive‟ (7) and „new‟ (12). Rankings for CDR analysts (C1 and C2) and expert (D) are shown left to right within each plot.   Saaty (1980) considers an AHP 'consistency ratio' of < 0.1 acceptable; "in practice, however, consistency ratios exceeding 0.10 occur frequently" (Karlsson and Ryan, 1997). Those achieved here range from 0.03 to 0.21 for data-, interface- and interaction-related possible improvements, but the results from the „new-related‟ group are noticeably less consistent, ranging from 0.43 to 0.69. Analyst C1 is more consistent than the others throughout. C1's relative preferences across the 35 possible improvements are not as strong as those of C2 and D, as measured by the Gini coefficient (C1: 0.27; C2: 0.48; D: 0.42), calculated from Lorenz curves (Lorenz, 1905) of the same data.  Table 1. AHP consistency ratios for the four different groups and overall group comparison of the 35 possible improvements (low is more consistent).   User \ Group    „data‟    „interface‟  „interaction‟      „new‟   group comparison CDR Analyst „C1‟ 0.03 0.09 0.04 0.49 0.07 CDR Analyst „C2‟ 0.06 0.20 0.21 0.69 0.06 Geovis expert „D‟ 0.16 0.10 0.21 0.43 0.04  3.2 Qualitative findings CDR analysts spent considerable time before the AHP exercise clarifying details of the „new‟ candidate improvements. This resonates with our problems mediating geovisualization possibilities to these analysts (Lloyd et al., 2007) and parallels the difficulties experienced in identifying „undreamed of‟ requirements (Robertson 2001). Comments made during the task include concerns about individual‟s consistency; concerns at the descriptions provided not differentiating sufficiently for some comparisons; and unprompted explanations being given for scores.  The CDR analysts found the AHP to be efficient and meaningful - preferred candidate improvements were successfully identified. The „expert‟ experience was less positive - a tendency to focus on the process and one‟s own consistency rather than the detail of the improvements was noted through participation, as were difficulties in interpreting improvement descriptions consistently. Achieving consistency was an important aim for all users, and two of the three participants were concerned after awarding scores of „1‟ frequently in succession. One of the analysts summed up their understanding of the proposed tools as “a guess on the back of what you are telling me”, indicating that earlier difficulties reported in mediating geovisualization to analysts continue and may be exacerbated by the coding and grouping required for AHP.  4. Conclusions  The two analysts have very different dispersions and different consistency ratios, but their rankings are indistinguishable, supporting the notion that the AHP is robust. The priorities of the expert are markedly different despite the high levels of engagement between analyst and expert throughout the human-centred development process. Geovisualization applications are predominantly „expert‟ driven (Fuhrmann et al., 2005) and so the discrepancies in terms of priorities are an important finding that should be explored further with other analysts and ‟expert‟ developers.   Given the poor consistency in ranking 'new-related' improvements, such rankings clearly cannot be relied upon to indicate priorities within this group. But the fact that „new‟ candidate improvements are ranked inconsistently by all subjects suggests particular uncertainty about their nature and/or possible benefits. The issue may be one of communication and interpretation - unfamiliar improvements are more difficult to describe, communicate and interpret consistently with the coding, grouping and succinct descriptions required for pairwise comparison in the context of ~350 possibilities. Including the kinds of complex novel visual features typical of geovisualization as possible improvements may thus affect the working of the AHP. This is despite our efforts to expose the CDR analysts to geovisualization techniques and prototypes over an extended period and providing detailed descriptions prior to and during the AHP process. The time spent by the analysts at the outset and the qualitative data lend weight to this conclusion, confirming our earlier findings on difficulties in mediating geovisualization to these users (Lloyd et al., 2008). We also note the understandable focus of the analysts on prototype improvements that have the most bearing on their current activities rather than on innovation. This may be another limitation of the AHP, as we have previously observed these analysts being more open to innovation when not asked to prioritise - indeed all 350 candidate improvements were suggested by these users working with geovisualization prototypes in our human-centred design process (Lloyd et al., 2008).   Consequently, future application of AHP in geovisualization might variously:  involve all parties in the AHP concurrently so that concepts can be discussed and interpretations clarified - AHP as a collaborative process to mediate shared understanding of priorities  provide visual descriptions/stimuli with demos, videos or presentations prior to and during the process so that the candidate improvements are agreed  use fewer, more specific, candidate improvements - sampling rather than aggregation  run the AHP against different scenarios to establish (for example) current and future priorities  weight the results by analyst based on criteria such as consistency (from the consistency ratio) or dispersion (from the Gini coefficient)  A variant of the classic knapsack problem allowed us to determine how the AHP output can help prioritise possible improvements under the constraint of different value solutions and developer costs. Results reveal that the analysts focus just as strongly on known functionality when development resources are limited, even when current tasks provide opportunity for beneficial geovisualization (Lloyd et al., 2007).   Whilst showing how a decision support technique can be successfully employed, we suggest that the nature of geovisualization may cause difficulties for those seeking to differentiate between candidate improvements, and may not provide an unambiguous development roadmap. Approaches to developing prototypes rapidly in collaboration with prospective users through „patchwork prototyping‟ (Jones et al., 2007), or establishing requirements in ways that involve creativity (Maiden et al., 2004) may be beneficial in resolving the different perspectives identified here.  5. Acknowledgements  We gratefully acknowledge EPRSC and Leicestershire County Council support via CASE award EP/C547438/1, the help of the LCC CDR team, and the anonymous reviewers for their comments.  Biography  David Lloyd is a PhD candidate researching the use of human-centred techniques in geovisualization at the giCentre, City University London.  Jason Dykes is a Senior Lecturer in Geographic Information at the giCentre, City University London with interests in geovisualization techniques, tools, design, application and evaluation.  Robert Radburn is a Senior Research Officer at LCC and holds an ESRC UPTAP research fellowship at the giCentre, City University London to develop capacity for visual analysis in local government.   References  Dodgson, J., Spackman, M., Pearman, A. & Phillips, L.(2000), pp38-61 Multi-criteria analysis manual   Fuhrmann, S., Ahonen-Rainio, P., Edsall, R., Fabrikant, S., Koua, E., Tobon, C., Ware, C. & Wilson, S. (2005), Making Useful and Useable Geovisualization: Design and Evaluation Issues, in J. Dykes (ed.), Exploring Geovisualization, Chapter 28, p556, Elsevier, Oxford. Jones, M.C., Floyd, I.R. & Twidale, M.B. (2007), Patchwork Prototyping with Open-Source Software, in K. St.Amant & B. Still (eds), The Handbook of Research on Open Source Software: Technological, Economic, and Social Perspectives, pp133-8, IGI Global, Hershey. Karlsson, J., Olsson, S. & Ryan, K. (1997). Improved practical support for large-scale requirements prioritising. Requirements Engineering, 2(1), 51-60. Karlsson, J. & Ryan, K. (1997). A cost-value approach for prioritizing requirements. Software, IEEE, 14(5), 67-74. Lloyd, D., Dykes, J. & Radburn, R. (2007), Understanding geovisualization users and their requirements – a user-centred approach GIS Research UK 15th Annual Conference, ed. A.C. Winstanley, Maynooth, Ireland.  Lloyd, D., Dykes, J. & Radburn, R. (2008), Mediating geovisualization to potential users and prototyping a geovisualization application, GIS Research UK 16th Annual Conference, ed. D. Lambrick, Manchester, England. Lorenz, M.O. (1905). Methods of Measuring the Concentration of Wealth. Publications of the American Statistical Association, 9(70), 209-219. Maiden, N., Gizikis, A. & Robertson, S. (2004). Provoking creativity: imagine what your requirements could be like. IEEE Software, 21(5), 68-75. Malczewski, J. (2006). GIS based multicriteria decision analysis: a survey of the literature. International Journal of Geographical Information Science, 20(7), 703-726. Robertson, S. (2001). Requirements trawling: techniques for discovering requirements. International Journal of Human-Computer Studies, 55(4), 405-422. Saaty, T.L. (1977). A scaling method for priorities in hierarchical structures. Journal of Mathematical Psychology, 15(3), 234-281. Saaty, T.L. (1980). The Analytic Hierarchy Process. McGraw-Hill New York. Wasil, E. & Golden, B. (2003). Celebrating 25 years of AHP-based decision making. Computers and Operations Research, 30(10), 1419-1420. 

Using the Analytic Hierarchy Process to prioritise candidate improvements to a geovisualization application

Abstract

Similar works

Full text

Available Versions

City Research Online