Location of Repository

Between analysis and transformation: technology, methodology\ud and evaluation on the SPLICE project

By Mark Johnson, Peter Lager, Bill Pollard, Graham Hall, Miranda Edwards, Pete Whitfield and Rupert Ward


This paper concerns the ways in which technological change may entail methodological development in e-learning research. The focus of our argument centres on the subject of evaluation in e-learning and how technology can contribute\ud to consensus-building on the value of project outcomes, and the identification of mechanisms behind those outcomes.\ud \ud \ud We argue that a critical approach to the methodology of evaluation which harnesses technology in this way is vital to agile and effective policy and strategymaking in institutions as the challenges of transformation in a rapidly changing educational and technological environment are grappled with.\ud \ud \ud With its focus on mechanisms and multiple stakeholder perspectives, we identify Pawson and Tilley’s ‘Realistic Evaluation’ as an appropriate methodological approach for this purpose, and we report on its use within a JISC-funded project on\ud social software, SPLICE (Social Practices, Learning and Interoperability in Connected Environments). The project created new tools to assist the identification of mechanisms responsible for change to personal and institutional technological practice. These tools included collaborative mind-mapping and focused questioning, and tools for the animated modelling of complex mechanisms. By using these tools, large numbers of project stakeholders could engage in a process where they were encouraged to articulate and share their theories and ideas as to why project outcomes occurred. Using the technology, this process led towards the identification and agreement of common mechanisms\ud which had explanatory power for all stakeholders.\ud \ud \ud In conclusion, we argue that SPLICE has shown the potential of technologicallymediated Realistic Evaluation. Given the technologies we now have, a methodology based on the mass cumulation of stakeholder theories and ideas about mechanisms is feasible. Furthermore, the summative outcomes of such a process are rich in explanatory and predictive power, and therefore useful to the immediate and strategic problems of the sector. Finally, we argue that as well as generating better explanations for phenomena, the evaluation\ud process can itself become transformative for\ud stakeholders

Topics: LC5201, LB2300
Publisher: Association for Learning Technology
Year: 2009
OAI identifier: oai:eprints.hud.ac.uk:7649
Downloaded from http://alt.ac.uk

Suggested articles



  1. (2001). (eds) Education in cyberspace. London: RoutledgeFalmer
  2. (1990). A cognitive perspective on medical expertise: theory and implication. doi
  3. (1998). a) The Net Generation and the School. Milken Family Foundation.
  4. (2008). Active Learning in Higher Education doi
  5. (2008). An investigation of differences in undergraduates’ academic use of the internet. doi
  6. (2009). Collaboration and the Net generation: The changing characteristics of first year university students. doi
  7. (2001). Computer assisted learning in undergraduate medical education. doi
  8. (2008). Creating the future: The Use and Misuse of Scenarios.
  9. (2002). Cultivating Communities of Practice. A guide to manage knowledge. doi
  10. (2005). Deceit, desire and control: the identities of learners and teachers in cyberspace.
  11. (2008). Devising Futures for Universities in a Multi-level Structure: A Methodological Experiment. doi
  12. (2001). Digital Natives, Digital Immigrants Part II: Do they really think differently? On the Horizon. doi
  13. (2009). Discourse Analysis Online, available online extra.shu.ac.uk/daol/articles/v1/n1/a3/ thurlow2002003-paper.html [accessed on
  14. (2004). E-moderating: the key to teaching and learning online (2nd ed.). London and New York.: Routledge Salmon,
  15. (1999). Educating the Net generation. doi
  16. (1987). Educating the reflective practitioner. Address to the 1987 meeting of the American Educational Research Association. Available at educ.queensu.ca/~russellt/ howteach/schon87.htm [accessed on February 23rd
  17. (2008). Educational usages of podcasting. doi
  18. (2005). Effective e-learning for health professionals and students — barriers and their solutions. A systematic review of the literature — findings from the HeXL project. Health Info Libr doi
  19. (2009). Explorations in course-casting: podcasts in higher education. doi
  20. (2005). Faceless students, virtual places: emergence and communal accountability in online classrooms. [online] doi
  21. (2008). First year students’ experiences with technology: Are they really digital natives? ‘Questioning the net generation: A collaborative project in Australian higher education’.
  22. (2008). Future Learning: Desire or Fate? Paper presented at JISC Innovating e-Learning online conference 2008. Available at www.online-conference.net/jisc/ content2008/salmon/salmon.pdf [accessed on
  23. (2008). Great expectations of ICT: How Higher Education institutions are measuring up, www.jisc.ac.uk/publications/publications/greatexpectations [accessed Nov 4th
  24. (1998). Growing up digital: the rise of the Net generation. doi
  25. (2008). Grown up digital: How the Net generation is changing your world. doi
  26. (2009). H. sapiens digital: From digital immigrants and digital natives to digital wisdom. doi
  27. (2009). Impact of Podcasts in teacher Education: from Consumers to Producers.
  28. (2008). Influence of Podcasts Characteristics on Higher Students’ Acceptance. In
  29. (2008). Information behaviour of the researcher of the future www.jisc.ac.uk/ whatwedo/programmes/resourcediscovery/googlegen.aspx [accessed Nov 3rd
  30. (2008). Information Behaviour of the Researcher of the Future. Ciber briefing report.
  31. (2009). Information Design for Podcasts. In C. Montgomerie &
  32. (2008). Information Systems Committee doi
  33. (2005). Learning as peripheral participation in communities of practice: a reassessment of key concepts in workplace learning. doi
  34. (2007). Listen and learn: A systematic review of the evidence that podcasting supports learning in higher education.
  35. (2007). Magic Quadrant for Business Process Management Suites.
  36. (2002). Mobile Technology Appropriation in a the Seminario sobre Desarrollo Económico. Desarrollo Social y Comunicaciones Móviles en América Latina, Buenos Aires,
  37. (2003). Mobile Transformations of ‘Public’ and ‘Private’ Life’. doi
  38. (2008). Open Teaching: The Key to sustainable and effective open education.
  39. (2008). Opening Up Education: The Collective Advancement of Education through Open Technology, Open Content, and Open Knowledge. doi
  40. (1995). Outcome Controllability and Counterfactual Thinking. Personality and Social Psychology Bulletin, doi
  41. (2008). Personal profiles: Facilitating participation in Learning Networks. Available at hdl.handle.net/1820/1190, 1.TENC: Publications and Preprints.
  42. (2006). Podcasting for Learning in Universities. New York:
  43. (2002). Reflexive dialogical practice in management learning. doi
  44. (1991). Situated Learning: Legitimate peripheral participation. doi
  45. (2002). Sleep loss and fatigue in residency training: a reappraisal. JAMA: doi
  46. (2004). Student choice of anonymity for learner identity in online learning discussion forums.
  47. (2007). Talk the talk: Learner-generated podcasts as catalysts for knowledge creation. doi
  48. (1996). Technology and the self: From the essential to the sublime. In doi
  49. (1980). The Cost of Thinking. doi
  50. (2009). The Net Generation enters university: What are the implications for Technology Enhanced Learning?
  51. (1982). The Uses of the University. Cambridge, MA.,
  52. (1990). The Written World: On the Theory and Practice of Computer Conferencing.” In Mindweave: Communication, Computers and Distance Education, edited by Robin Mason and Anthony Kaye,
  53. (2005). There’s Something in the Air: Podcasting
  54. (2007). Top of the pods — In search of podcasting pedagogy for language learning. doi
  55. (2008). Towards a user-oriented analytical approach to learning design. [research paper].
  56. (2009). Understanding and Evaluating BPM Suites. Bruse Silver Associates. Available at www.bpminstitute.org/articles/article/article/bpms-watch-understanding-andevaluating-bpm-suites.html [accessed on

To submit an update or takedown request for this paper, please submit an Update/Correction/Removal Request.