Location of Repository

Systemic Evaluation Methodology for Technology Supported Learning

By Diane Hart


Developments in information and communication technologies have opened up new possibilities for organising and supporting learning activity. Appropriate evaluation methodology is required to help practitioners understand how and why their implemented learning designs work, so that they can improve their practice. It is argued that progress in learning about good practice in this domain has been slow because of inadequate methodology for inquiry into the complexity of organised learning activity. Methodology is needed to develop a better understanding of the social factors that influence learning and its outcomes, and the role of technology in this process.\ud This research has investigated the use of systems thinking in evaluation methodology. It is argued that using systems thinking to underpin evaluation practice provides a theoretically robust evaluation methodology. The research principally draws from critical systems theory to rationalise an approach to integrating ideas from different systems models, frameworks and methodologies, in particular those from social systems theory (Churchman 1971), Soft Systems Methodology (Checkland 1981), cybernetic systems from a human-centred perspective (Beer 1972, 1979, 1982, 1985) and Critical Systems Heuristics (Ulrich 1983). The central contribution is evaluation methodology that has been found to work in practice and has been rigorously tested through an action research approach across multiple cases. It is argued that the methodology helps to further clarify the complexity associated with evaluating organised human activity, for example associated with the roles, relationships and activity of multiple stakeholders with an interest in the activity being evaluated and the evaluation activity, and the relationship between strategies at different levels of organisation such as projects, programs, and the wider society in which they are situated. This leads to improvement in the relevancy of information generated by evaluation for answering different questions of interest to these multiple stakeholders. \u

Publisher: Management School (Sheffield)
Year: 2010
OAI identifier: oai:etheses.whiterose.ac.uk:1354

Suggested articles



  1. (2004). A Churchmanian theory of knowledge management system design,
  2. (1997). A competence perspective on strategic learning and knowledge management, in
  3. (1996). A comprehensive typology of program evaluation,
  4. (1996). A critical perspective on action research as a method for information systems research,
  5. (1994). A dynamic theory of organizational knowledge creation,
  6. (1999). A framework for pedagogical evaluation of e-learning environments,
  7. (2005). A history and critique of quality evaluation in the UK,
  8. (2005). A meta-analysis of 37 projects funded under the Learning and Teaching Development Grant Scheme,
  9. (2001). A systems thinking framework for knowledge management,
  10. (1996). A world larger than formative and summative,
  11. (1975). Academic and Entrepreneurial Research: Consequences of diversity in federal evaluation studies.
  12. (2006). Accessed online at http://www.hefce.ac.uk/pubs/rdreports/2005/rd02_05/, 6
  13. (2000). Action learning and action research: improving the quality of teaching and learning,
  14. (1946). Action research and minority problems,
  15. (2009). An interdisciplinary approach to enhancing sustainable development teaching in the higher education built environment curriculum: Learning from a curriculum development project at the University of Sheffield,
  16. (1956). An introduction to cybernetics,
  17. (2007). Analysis, synthesis, systems thinking and the scientific method: Rediscovering the importance of open systems,
  18. (1998). Applying a Theory of Change Approach,
  19. (2007). Barriers to integrating quantitative and qualitative research,
  20. (2003). Benchmarking the process of “interpretive” research in information systems,
  21. (2001). Bridging the relevance gap: Aligning stakeholders in the future of management research,
  22. (1999). Building capacity for organisational learning through evaluative inquiry,
  23. (2002). Building collaborative capacity through ‘Theories of Change’: Early lessons from the evaluation of Health Action Zones in
  24. (2004). Building knowledge about community change: Moving beyond evaluations,
  25. (1993). By their fruits ye shall know them”: A reply to Jackson, Green,
  26. (2000). Case Studies,
  27. (1982). Choices for the manager: A guide to managerial work and behaviour,
  28. (2008). Closing the loop: Promoting synergies with other theory building approaches to improve system dynamics practice,
  29. (1998). Communities of practice: Learning as a social system, originally published in Systems Thinker. Accessed online at http://www.co-i-l.com/coil/knowledge-garden/cop/lss.shtml,
  30. (1998). Communities of practice. Learning, meaning and identity, Cambridge:
  31. (2000). Competing interpretations of computer supported co-operative work,
  32. (2006). Complementarism versus pluralism: Are they different and does this matter?
  33. (2008). Concurrent meta-evaluation: A critique,
  34. (2007). Constructing Theories of Change: Methods and sources,
  35. (1981). Creating the corporate future,
  36. (2006). Creative Holism: A critical systems approach to complex problem situations,
  37. (1991). Creative problem solving: Total systems intervention,
  38. (2001). Critical facilitation: Learning through intervention
  39. (1983). Critical Heuristics of Social Planning: A new approach to practical philosophy.
  40. (1987). Critical heuristics of social systems design,
  41. (2007). Critical heuristics: A contribution to addressing the vexed question of so-called knowledge management,
  42. (1997). Critical Management Learning,
  43. (1992). Critical systems thinking and pluralism: A new constellation, Ph.D Thesis,
  44. (2001). Critical systems thinking and practice,
  45. (1996). Critical systems thinking, paradigms, and the modelling space,
  46. (1997). Critiquing multimethodology as metamethodology: Working towards pragmatic pluralism, in
  47. (2002). Cultivating communities of practice,
  48. (1948). Cybernetics,
  49. (1998). Decoding program logic models, from Workshop presented at the Annual Meeting of the American Evaluation Association,
  50. (2004). Designing for learning: an update on the Pedagogy strand of the JISC e-learning programme. Accessed online at http://www.jisc.ac.uk/uploaded_documents/ACFAB9.doc,
  51. (2003). Designing knowledge management Systems for a EuroMediterranean network of Practitioners: Preparatory work for the MEDFORIST project,
  52. (2006). Developing a participative approach to supporting the design and evaluation of new learning and teaching initiatives -International Consortium for Educational Development conference,
  53. (1971). Developing effective managers: a new approach to business education,
  54. (1985). Diagnosing the system for organizations,
  55. (2002). E-tivities: The key to active online learning,
  56. (1987). Educating the reflective practitioner: Toward a new design for teaching and learning in the professions,
  57. (1973). Education for critical consciousness, London: Sheed and Ward.
  58. (2005). Education Research and Information
  59. (2003). Emancipatory potential of action learning: a critical analysis,
  60. (1994). Empowerment evaluation,
  61. (1981). Evaluating Evaluation Methods,
  62. (1975). Evaluating the arts in education: A responsive approach,
  63. (1999). Evaluation Research: An introduction to principles, methods and practice,
  64. (1972). Evaluation Research: Methods of assessing program effectiveness,
  65. (2004). Evaluation: A systematic approach, 7 th Edition. Thousand Oaks,
  66. (2003). Evidence-based practice and e-learning in Higher Education: can we and should we?
  67. (2003). Evidence, Understanding and Complexity: Evaluation in non-linear systems,
  68. (1959). Experience and reflection,
  69. (1984). Experiential learning: Experience as the source of learning and development,
  70. (1996). Experiential management education as the practice of change in
  71. (1996). Exploring internal stickiness: Impediments to the transfer of best practice within the firm,
  72. (2009). Exploring stakeholder engagement in impact evaluation planning in educational development work,
  73. (2004). Exploring the missing links between cause and effect: A conceptual framework for understanding micro-macro conversions in programme evaluation,
  74. (1996). Foucault plays Habermas: An alternative philosophical underpinning for critical systems thinking.
  75. (1991). Foundations of program evaluation: Theories of practice,
  76. (1989). Fourth Generation Evaluation,
  77. (2003). Freedom from command and control: A better way to make the work work,
  78. (1970). Freedom in a rocking boat: changing values in an unstable society,
  79. (2005). Freeing the chi of change: the Higher Education Academy and enhancing teaching and learning
  80. (1997). From metatheory to multimethodology, in
  81. (1999). Funding Council for England
  82. (2008). Funding Council for England (2002) Teaching Quality Enhancement Fund: funding arrangements 2002-03 to 2004-5. Reference 02/24, Bristol: Higher Education Funding Council for England. Accessed online at http://www.hefce.ac.uk/pubs/hefce/2002/02_24.htm, 29
  83. (2008). Funding Council for England (2006a) Teaching Quality Enhancement Fund: funding arrangements 2006-7 to 2008-9. Reference 2006/11, Bristol: Higher Education Funding Council for England. Accessed online at http://www.hefce.ac.uk/pubs/hefce/2006/06_11/, 29
  84. (2006). Funding Council for England (2006b) Learning and teaching evaluation framework. Accessed online at http://www.hefce.ac.uk/learning/evaluation, 6
  85. (1994). Handbook of practical program evaluation,
  86. (1997). How can theory-based evaluation make greater headway?
  87. (1983). Human systems are different, London: Harper and Row.
  88. (1975). Impediments to the use of management information systems,
  89. (2003). Implementing learning and teaching strategies, Teaching Quality Enhancement Fund National Co-ordination Team Report, Buckingham: Open University. Accessed online at http://www.heacademy.ac.uk/documents/ilts.pdf,
  90. (1961). Industrial Dynamics,
  91. (2006). Integrating quantitative and qualitative research: how is it done?
  92. (2007). Introduction in
  93. (2000). It is what one does: Why people participate and help others in electronic communities of practice,
  94. (2001). Joining the dots: Reconceptualizing educational development,
  95. (1981). Joint Committee on Standards for Educational Evaluation
  96. (2005). Joint Information Services Committee report. Accessed online at http://www.jisc.ac.uk/uploaded_documents/Review%20models.doc,
  97. (1961). Knowing and being, reprinted in
  98. (1972). Knowledge and Human Interests,
  99. (2001). Knowledge and organization: A social-practice perspective,
  100. (1998). Knowledge assets: Securing competitive advantage in the information economy,
  101. (2000). Knowledge management mistakes. July Edition of Computer World. Accessed online at www.computerworld.com, 20
  102. (1995). Knowledge, knowledge work and organizations: An overview and interpretation,
  103. (2003). Last accessed online at http://www.eumedis.net/en/about/objectives/,
  104. (1995). Learning from failure: the systems approach,
  105. (2004). Logic Model Development Guide,
  106. (2002). Long-term outcomes of educational action research projects,
  107. (1996). Making sense of management: A critical introduction,
  108. (1967). Managers and their jobs,
  109. (2007). Managing e-learning group processes using teleological enquiring principles,
  110. (2000). Managing knowledge: Building blocks for success,
  111. (2002). Managing viable knowledge,
  112. (2000). Meta-evaluation as a tool for learning: A case study of the European Structural Fund Evaluations in Finland,
  113. (1978). Mind in Society: The development of higher psychological processes, Cambridge:
  114. (2002). Mission Impossible? Communicating and sharing knowledge via information technology,
  115. (1997). Mixing methods: Developing Systemic Intervention, in
  116. (1993). Motivating knowledge workers: the challenge for the 1990s, Longe Range Planning,
  117. (2000). Motivation, knowledge transfer and organizational forms,
  118. (1997). Multimethodology: The theory and practice of combining management science methodologies,
  119. (1996). Multimethodology: Towards a framework for critical pluralism,
  120. (1997). Multiparadigm multimethodology, in
  121. (2008). Of methods and methodology,
  122. (1995). Open systems evaluation and the logic model: programme planning and evaluation tools, Evaluation and Program Planning,
  123. (1991). Organizational learning and communities of practice: Toward a unified view of working, learning and innovation,
  124. (1997). Organizational learning and the learning organization: A dichotomy between descriptive and prescriptive research,
  125. (1996). Organizational learning II: Theory, Method and practice,
  126. (1978). Organizational learning: A theory of action perspective,
  127. (1998). Organizational learning: Diverging communities of practice?
  128. (2002). Organizing reflection,
  129. (2007). Participative impact evaluation– How can it help improve the student experience of UK higher education?, UK Evaluation Society Conference,
  130. (1986). Piaget’s theory of knowledge,
  131. (1997). Pluralism in systems thinking and practice,
  132. (1980). Power/knowledge,
  133. (1988). Principles of instructional design, Third edition,
  134. (2000). Problematizing participation: A critical review of approaches to participation in evaluation theory,
  135. (2006). Prosperity for all in the global economy- world class skills, final report of the Leitch review of skills,
  136. (1999). Purposes and use: What can we expect?
  137. (2006). Quality Assurance Agency
  138. (1997). Realistic evaluation,
  139. (2000). Reconciling process and outcome in evaluating community initiatives,
  140. (2008). Reflecting on reflexivity: Reflexive textual practices in organization and management theory,
  141. (2000). Reflective practice in the civil society: the contribution of critically systemic thinking,
  142. (2000). Reflexive methodology: New vistas for qualitative research,
  143. (1993). Rethinking university teaching: A framework for the effective use of educational technology,
  144. (2004). Review: developing e-learning models for the JISC Practitioner Communities.
  145. (2004). Scale, complexity and the representation of theories of change,
  146. (2000). SECI, Ba and leadership: a unified model of dynamic knowledge creation,
  147. (2005). Select Committee on Education and Skills, Third Report, London: United Kingdom Parliament. Accessed online at http://www.publications.parliament.uk/pa/cm200405/cmselect/cmeduski/205/20502.htm,
  148. (1967). Sense-giving and sense-reading, reprinted in
  149. (1991). Situated Learning: Legitimate peripheral participation, Cambridge:
  150. (1985). Social systems theory and practice: the need for a critical approach,
  151. (1993). Some difficulties of ecological thinking, considered from a critical systems perspective: a plea for critical holism,
  152. (1985). stakeholder-based evaluation and value judgements,
  153. (1980). Standards for Evaluation (draft),
  154. (2005). Survey: Use of ICT and e-learning. Accessed online at http://www.elearningscotland.org,
  155. (2008). System dynamics as model-based theory building,
  156. (2004). Systemic evaluation methodology: The emergence of social learning from environmental ICT prototypes,
  157. (2000). Systemic Intervention: Philosophy, methodology and practice,
  158. (2007). Systems concepts in evaluation: An expert anthology, Point Reyes, Ca.: EdgePress. Accessed online at http://www.wkkf.org, 29
  159. (1991). Systems methodology for the management sciences,
  160. (1990). Systems thinking and critique I. What is critique? Systems Practice,
  161. (2008). Systems thinking in the public sector: the failure of the reform regime and a manifesto for a better way,
  162. (1981). Systems thinking, systems practice,
  163. (1999). Systems thinking, systems practice: includes a 30-year retrospective,
  164. (2003). Systems thinking: Creative Holism for managers,
  165. (1999). Teaching for quality learning at University: What the student does, Buckingham:Society for Research into Higher Education and Open
  166. (1965). The art of judgement,
  167. (1972). The brain of the firm,
  168. (1981). The brain of the firm. 2 nd Edition,
  169. (2002). The construction of 'communities of practice' in the management of innovation,
  170. (1990). The core competence of the corporation,
  171. (1971). The design of inquiring systems,
  172. (1999). The economies of online cooperation: Gifts and public goods in cyberspace,
  173. (1995). The essence of business process engineering,
  174. (1990). The fifth discipline: The art and practice of the learning organization,
  175. (1998). The foundations of social research,
  176. (1979). The heart of the enterprise,
  177. (2004). The impact of a national policy to enhance teaching quality and status,
  178. (1995). The knowledge creating company: How Japanese companies create the dynamics of innovation,
  179. (2004). The limits of a technological fix to knowledge management: Epistemological, political and cultural issues in the case of intranet implementation,
  180. (1964). The logic of tacit inference, reprinted in
  181. (1992). The manual of learning styles,
  182. (2001). The meta-evaluation imperative,
  183. (1967). The Methodology of evaluation’ in
  184. (1973). The nature of managerial work,
  185. (1982). The nature of soft systems thinking: The work of Churchman, Ackoff and Checkland,
  186. (1994). The new production of knowledge: The dynamics of science and research in contemporary societies,
  187. (1972). The pedagogy of the oppressed,
  188. (2005). The practice and the theory: Lessons from the application of a Theories of Change approach,
  189. (1955). The practice of management,
  190. (2001). The reflective institution: Assuring and enhancing the quality of teaching and learning,
  191. (1983). The reflective practitioner: how professionals think in action,
  192. (2001). The relationship of systems thinking to action research,
  193. (1996). The rise of the network society,
  194. (1993). The road to emancipation is through organizational development: A critical evaluation of Total Systems Intervention, Systems Practice,
  195. (1998). The role of experience in formulating theories of evaluation practice.
  196. (1991). The scene for interpretive systemology,
  197. (2001). The sociological foundations of organizational learning,
  198. (1979). The systems approach and its enemies,
  199. (1966). The Tacit Dimension, London: Routledge and Paul.
  200. (1998). The theory and practice of boundary critique: developing housing services for older people,
  201. (1996). The theory behind practical evaluation,
  202. (1984). The theory of communicative action, Volume One: Reason and Rationalisation of Society,
  203. (1984). The theory of communicative action, Volume Two: The critique of functionalist reason,
  204. (1988). The Toyota Production System,
  205. (2007). Theories of Change and Realistic Evaluation: Peas in a pod or apples and oranges?
  206. (1994). Theory-driven evaluation: Need, difficulties and options,
  207. (1990). Theory-driven evaluations,
  208. (1989). Total Quality Management,
  209. (2001). Towards a second generation of KM? The people management challenge,
  210. (1984). Towards a system of systems methodologies,
  211. (1997). Towards critical pluralism, in
  212. (2003). Towards Process Modelling in
  213. (1996). Towards the theory and practice of competencebased competition,
  214. (2000). Transforming self and society: A “Critical Appreciation” model,
  215. (1996). Types of evaluation and types of evaluator,
  216. (1994). Understanding and evaluating methodologies,
  217. (1986). Understanding computers and cognition: A new foundation for design,
  218. (2000). Understanding management research: An introduction to epistemology, London: Thousand Oaks;
  219. (2004). Using data matrices,
  220. (2005). Using evaluation to create ‘provisional stabilities’: Bridging innovation in higher education change processes,
  221. (2004). Using templates in the thematic analysis of text, in
  222. (1986). Utilization-focussed evaluation,
  223. (1999). Virtual communities as communities: Net surfers don’t ride alone,
  224. (2001). What educational development units do: five years on,
  225. (1999). What is instructional-design theory and how is it changing? In Reigeluth,
  226. (1995). What is this thing called critical systems thinking? In Ellis, K. et al (Eds) Critical issues in systems theory and practice,
  227. (1998). Working knowledge: How organizations manage what they know,
  228. (2004). Workshop agreement: European guide to good practice in knowledge management,

To submit an update or takedown request for this paper, please submit an Update/Correction/Removal Request.