This paper deals with self-organized contextualized feedback loops. Based on a summary of related work in the domain of health and medicine we discuss the potential of personal informatics systems to build self-organized feedback contextualized loops that are controlled and initiated by learners. A design model is introduced and future research is discussed

Kalz, Marco

English

Open University of the Netherlands Research Portal

Open Universiteit www.ou.nl Designing Self-Organized Contextualized FeedbackLoopsCitation for published version (APA):Kalz, M. (2013). Designing Self-Organized Contextualized Feedback Loops.Document status and date:Published: 01/07/2013Document Version:Peer reviewed versionDocument license:CC BY-SAPlease check the document version of this publication:• A submitted manuscript is the version of the article upon submission and before peer-review. There can be important differences betweenthe submitted version and the official published version of record. People interested in the research are advised to contact the author for thefinal version of the publication, or visit the DOI to the publisher's website.• The final author version and the galley proof are versions of the publication after peer review.• The final published version features the final layout of the paper including the volume, issue and page numbers.Link to publicationGeneral rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright ownersand it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.• Users may download and print one copy of any publication from the public portal for the purpose of private study or research.• You may not further distribute the material or use it for any profit-making activity or commercial gain• You may freely distribute the URL identifying the publication in the public portal.If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license above, pleasefollow below link for the End User Agreement:https://www.ou.nl/taverne-agreementTake down policyIf you believe that this document breaches copyright please contact us at:pure-support@ou.nlproviding details and we will investigate your claim.Downloaded from https://research.ou.nl/ on date: 28 Oct. 2023Please cite this pre-print as: Kalz, M. (2013). Designing Self-Organized Feedback Loops. In Whitelock, D., Warburton, W., Wills, G., and Gilbert, L. (Eds.), International Conference on Computer Assisted Assessment (CAA 2013), University of Southampton, http://caaconference.com. Page 1  Designing self-organized contextualized feedback loops Marco Kalz Centre for Learning Sciences and Technologies, Open University of the Netherlands marco.kalz@ou.nl Abstract This paper deals with self-organized contextualized feedback loops. Based on a summary of related work in the domain of health and medicine we discuss the potential of personal informatics systems to build self-organized feedback contextualized loops that are controlled and initiated by learners. We introduce a design model and discuss future research. Introduction Feedback is one of the most effective interventions that have a clear impact on learning success. We define educational feedback with Hattie & Timperley (2007) as “information provided by an agent … regarding one’s performance or understanding”. In a meta-review by Hattie (2009) several educational interventions have been summarized and evaluated according to their effect size and impact on achievement of learners. With an effect size of 0.8 feedback is one of the most powerful interventions that have direct effects in the learning outcomes of learners. There has been extensive research on feedback. Although Shute (2008) has concluded that there is “no consistent pattern of results” in a recent meta-analysis the following components of effective feedback practices in higher-education have been identified by Evans among others (2013):  • A range and choice of assessment opportunities should be given to learners • Availability of learning resources in a digital environment to enable self-directed learning • Feedback and assessment opportunities from the very beginning • Developing self-assessment literacy • Clear and focused feedback including pinpointing areas of improvement Designing self-organized contextualized feedback loops Page 2 In a recent book by Boud & Molloy (2013) the critical role of the learner in the feedback process is stressed. The authors point to the growing dissatisfaction of students with current feedback practices and the need to enlarge the focus from single feedback interventions to longer feedback episodes in which the preparation phase and the interpretation phase is also taken into account for effective feedback practices. In addition several contributions in the book and also the review by Evans (2013) stress the importance to give learners more control over the feedback process. This mission statement for future educational feedback design is even more important when we take into account that in other contexts learners have already the means to design their own feedback channels and feedback opportunities.  The pervasive use of mobile technologies and social media have thus enabled a practice of what we call self-organized contextualized feedback. This feedback is actively triggered by a learner and it can take into account the current context. This trend is in line with the argument by Hattie & Timperley (2007) that feedback needs to regard the context in which it is provided to be powerful in effect and with the current discussion about feedback loops. In contrast to the low level of formative assessment in formal learning contexts, learners are using more and more so called feedback loops in their private life to document and control their behavior, define goals and provide and receive feedback. These feedback loops are in recent times mostly designed with the help of mobile and pervasive technologies. Examples of these feedback loops come from sports (run tracking including public performance measuring), medicine (tracking of healthy behaviour or medication), energy saving or dedicated self-improvement programs (e.g. management training). The website of the Quantified Self Community1 lists more than 500 tools that individuals use to track a behaviour and implement triggers for change. According to (Goetz, 2011) feedback loops consist of four distinct stages:  Figure 1: Feedback loops  • Data collection: A behaviour or performance must be measured, captured and stored • Evidence presentation: The data must be presented to the learner in an easy and understandable way and in a context that makes it emotionally resonant • Consequence: The information shows possible paths ahead • Action: The feedback loop end with a concrete action that again can produce new data so that the feedback loop can run again  We envision this process of ongoing feedback loops to be transferred to the learning                                             1 http://quantifiedself.com  Designing self-organized contextualized feedback loops Page 3 context. Mory (2004) has analysed the different variables influencing the design of feedback interventions. These variables are among others type of information content, complexity, timing and type of error analysis. She states that newer technologies might offer “more instructional delivery options and a wider variety of modalities through which to deliver feedback”. While the timing of feedback has been discussed in this meta-review under the light of immediate versus delayed feedback we believe that with the advancement of mobile and pervasive technologies a new type of feedback emerged called contextualized feedback. This contextualized feedback can use sensors to bind the timing of feedback to different prerequisites. It is thus possible to use location sensors like GPS in combination with the timing of feedback. After arriving at a specific location learners could for example receive feedback prompts that trigger them to start the next learning activity. In this paper we discuss the potential of the use of pervasive technologies and self-organized feedback by learners. In the next part we summarize related work mostly from the field of health and behavior change and propose a model for self-organized contextualized feedback. Last but not least we discuss future research. Self-organized contextualized feedback: Related work Several studies show the power of self-organized contextualized feedback. In a recent meta-analysis the effectiveness of mobile technology for health behavior change and disease management is analysed by Free et al. (2013). It is argued in this meta-review that mobile technologies are effective for reaching behaviour change since they allow a temporal alignment and the provision of feedback when it is most relevant. The authors provide an example from smoke cessation in which feedback interventions are delivered when the users experience cravings due to withdrawal from nicotine. This is an interesting example from a new type of feedback provided in many health apps and behavior change apps. The daily routines and expected behavior is taken as a structure for delivering feedback in context. In another randomized controlled trial Hurling et al. (2007) have controlled the impact of a mobile support program for physical activity. Again the test group was given a support system in which the participants have actively scheduled activities for the upcoming week to be compared later with the real performance to provide feedback. Michie, Van Stralen, & West (2011) provided a good overview about related research. Besides these case studies there is also an emerging research focus in computer science that deals with so called personal informatics systems. Li, Dey, & Forlizzi (2010) define personal informatics systems as tools that “help people collect personally relevant information for the purpose of self-reflection and gaining self-knowledge”. Personal informatics system support users in 5 process stages: The preparation, collection and integration of information leading to a stage of reflection and action. In these stages user-driven input can be combined with system-driven input. This results in a partly adaptive and partly adaptable system that will allow learners to constantly repeat feedback loops, improve their learning and initiate meta-cognitive reflection. A model for self-organized contextualized feedback Based on the earlier discussion in the paper we have developed a model for self-organized contextualized feedback (see fig 2). In the center of this model is the learner who is involved in an ongoing feedback cycle and equipped with mobile and pervasive technologies giving him access to personal informatics systems to prepare and collect evidence to trigger feedback that gets integrated into the next actions. To receive Designing self-organized contextualized feedback loops Page 4 feedback the learner has access to three types of technological services: Automated assessment and feedback services, feedback spaces (for asynchronous feedback) and feedback channels (for direct contact to a domain expert). This feedback is directed at different types of feedback mirrors. These mirrors are one self, peers or a more knowledgeable other/tutor. The feedback type can differ according to Hattie & Timperley (2007) in feedback directed at the goals of the learner (feed-up), the current performance (feedback) or the next steps in the learning process (feed forward).   Figure 2: Design model for self-organized contextualized feedback loops Again we follow Hattie & Timperley (2007) an their proposal to differentiate between four different levels on which feedback can have an impact, namely the task, process, self-regulation or the personality of the learner. Discussion and future research In this paper we have introduced the concept of self-organized contextualized feedback loops. Due to the need to change feedback practices and the increasing gap between self-organized feedback loops in a private context and in higher education we have proposed a design model for self-organized contextualized feedback loops that combines components of personal informatics systems with state-of-the-art concepts for educational feedback.  We are currently using the design model above for the development and testing of a mobile app called LearnFrame. Since the student population at the Open University of the Netherlands consists mostly of professionals that study part time and who have to combine their learning process with their job and family life, the efficient usage of smaller time frames is an important factor for their study success. Based on a collection of typical daily patterns and the definition of recurring contexts the learners are able to trigger reminders and feedback messages focused on the learning process level. In the future this basic learning process support should be enriched by self-assessment and peer-assessment and peer-feedback via feedback spaces. The mobile app will be evaluated in Designing self-organized contextualized feedback loops Page 5 its potential to change the learners’ perception about assessment and the level of control and self-direction in the feedback process. References Boud, D., & Molloy, E. (Eds.). (2013). Feedback in Higher and Professional Education: Understanding it and Doing it Well. Oxon, UK: Routledge. Evans, C. (2013). Making Sense of Assessment Feedback in Higher Education. Review of Educational Research, 83(1), 70–120. Free, C., Phillips, G., Galli, L., Watson, L., Felix, L., Edwards, P., Patel, V., et al. (2013). The effectiveness of mobile-health technology-based health behaviour change or disease management interventions for health care consumers: a systematic review. PLoS medicine, 10(1), e1001362. doi:10.1371/journal.pmed.1001362 Goetz, T. (2011, June). Harnessing the power of feedback loops. Wired Magazine, June. Retrieved from http://www.wired.com/magazine/2011/06/ff_feedbackloop/ Hattie, J, & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. Hattie, JAC. (2009). Visible learning: A synthesis of 800+ meta-analyses on achievement. Oxford, UK: Routledge.  Hurling, R., Catt, M., Boni, M. De, Fairley, B. W., Hurst, T., Murra, P., & Sodhi, A. R. J. S. (2007). Using Internet and Mobile Phone Technology to Deliver an Automated Physical Activity Program: Randomized Controlled Trial. Journal of Medical Internet Research, 9(9). Li, I., Dey, A., & Forlizzi, J. (2010). A stage-based model of personal informatics systems. In E. Mynatt, S. D, G. Fitzpatrick, S. Hudson, K. Edwards, & T. Rodden (Eds.), Proceedings of the 28th international conference on Human factors in computing systems - CHI  ’10 (pp. 557–566). Atlanta, Georgia, USA: ACM. Michie, S., Van Stralen, M. M., & West, R. (2011). The behaviour change wheel: A new method for characterising and designing behaviour change interventions. Implementation science, 6(1), 42. Mory, E. H. (2004). Feedback research revisited. In D. H. Jonassen (Ed.), Handbook of research on educational communications and technology (Vol. 45, pp. 745–784). Lawrence Erlbaum Associates. doi:10.1007/s00127-009-0052-2 Shute, V. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189.    

Designing Self-Organized Contextualized Feedback Loops

https://research.ou.nl/files/946618/CAA-2013-mkalz-dspace.pdf

Designing Self-Organized Contextualized Feedback Loops

Abstract

Similar works

Full text

Available Versions

Open University of the Netherlands Research Portal