Delving into instructor‐led feedback interventions informed by learning analytics in massive open online courses

Producción CientíficaBackground:Providing feedback in massive open online courses (MOOCs) is chal-lenging due to the massiveness and heterogeneity of learners' population. Learninganalytics (LA) solutions aim at scaling up feedback interventions and supportinginstructors in this endeavour.Paper Objectives:This paper focuses on instructor-led feedback mediated by LAtools in MOOCs. Our goal is to answer how, to what extent data-driven feedback isprovided to learners, and what its impact is.Methods:We conducted a systematic literature review on the state-of-the-art LA-informed instructor-led feedback in MOOCs. From a pool of 227 publications, weselected 38 articles that address the topic of LA-informed feedback in MOOCs medi-ated by instructors. We applied etic content analysis to the collected data.Results and Conclusions:The results revealed a lack of empirical studies exploring LA todeliver feedback, and limited attention on pedagogy to inform feedback practices. Our find-ings suggest the need for systematization and evaluation of feedback. Additionally, there isa need for conceptual tools to guide instructors' in the design of LA-based feedback.Takeaways:We point out the need for systematization and evaluation of feedback. Weenvision that this research can support the design of LA-based feedback, thus contribut-ing to bridge the gap between pedagogy and data-driven practice in MOOCs.Consejo de Investigación de Estonia (PSG286)Ministerio de Ciencia e Innovación - Fondo Europeo de Desarrollo Regional y la Agencia Nacional de Investigación (grant PID2020-112584RB-C32) and (grant TIN2017-85179-C3-2-R)Junta de Castilla y León - Fondo Social Europeo y el Consejo Regional de Educación (grant E-47-2018-0108488

Assisting Instructors in the Design and Provision of Personalized LA-informed Feedback in MOOCs

Massive Open Online Courses (MOOCs) have gained increasing prominence in the educational landscape over the last decade. Despite their educational benefits and global adoption, MOOCs are still accompanied by several challenges that have an impact on the learning experience. The provision of timely and personalised feedback is one important challenge, often associated with learner disengagement. The field of Learning Analytics (LA) provides opportunities for scaling up the feedback interventions, enabling automated or semi-automated interventions. Yet, these LA solutions often lack course contextualisation, pedagogical grounding, and guidance for MOOC instructors in understanding and using such feedback tools. Building on this context, the current dissertation aims to assist instructors in the design and provision of personalised LA-informed feedback in MOOC environments. To this end, this dissertation proposes the accomplishment of three research objectives following a Design-Based Research methodological approach. This methodological approach resulted into four cycles, in which we employed a human-centred approach, involving MOOC instructors (and other stakeholders) both in the identification of the research problems, and in the design and refinement of the suggested proposals. The first research objective deals with understanding the current state of instructor-led LA-informed feedback in MOOCs. Accordingly, we conducted a systematic literature review to understand the current LA proposals that support personalised interventions in MOOCs, and their implications for learners and instructors. The second research objective delves into the need of helping MOOC instructors to shape personalised and contextualised feedback. In response to this need, this dissertation proposes a conceptual framework named FeeD4Mi. FeeD4Mi includes a 5-dimension conceptual structure, accompanied by a process, a set of catalogues and a set of recommendations based on the identified learners’ problems for their courses. The third research objective aims at providing a manageable design and provision of personalised and contextualised feedback strategies in MOOCs. To this end, this dissertation proposes a set of design guidelines that can be transformed in a web-based tool to incorporate the FeeD4Mi catalogues, process and recommendations in a digital format, enabling the computer-interpretable representation of feedback strategies in MOOCs and their automatic or semi-automatic implementation during the enactment of MOOCs. In total, four evaluative studies served to iteratively refine and assess FeeD4Mi regarding its completeness, usefulness for MOOC instructors, and impact of the feedback strategies designed on learners and instructors. At the same time, we assessed with MOOC instructors the usability, temporal workload, and potential adoption of the digital version of the framework. The results indicated the added value of the framework in guiding instructors in the design and provision of LA-informed feedback in MOOCs. Furthermore, the evaluators highlighted the flexibility of the tool, the possibility to automate feedback strategies and the usefulness in reflecting on potential learners’ problems, LA-based indicators and feedback reactions. Finally, the results of the evaluative studies also pointed out further research directions of feedback MOOCs and in other educational contexts where delivering feedback is also challenging (e.g., online, or hybrid teaching).Los cursos masivos abiertos en línea (MOOC por sus siglas en inglés) han recibido una gran atención en el panorama educativo en las últimas décadas. A pesar de sus beneficios educativos y su adopción global, los MOOC vienen acompañados de varios desafíos que afectan al desarrollo de la experiencia de aprendizaje. La provisión de retroalimentación personalizada y en un tiempo adecuado es uno de esos desafíos, a menudo asociado con la pérdida de interés de los estudiantes. El campo de las Analíticas de Aprendizaje (LA por sus siglas en inglés) brinda oportunidades para ampliar las intervenciones de retroalimentación, permitiendo la configuración de intervenciones automáticas o semiautomáticas. Sin embargo, estas soluciones de LA carecen de contextualización del curso, de una base pedagógica y orientación para los instructores de MOOC en la comprensión y uso de las herramientas. De acuerdo con estas ideas, esta Tesis Doctoral tiene como objetivo ayudar a los instructores en el diseño y la provisión de retroalimentación (feedback en inglés) personalizada basada en analítica de aprendizaje en entornos MOOCs. Esta tesis aborda tres objetivos de investigación siguiendo un enfoque metodológico de Investigación Basada en Diseño. La Investigación Basada en el Diseño se desarrolló en cuatro ciclos metodológicos, durante los cuales empleamos un enfoque "centrado en el ser humano", involucrando a los instructores MOOC tanto en la identificación de los problemas de investigación como en el diseño y refinamiento de las propuestas sugeridas. El primer objetivo de investigación trata de comprender cuál es el estado actual del uso de feedback informado por LA dirigida por un instructor en los MOOCs. En consecuencia, se realizó una revisión sistemática de la literatura para comprender cómo las herramientas actuales respaldan la necesidad de desarrollar intervenciones personalizadas en los MOOC y comprender cuál es el impacto que tiene en los estudiantes o profesores. El segundo objetivo de investigación aborda la necesidad de ayudar al profesorado de MOOC en el diseño de feedback personalizado y contextualizado. Atendiendo a esta necesidad, esta Tesis Doctoral propone un marco conceptual denominado FeeD4Mi. FeeD4Mi consiste en una estructura conceptual de 5 dimensiones, acompañada de un proceso, un conjunto de catálogos, y un conjunto de recomendaciones para ayudar al profesorado en el diseño y provisión de LA- feedback en cursos MOOC. El tercer objetivo de investigación pretende facilitar el diseño y puesta en marcha del uso de feedback basado en LA en MOOCs. Para ello, esta tesis propone un conjunto de guías de diseño para incorporar FeeD4Mi en una herramienta web, incluyendo sus catálogos, procesos y recomendaciones. Dicha herramienta permite así la representación digital de estrategias de feedback para MOOCs, y su implementación automática o semiautomática durante la ejecución del curso. Se realizaron diversos estudios evaluativos que sirvieron para refinar y evaluar iterativamente FeeD4Mi (y sus componentes asociados) tanto con estudiantes como con instructores. A su vez, se evaluó la usabilidad, la carga de trabajo y la posible adopción digital del marco FeeD4Mi en su versión web (e-FeeD4Mi). Los resultados indicaron el valor añadido del marco para guiar a los instructores en el diseño y la provisión de intervenciones de retroalimentación informadas por LA en los MOOC. Además, los participantes destacaron la flexibilidad que les ofreció la herramienta, la posibilidad de automatizar estrategias de retroalimentación y los diseños interpretables por computadora que soportan la conexión con los indicadores de las plataformas MOOC. Finalmente, los resultados de los estudios señalaron nuevas direcciones de investigación en el área de la retroalimentación en los MOOC y en otros contextos educativos (enseñanza en línea o híbrida) donde brindar retroalimentación también es un desafío.Escuela de DoctoradoDoctorado en Investigación Transdisciplinar en Educació

International Conference on Computers in Education. Asia-Pacific Society for Computers in Education (29º. 2021)

Producción CientíficaDetecting learners who face problems in MOOCs usually poses difficulties due to the high instructor-learners ratio, the diversity of the population, and the asynchronous participation mode. Existing solutions mainly draw on self-reported problems in discussion forums and on dashboards displaying learners’ activity traces. However, these approaches cannot scale up easily or do not consider the course learning design. This paper presents a conceptual framework aimed at guiding MOOC instructors in the identification of potential learners' problems and indicators of such problems, considering the learning design of the course (e.g., types of activities, difficulty, etc.). An instrumental qualitative case study served for the evaluation and refinement of the framework. The results showed that the framework positively helped instructors to reflect on potential learners’ problems they had not considered beforehand, and to associate such problems with a set of indicators related to their learning designs.Junta de Castilla y León (grant VA257P18)Fondo Europeo de Desarrollo Regional - Agencia Estatal de Investigación (TIN2017-85179-C3-2-R, PID2020-112584RB-C32

EMOOCs 2019: Sixth European MOOCs Stakeholders Summit

Producción CientíficaAlthough Massive Open Online Courses (MOOCs) have been reported as an effective educational tool offering numerous opportunities in online learning, the high dropout rates and the lack of learners’ motivation are factors concerning researchers and instructors. The one-size-fits-all instructional approach that most courses follow, failing to address the individual needs of learners, has been seen as their weakest point. Recent efforts focus on the inclusion of active learning pedagogies in MOOCs to stimulate the interaction among the participants and to keep them engaged. However, taking into account that in these massive contexts the learners face several issues while trying to keep up with the course, the incorporation of active learning strategies may introduce additional problems to the learning process. This study explores the problems that learners experienced in a MOOC implementing collaboration and gamification strategies. As the results reveal, the introduction of collaborative learning activities can generate additional problems to learners and for that reason, a careful design and a proper scaffolding is needed in an early stage to overcome the problems that will occur. No significant problems were reported regarding the implementation of gamification elements.Ministerio de Ciencia, Innovación y Universidades - Fondo Europeo de Desarrollo Regional (projects TIN2017-85179-C3-2-R and TIN2014-53199-C3-2R)Junta de Castilla y León - Fondo Europeo de Desarrollo Regional (project VA257P18)Comisión Europea (project 588438-EPP-1-2017-1-EL-EPPKA2-KA

Learning Analytics Summer Institute (LASI Spain 2019)

Producción CientíficaMultimodal Learning Analytics (MMLA) uncovers the possibility to get a more holistic picture of a learning situation than traditional Learning Analytics, by triangulating learning evidence collected from multiple modalities. However, current MMLA solutions are complex and typically tailored to specific learning situations. In order to overcome this problem we are working towards an infrastructure that supports MMLA and can be adapted to different learning situations. As a first step in this direction, this paper analyzes four MMLA scenarios, abstracts their data processing activities and extracts a Data Value Chain to model the processing of multimodal evidence of learning. This helps us to reflect on the requirements needed for an infrastructure to support MMLA.European Union’s Horizon 2020 research and innovation programme (grant 669074)Ministerio de Ciencia, Innovación y Universidades - Fondo Europeo de Desarrollo Regional (projects TIN2017-85179-C3-2-R / TIN2014-53199- C3-2-R)Junta de Castilla y León - Fondo Europeo de Desarrollo Regional (project VA257P18)Comisión Europea (project 588438-EPP-1-2017-1-EL-EPPKA2- KA

EC-TEL 2022

Producción CientíficaThe provision of personalized and timely feedback can become challenging when shifting from face-to-face to online learning. Feedback is not only about providing support to students, but also about identifying when and which students need what kind of support. Usually, educators carry out such activities manually. However, the manual identification, personalization and provision of feedback might turn unmanageable, especially in large-scale environments. Previous works proposed the use of data-driven tools to automate the feedback provision with the active involvement of human agents in its design. Nevertheless, to the best of our knowledge, these tools do not guide instructors in the process of feedback design and sense-making of the data-driven information. This paper presents e-FeeD4Mi, a web-based tool developed to support instructors in the design and automatic enactment of feedback in multiple virtual learning environments. We developed e-FeeD4Mi following a Design-Based Research approach and its potential for adoption has been evaluated in two evaluation studies.Proyecto PID2020-112584RB-C32 financiado por MCIN/AEI/10.13039/50110001103

European Conference on Technology Enhanced Learning: Transforming Learning with Meaningful Technologies, EC-TEL 2019 (14º. 2019. Delft, The Netherlands)

Producción CientíficaLack of timely instructors' support when the learners are struggling with the course contents and activities is one frequent problem of MOOC learn-ers. The early identification of these learners could help instructors spend part of their limited time assisting them and avoiding potential dropouts. This paper pre-sents a MOOC case study that explores the behavior of learners who reported problems in private messages and discussion forums. The study aimed at the identification of parameters that might allow the detection of learners struggling with different course aspects. As the results suggested, the comparison of the learners’ activity traces reveals some common sequences that in the future could facilitate the identification of learners facing problems, even without reporting them. On the other hand, statistical analyses on learners’ behavior showed non-significant differences between the learners reporting putting their maximum ef-fort to overcome a problem before asking for help and the ones who did not.Ministerio de Ciencia, Innovación y Universidades - Fondo Europeo de Desarrollo Regional (projects TIN2017-85179-C3-2-R / TIN2014-53199- C3-2R)Junta de Castilla y León - Fondo Europeo de Desarrollo Regional (project grants BOCYL-D-07062018-6 / VA257P18)Comisión Europea (project 588438-EPP-1-2017-1-EL-EPPKA2-KA