Learning analytics support to teachers' design and orchestrating tasks

Background: Data-driven educational technology solutions have the potential to support teachers in different tasks, such as the designing and orchestration of collaborative learning activities. When designing, such solutions can improve teacher understanding of how learning designs impact student learning and behaviour; and guide them to refine and redesign future learning designs. When orchestrating educational scenarios, data-driven solutions can support teacher awareness of learner participation and progress and enhance real time classroom management. Objectives: The use of learning analytics (LA) can be considered a suitable approach to tackle both problems. However, it is unclear if the same LA indicators are able to satisfactorily support both the designing and orchestration of activities. This study aims to investigate the use of the same LA indicators for supporting multiple teacher tasks, that is, design, redesign and orchestration, as a gap in the existing literature that requires further exploration. Methods: In this study, first we refer to the previous work to study the use of different LA to support both tasks. Then we analyse the nature of the two tasks focusing on a case study that uses the same collaborative learning tool with LA to support both tasks. Implications: The study findings led to derive design considerations on LA support for teachers’ design and orchestrating tasks

Teacher Orchestration Load: What Is It and How Can We Lower the Burden?

This report provides details of a workshop conducted as an online pre-conference event during the ISLS Annual Meeting 2021. The workshop consisted of two main adjoining parts focusing on its two themes: orchestration load and related teacher support tools. The main findings of the workshop showed that (1) a definition of orchestration load requires further elaboration, (2) there are limited ways to measure this notion, and (3) attention should be paid to sharing orchestration load among other actors, e.g., students, intelligent agents, that may facilitate the simplification of activity regulation. Balancing orchestration load among multiple actors may lower the load experienced by the teachers in real-time in authentic educational contexts

The Orchestration of computer-supported collaboration scripts with learning analytics

Computer-supported collaborative learning (CSCL) creates avenues for productive collaboration between students. In CSCL, collaborative learning flow patterns (CLFPs) provide pedagogical rationale and constraints for structuring the collaboration process. While structured collaboration facilitates the design of favourable learning conditions, orchestration of collaboration becomes an important factor, as learner participation and real-world constraints can create deviations in real time. On the one hand, limited research has examined the orchestration challenges related to collaborative learning situations scripted according to CLFPs in authentic educational contexts to resolve collaboration at different scales. On the other hand, learning analytics (LA) can be used to provide proper technological tooling, infrastructure and support to orchestrate collaboration. To this end, this dissertation addresses the following research question: How can LA support orchestration mechanisms for scripted CSCL? To address this question, this dissertation first focuses on studying the orchestration challenges associated with scripted CSCL situations on small scales (in the classroom learning context) and large scales (in the distance learning context, specifically in massive open online courses [MOOCs]). In the classroom learning context, lack of teacher access to activity regulation mechanisms constituted a key challenge. In MOOCs, sustained student participation in multiple phases of the script was a primary challenge. The dissertation also focuses on studying the design of LA interventions that might address the orchestration challenges under examination. The proposed LA interventions range from human-in-control to machine-in-control in nature given the feasibility and regulation needs of the learning contexts under investigation. Following a design-based research (DBR) methodology, evaluation studies were conducted in naturalistic classrooms and in MOOCs to evaluate the effects of the proposed LA interventions and to understand the conditions for their successful implementation. The results of the evaluation studies conducted in the classroom context shed light on how teachers interpret LA data and how they action the resulting knowledge in authentic collaborative learning situations. In the distance learning context, the proposed interventions were critical in sustaining continuous flows of collaboration. The practical benefits and limitations of deploying LA solutions in real-world settings, as well as future research directions, are outlined.El aprendizaje colaborativo asistido por ordenador (CSCL) ofrece oportunidades para la colaboración productiva entre estudiantes. En CSCL, los patrones de flujo de aprendizaje colaborativo (CLFP) proporcionan un fundamento pedagógico y restricciones para estructurar el proceso de colaboración. Si bien la colaboración estructurada facilita el diseño de condiciones de aprendizaje favorables, la orquestación de dicha colaboración estructurada se convierte en un factor importante, ya que la participación del alumno y los condicionantes del mundo real pueden crear desviaciones en el momento de su realización. Por un lado, existe una investigación limitada sobre los desafíos de la orquestación de aprendizaje colaborativo guiado según los CLFP en contextos educativos auténticos a diferentes escalas. Por otro lado, la analítica del aprendizaje (LA) se puede utilizar para proporcionar las herramientas tecnológicas, la infraestructura y el apoyo adecuados para orquestar la colaboración. Con este fin, esta tesis doctoral plantea la siguiente pregunta de investigación: ¿Cómo puede LA apoyar los mecanismos de orquestación de guiones de CSCL? Para abordar esta pregunta, la tesis doctoral se centra, primero, en estudiar los desafíos de la orquestación en situaciones CSCL guiadas a pequeña escala (en el contexto del aula) y a gran escala (en el contexto de aprendizaje a distancia, específicamente en cursos masivos abiertos en línea [MOOC]). En el contexto del aula, un reto imporante es la falta de acceso de los docentes a los mecanismos de regulación de la actividad. En los MOOC, el reto principal es sostener la participación de los estudiantes a lo largo de las diversas fases del guión. La tesis doctoral también se centra en estudiar el diseño de intervenciones de LA que podrían abordar los retos de orquestación detectados. Dadas las necesidades de viabilidad y regulación de los contextos de aprendizaje investigados, las intervenciones de LA propuestas van desde acciones automáticas donde la “máquina está en control” a intervenciones que implican “control por humanos”. Siguiendo una metodología de investigación basada en el diseño (DBR), se han realizado estudios en aulas y en MOOCs para evaluar los efectos de las intervenciones de LA propuestas y comprender las condiciones para su buena implementación. Los resultados de la evaluación realizada en el contexto del aula arrojan luz sobre cómo los profesores interpretan los datos de LA y cómo actúan en consecuencia en situaciones auténticas de aprendizaje colaborativo. En el contexto de la educación a distancia, las intervenciones propuestas fueron fundamentales para mantener flujos continuos de colaboración. La tesis docotral describe los beneficios prácticos y las limitaciones a la hora de implementar soluciones de LA en entornos reales, así como las direcciones de investigación futuras

Synergies between humans and machines to support the orchestration of CSCL scripts at different scales

Comunicació presentada a: 14th International Conference on Computer-Supported Collaborative Learning (CSCL) celebrada del 7 a l'11 de juny de 2021 a Bochum, Alemanya.This study presents the orchestration challenges associated with scripted collaborative learning situations at different scales and how different Learning Analytics (LA) interventions may facilitate to address those issues. The proposed LA interventions were characterised as machine-in-control, human-in-control and hybrid approaches given different agents in charge of orchestration actions. A framing of the proposed LA interventions is presented considering also the different scales within which those interventions were deployed, in an attempt to seek the balance between different types of interventions.This work has been partially funded by FEDER, the National Research Agency of the Spanish Ministry of Science, Innovations and Universities MDM-2015-0502, TIN2017-85179-C3-3-R. Davinia Hernández-Leo acknowledges the support by ICREA under the ICREA Academia programme

Understanding the well-being impact of a computer-supported collaborative learning tool: the case of PyramidApp

Comunicació presentada a: 16th European Conference on Technology Enhanced Learning, EC-TEL celebrat del 20 al 24 de setembre de 2021 de manera virtual.The global efforts toward evaluating the impact of the use of data-driven technologies on humans’ well-being continue to establish societal guidelines for such systems to remain human-centric, serving humanity’s values and safeguarding well-being. In this paper, we apply the first activity of IEEE P7010 recommended practice, a methodology and a set of metrics, to understand the well-being impact of a web-based tool (PyramidApp) that allows teachers to design and deploy Pyramid-pattern based collaborative learning activities in classroom learning scenarios. The tool’s creators who are learning technology researchers (n = 2) and a sample of the tool’s users and stakeholders who are undergraduate students (n = 11), master students (n = 14) and instructors (n = 2) are engaged in surveys and interviews to investigate the tool’s well-being impact by reflecting on well-being indicators distributed to multiple well-being domains. The findings discuss possible impacts of the tool on the well-being domains of life satisfaction, affect, psychological state, community, education, government, human settlement and work. The creators also share views about the extent to which the use of IEEE P7010 increases their awareness of the intended and unintended impacts of their tool on well-being.This work has been partially funded by the EU Regional Development Fund and the National Research Agency of the Spanish Ministry of Science and Innovation under project grants TIN2017-85179-C3-3-R, PID2020-112584RB-C33. D. Hernández-Leo (Serra Húnter) acknowledges the support by ICREA under the ICREA Academia program. E. Hakami acknowledges the grant by Jazan University, Saudi Arabia

Flagging in teacher-facing orchestration dashboards: factors affecting its use in Pyramid CSCL debriefing

Comunicació presentada a: ICALT 2022 International Conference on Advanced Learning Technologies, celebrat del 1 al 4 de juliol de 2022 a Bucarest, Rumania.Teacher-led debriefing has the potential to positively affect learning gains when conducted at the end of collaborative learning activities. In order for debriefing to be effective, the teacher needs to base it on the learner’s process during the activity. Research in the field of Computer-Supported Collaborative Learning (CSCL) is proposing teaching-facing dashboards as tools that facilitate the monitoring and orchestration of activities. However, research has paid less attention to how these dashboards can support debriefing. We explore how adding a “flagging” feature to a CSCL orchestration dashboard can support debriefing by reporting a qualitative preliminary study in which the flagging feature was used during a Pyramid CSCL script activity. Results indicate that the dashboard interface design, number of student responses, number of errors in student responses, and whether student responses meet the teachers expectations most influence the use and utility of the feature. Additionally, we identified avenues for improving and extending the design of the feature.This work has been partially funded by the Ministry of Science and Innovation and the National Research Agency (PID2020-112584RBC33/MICIN/AEI/10.13039/501100011033) and ICREA under the ICREA Academia programme (D. Hernandez-Leo, Serra Hunter)

Towards estimating classroom orchestration load using physiological and self-perception measures

Comunicació presentada a: 14th International Conference on Computer-Supported Collaborative Learning (CSCL) celebrada del 7 a l'11 de juny de 2021 a Bochum, Alemanya.This poster presents the exploration of a method to estimate the notion of orchestration load using physiological measures in triangulation with self-perception measures in the classroom computer-supported collaborative learning (CSCL) context. Details of a pilot study conducted in which a teacher orchestrated CSCL activities under different supporting conditions are presented. Different facets of the orchestration load were disentangled in light of the study findings.This work has been partially funded by FEDER, the National Research Agency of the Spanish Ministry of Science, Innovations and Universities MDM-2015-0502, TIN2017-85179-C3-3-R. Davinia Hernández-Leo acknowledges the support by ICREA under the ICREA Academia programme

Studying collaboration dynamics in physical learning spaces: considering the temporal perspective through epistemic network analysis

The role of the learning space is especially relevant in the application of active pedagogies, for example those involving collaborative activities. However, there is limited evidence informing learning design on the potential effects of collaborative learning spaces. In particular, there is a lack of studies generating evidence derived from temporal analyses of the influence of learning spaces on the collaborative learning process. The temporal analysis perspective has been shown to be essential in the analysis of collaboration processes, as it reveals the relationships between students’ actions. The aim of this study is to explore the potential of a temporal perspective to broaden understanding of the effects of table shape on collaboration when different group sizes and genders are considered. On-task actions such as explanation, discussion, non-verbal interaction, and interaction with physical artefacts were observed while students were engaged in engineering design tasks. Results suggest that table shape influences student behaviour when taking into account different group sizes and different genders

Towards teacher orchestration load-aware teacher-facing dashboards

Comunicació presentada a: 10th International Learning and Analytics Conference (LAK 2020), el 24 de març de 2020, virtualment.In this workshop paper, we report a study conducted to investigate the use of tracking technologies to measure the teachers’ orchestration load when conducting colocated collaborative learning activities. We distinguish the orchestration load experienced by the teachers in the absence and presence of teacher supporting tools, i.e. teacher-facing dashboards. Electrodermal activity (EDA) sensor and other multimodal data including observations, log data and subjective responses to questionnaires have been collected to measure the teachers’ orchestration load in authentic collaborative learning scenarios. This workshop paper presents the study context, quantitative and qualitative data collection process undertaken and other considerations in detail.This work has been partially funded by FEDER, the national research agency of the Spanish Ministry of Science, Innovations and Universities MDM-2015-0502, TIN2017-85179-C3-3-R

Intelligent group formation in computer supported collaborative learning scripts

Comunicació presentada a: ICALT2017, celebrat del 3 al 7 de juliol a Timisoara, Romania.Well-structured collaborative learning groups scripted based on Collaborative Learning Flow Patterns (CLFPs) often result in successful collaborative learning outcomes. Formulation of such learner groups based on instructor defined criteria promises potentially effective performance of participating students. However, forming student groups manually based on multiple criteria often fails due to its complexity and the time limitations of practitioners. Hence, an intelligent assistance which supports adaptive collaboration scripting based on instructor defined criteria, while adhering to CLFPs is presented. Constraint Optimization techniques have been used for learner group formation and preliminary tests revealed that the proposed approach could be utilized when formulating student groups while satisfying team formation criteria.This research is funded by Spanish Ministry of Economy and Competitiveness (TIN2014-53199-C3-3-R, MDM-2015- 0502) and RecerCaixa (COT Project). Prof. H. Spoelstra from Open University of the Netherlands, Dr. C. Burt from The University of Melbourne and Prof. H. Ramalhinho from Universitat Pompeu Fabra are gratefully acknowledged