Towards Student Engagement Analytics: Applying Machine Learning to Student Posts in Online Lecture Videos

The use of online learning environments in higher education is becoming ever more prevalent with the inception of MOOCs (Massive Open Online Courses) and the increase in online and flipped courses at universities. Although the online systems used to deliver course content make education more accessible, students often express frustration with the lack of assistance during online lecture videos. Instructors express concern that students are not engaging with the course material in online environments, and rely on affordances within these systems to figure out what students are doing. With many online learning environments storing log data about students usage of these systems, research into learning analytics, the measurement, collection, analysis, and reporting data about learning and their contexts, can help inform instructors about student learning in the online context. This thesis aims to lay the groundwork for learning analytics that provide instructors high-level student engagement data in online learning environments. Recent research has shown that instructors using these systems are concerned about their lack of awareness about student engagement, and educational psychology has shown that engagement is necessary for student success. Specifically, this thesis explores the feasibility of applying machine learning to categorize student posts by their level of engagement. These engagement categories are derived from the ICAP framework, which categorizes overt student behaviors into four tiers of engagement: Interactive, Constructive, Active, and Passive. Contributions include showing what natural language features are most indicative of engagement, exploring whether this machine learning method can be generalized to many courses, and using previous research to develop mockups of what analytics using data from this machine learning method might look like

Challenges for IT-Enabled Formative Assessment of Complex 21st Century Skills

In this article, we identify and examine opportunities for formative assessment provided by information technologies (IT) and the challenges which these opportunities present. We address some of these challenges by examining key aspects of assessment processes that can be facilitated by IT: datafication of learning; feedback and scaffolding; peer assessment and peer feedback. We then consider how these processes may be applied in relation to the assessment of horizontal, general complex 21st century skills (21st CS), which are still proving challenging to incorporate into curricula as well as to assess. 21st CS such as creativity, complex problem solving, communication, collaboration and self-regulated learning contain complex constructs incorporating motivational and affective components. Our analysis has enabled us to make recommendations for policy, practice and further research. While there is currently much interest in and some progress towards the development of learning/assessment analytics for assessing 21st CS, the complexity of assessing such skills, together with the need to include affective aspects means that using IT-enabled techniques will need to be combined with more traditional methods of teacher assessment as well as peer assessment for some time to come. Therefore learners, teachers and school leaders must learn how to manage the greater variety of sorts and sources of feedback including resolving tensions of inconsistent feedback from different sources

Quantified Self Analytics Tools for Self-regulated Learning with myPAL

One of the major challenges in higher education is developing self-regulation skills for lifelong learning. We address this challenge within the myPAL project, in medical education context, utilising the vast amount of student assessment and feedback data collected throughout the programme. The underlying principle of myPAL is Quantified Self -- the use of personal data to enable students to become lifelong learners. myPAL is facilitating this with learning analytics combined with interactive nudges. This paper reviews the state of the art in Quantified Self analytics tools to identify what approaches can be adopted in myPAL and what gaps require further research. The paper contributes to awareness and reflection in technology-enhanced learning by: (i) identifying requirements for intelligent personal adaptive learning systems that foster self-regulation (using myPAL as an example); (ii) analysing the state of the art in text analytics and visualisation related to Quantified Self for self-regulated learning; and (iii) identifying open issues and suggesting possible ways to address them

Providing Intelligent and Adaptive Support in Concept Map-based Learning Environments

abstract: Concept maps are commonly used knowledge visualization tools and have been shown to have a positive impact on learning. The main drawbacks of concept mapping are the requirement of training, and lack of feedback support. Thus, prior research has attempted to provide support and feedback in concept mapping, such as by developing computer-based concept mapping tools, offering starting templates and navigational supports, as well as providing automated feedback. Although these approaches have achieved promising results, there are still challenges that remain to be solved. For example, there is a need to create a concept mapping system that reduces the extraneous effort of editing a concept map while encouraging more cognitively beneficial behaviors. Also, there is little understanding of the cognitive process during concept mapping. What’s more, current feedback mechanisms in concept mapping only focus on the outcome of the map, instead of the learning process. This thesis work strives to solve the fundamental research question: How to leverage computer technologies to intelligently support concept mapping to promote meaningful learning? To approach this research question, I first present an intelligent concept mapping system, MindDot, that supports concept mapping via innovative integration of two features, hyperlink navigation, and expert template. The system reduces the effort of creating and modifying concept maps while encouraging beneficial activities such as comparing related concepts and establishing relationships among them. I then present the comparative strategy metric that modes student learning by evaluating behavioral patterns and learning strategies. Lastly, I develop an adaptive feedback system that provides immediate diagnostic feedback in response to both the key learning behaviors during concept mapping and the correctness and completeness of the created maps. Empirical evaluations indicated that the integrated navigational and template support in MindDot fostered effective learning behaviors and facilitating learning achievements. The comparative strategy model was shown to be highly representative of learning characteristics such as motivation, engagement, misconceptions, and predicted learning results. The feedback tutor also demonstrated positive impacts on supporting learning and assisting the development of effective learning strategies that prepare learners for future learning. This dissertation contributes to the field of supporting concept mapping with designs of technological affordances, a process-based student model, an adaptive feedback tutor, empirical evaluations of these proposed innovations, and implications for future support in concept mapping.Dissertation/ThesisDoctoral Dissertation Computer Science 201

Involve Me! Using Developmentally Appropriate Practices to Support a Rigorous Kindergarten Program: The Effects on Engagement and Attitude

abstract: Chi and Wylie’s (2014) Interactive Constructive Active Passive Framework (ICAP) was used as the foundation of a teacher led intervention using small group instruction with manipulatives during mathematics instruction to provide developmentally appropriate instruction to kindergarten students in a rigorous academic program. This action research mixed-methods study was conducted in a full-day self-contained kindergarten classroom to ascertain the effects of this mathematics instruction method on students’ levels of engagement and attitudes. Over the course of six months, twenty mathematics lessons were recorded to gather data for the study. Quantitative data included measuring time-on-task, teacher behaviors ICAP level, student behaviors ICAP level, as well as a Student Attitude Survey that was conducted at the conclusion of the study. The Student Attitude Survey was presented in a modified Likert Scale format due to the age and reading ability of the participants. Qualitative data was gathered in the form of lesson transcripts. Twenty-two students and one classroom teacher participated in the study. Students ranged in age from five to six years old, and eleven participants (50%) were male. The results of the study showed that the use of small group hands-on instruction in mathematics had a positive effect on student engagement based on students’ time-on-task during the activity, as well as positive student attitudes toward mathematics as indicated on the Student Attitude Survey. Lesson transcripts and both teacher and student ICAP rubrics provided further support for the innovation.Dissertation/ThesisDoctoral Dissertation Leadership and Innovation 201

A Longitudinal Examination of Student Approaches to Learning and Metacognition

Student Approaches to Learning (SAL) mainly consists of two contradictory approaches (surface and deep learning) to learning that have been extensively studied in educational research. Metacognition, which refers to the process of thinking about one’s thinking, has been shown to play a crucial role in helping students shift from a surface to a deep approach to learning. The current study collected data using two questionnaires (RSPQ-2F&amp; MAI) from 1329 students. Both metacognition and learning approaches showed medium correlations and an effect of the year of study. A crossed-lagged model shows no effect of deep learning on metacognitive knowledge or regulation, although this does increase significantly over time. Overall, the study’s findings suggest a complex yet clear relationship between student learning approaches and their final grade outcomes. Students will lean towards more surface learning as their (perceived) workload increases and assessments become more challenging. These findings suggest that teachers and policy makers should seek ways to increase deep learning methods, possibly using metacognitive skills training.</jats:p

From ABC’s to 3P’s (and a couple of T’s): Exploring Factors Affecting Student Learning in Higher Education and the Need for an Updated Educational Framework

Higher education is a challenging landscape to investigate, as it encompasses a diverse range of student backgrounds and requires a focus on developing metacognitive thinking skills, creating effective learning environments, and promoting student engagement and motivation. Over the years, various frameworks have been developed to describe the learning experiences and processes of higher education students. However, the rapidly changing nature of the 21st century demands that educational researchers and universities re-evaluate the established teaching and learning frameworks. This has become especially clear during the COVID-19 pandemic, where the sudden shift to online learning has highlighted the need for flexible and adaptable approaches to education. As we move forward, it is essential to continue to examine and develop effective frameworks that can support students in navigating the challenges of higher education and beyond. Three of the studies presented as part of this PhD have been published as journal papers or book chapters, while the other two are under review. In addition to the main chapter publications, several other publications have been submitted to various international journals evaluating current frameworks and providing suggestions for alternative interventions. This PhD aims to explore several factors that influence student learning promoting a revision to the well-established educational framework of Biggs’ (1993) 3P model. Despite its age this model continues to be widely used in higher education, emphasising the importance of three factors: presage, process, and product when considering the factors that affect student outcomes. The aim and the rationale of this PhD, along with a broader discussion on the widely used Higher Education frameworks are presented in the introduction, while an adapted model (3P2T) is proposed in the discussion. Each of the studies presented is related to either one or a combination of Biggs’ (1993) three factors affecting student outcomes. Specifically, study one explores the impact of prior learning and knowledge on student academic performance. This study explored the effects of prior knowledge on first-year Psychology students' academic achievements through ordinal regressions and correlations. In order to explore the role of digital learning tools in Higher Education and their potential benefits during disruptive events for learning (i.e., industrial strikes), study two compares students' Virtual Learning Environment (VLE) behaviour across three consecutive first-year undergraduate Psychology cohorts, in which one year was impacted by industrial strikes. Next, study three, empirically explores the relationship between students' learning approaches, metacognition, and academic performance using longitudinally collected data. Results suggest that a further investigation of these and other factors affecting student outcomes should be explored. Study four does exactly this and presents a new questionnaire, adapting items from the widely used Motivated Strategies for Learning Questionnaire (MSLQ; Pintrich et al., 1991) including three new key themes of course utility, procrastination, and use of diverse sources and test anxiety. Finally, the last study, study five, qualitatively explores the multiple transitions that students undergo as they move from secondary to tertiary education (University), including changes in education, student socialisation, and emotions. Students arrive with expectations about their University experience, based on their understanding of what it means to study at this level, and the interviews explore how these expectations manifest and change throughout their degree. The study used thematic analysis to identify five key themes that shape students' experiences: prior experience, adjustment to university, staff relationships, the experience of studying, and future plans. Overall, the five studies employed various qualitative, quantitative, and analytical research methods in order to investigate how current Higher Education changes may affect student learning experience across different stages over their degree. The main findings of this research project argue that the need of an updated version of well-established educational frameworks (i.e., Biggs’ 1993 3P model) is necessary. Such necessity is driven by the changes in University learning processes, student expectations and engagement, use of learning technologies, and the demographic pool now entering Higher Education. The research findings suggest that educational policymakers and University teachers should consider factors such as digital learning tools, diverse populations, and new teaching and learning methodologies to ensure the continuation of educational framework relevance and usefulness. Applying this consideration will guide the design and delivery of Higher Education, allowing teachers to tailor their approaches to meet the needs of ever growing and diversifying range of students

An Investigation of Student Cognitive Engagement in the STEM Classroom—A Compilation of Faculty and Student Perspectives

Student engagement has been the focus of much engineering education research, in large part due to its ties to student learning. Widely considered to be a meta-construct, student engagement is often broken down into behavioral, emotional, and cognitive components. Reasons for ongoing research on student cognitive engagement are twofold: educators often have difficulty assessing students’ cognitive engagement due to its inherently unobservable qualities, and cognitive engagement has been seen as a predictor of deep learning and success in students. In the body of works that make up this dissertation, I researched student cognitive engagement from multiple vantage points. First, I worked to develop an instrument to measure student cognitive engagement. The outcome was the Student Course Cognitive Engagement Instrument (SCCEI), which showed evidence of validly measuring five distinct modes of cognitive engagement. The SCCEI aimed to allow instructors to evaluate the sophistication of the student cognitive engagement in their class. To support this aim, I analyzed student interviews to deepen my understanding of how students interpret items probing cognitive engagement. This allowed the SCCEI to be modified to more accurately represent the realities of students’ engagement experience. Included was the measurement of engagement inside and outside the classroom; evidence was found that students report statistically different on their modes of engagement depending on the context (inside or outside the classroom). I also interviewed students to gain a more holistic understanding of their engagement. These high-achieving, upper division engineering students provided insight on the major factors that shaped their engagement throughout college. Finally, I utilized our instrument, the SCCEI, in tandem with an instrument to measure instructional practices to explore the relationship between the two constructs. The results of these works broadly suggest that student cognitive engagement can indeed be measured by self-report, and instructors play a critical role in shaping their students’ engagement. Furthermore, findings suggest that instructional practices are indeed correlated with modes of student cognitive engagement. Compiled, this body of work adds to the growing literature on student cognitive engagement, and how the engineering discipline can continue to move towards the betterment of students