Urgency Analysis of Learners’ Comments: An Automated Intervention Priority Model for MOOC

Recently, the growing number of learners in Massive Open Online Course (MOOC) environments generate a vast amount of online comments via social interactions, general discussions, expressing feelings or asking for help. Concomitantly, learner dropout, at any time during MOOC courses, is very high, whilst the number of learners completing (completers) is low. Urgent intervention and attention may alleviate this problem. Analysing and mining learner comments is a fundamental step towards understanding their need for intervention from instructors. Here, we explore a dataset from a FutureLearn MOOC course. We find that (1) learners who write many comments that need urgent intervention tend to write many comments, in general. (2) The motivation to access more steps (i.e., learning resources) is higher in learners without many comments needing intervention, than that of learners needing intervention. (3) Learners who have many comments that need intervention are less likely to complete the course (13%). Therefore, we propose a new priority model for the urgency of intervention built on learner histories – past urgency, sentiment analysis and step access

Serendipitous Gains of Explaining a Classifier - Artificial versus Human Performance and Annotator Support in an Urgent Instructor-Intervention Model for MOOCs

Determining when instructor intervention is needed, based on learners’ comments and their urgency in massive open online course (MOOC) environments, is a known challenge. To solve this challenge, prior art used autonomous machine learning (ML) models. These models are described as having a "black-box" nature, and their output is incomprehensible to humans. This paper shows how to apply eXplainable Artificial Intelligence (XAI) techniques to interpret a MOOC intervention model for urgent comments detection. As comments were selected from the MOOC course and annotated using human experts, we additionally study the confidence between annotators (annotator agreement confidence), versus an estimate of the class score of making a decision via ML, to support intervention decision. Serendipitously, we show, for the first time, that XAI can be further used to support annotators creating high-quality, gold standard datasets for urgent intervention

Mapeamento automático de rotas para vias expressas de ônibus: uma abordagem com algoritmos genéticos com ênfase na cidade de Boa Vista - RR

Apesar dos benefícios de uma via expressa de ônibus, o trabalho de engenharia para mapear quais vias devem compor a via expressa é complexo, já que existem exponenciais possibilidades de combinação. Note que computacionalmente falando, uma cidade pode ser vista como um grafo direcionado G e uma via expressa pode ser representada por um subgrafo G', onde G' ⊆ G. Dessa forma, a tarefa de selecionar as vias que irão compor uma via expressa pode ser representada por uma variação do problema do caminho com ciclo de peso máximo, onde esse caminho seria um ciclo conexo com maior quantidade de vias dentro dos parâmetros do grafo da cidade. Nesse sentido, o presente estudo tem o propósito de selecionar rotas que formem ciclos que maximizem o número de vias com alta demanda. Além disso, esse ciclo deve impreterivelmente passar por um terminal de ônibus da cidade, já que esse é o ponto de convergência dos ônibus. Como prova de conceito, optou-se por usar os dados da cidade de Boa Vista - RR. Visto que algoritmos genéticos apresentam alternativas viáveis para problemas de aproximação, optou-se por usar essa técnica para automatizar o processo de seleção e otimização da topologia de rotas para via expressas de ônibus. Os melhores resultados foram atingidos utilizando o algoritmo genético NSGA-II, gerando uma rota que cobre 120 arestas do grafo representado por avenidas da cidade. A convergência do AG foi atingida após 500 gerações. Por fim, destaca-se ainda que o método pode ser generalizado para outros cenários de experimentação, potencialmente sendo aplicado em outros centros urbanos com uma rede de avenidas e pelo menos um terminal de ônibus

Mapeamento automático de rotas para vias expressas de ônibus: uma abordagem com algoritmos genéticos com ênfase na cidade de Boa Vista - RR

Apesar dos benefícios de uma via expressa de ônibus, o trabalho de engenharia para mapear quais vias devem compor a via expressa é complexo, já que existem exponenciais possibilidades de combinação. Note que computacionalmente falando, uma cidade pode ser vista como um grafo direcionado G e uma via expressa pode ser representada por um subgrafo G', onde G' ⊆ G. Dessa forma, a tarefa de selecionar as vias que irão compor uma via expressa pode ser representada por uma variação do problema do caminho com ciclo de peso máximo, onde esse caminho seria um ciclo conexo com maior quantidade de vias dentro dos parâmetros do grafo da cidade. Nesse sentido, o presente estudo tem o propósito de selecionar rotas que formem ciclos que maximizem o número de vias com alta demanda. Além disso, esse ciclo deve impreterivelmente passar por um terminal de ônibus da cidade, já que esse é o ponto de convergência dos ônibus. Como prova de conceito, optou-se por usar os dados da cidade de Boa Vista - RR. Visto que algoritmos genéticos apresentam alternativas viáveis para problemas de aproximação, optou-se por usar essa técnica para automatizar o processo de seleção e otimização da topologia de rotas para via expressas de ônibus. Os melhores resultados foram atingidos utilizando o algoritmo genético NSGA-II, gerando uma rota que cobre 120 arestas do grafo representado por avenidas da cidade. A convergência do AG foi atingida após 500 gerações. Por fim, destaca-se ainda que o método pode ser generalizado para outros cenários de experimentação, potencialmente sendo aplicado em outros centros urbanos com uma rede de avenidas e pelo menos um terminal de ônibus

Solving the imbalanced data issue: automatic urgency detection for instructor assistance in MOOC discussion forums

In MOOCs, identifying urgent comments on discussion forums is an ongoing challenge. Whilst urgent comments require immediate reactions from instructors, to improve interaction with their learners, and potentially reducing drop-out rates—the task is difficult, as truly urgent comments are rare. From a data analytics perspective, this represents a highly unbalanced (sparse) dataset. Here, we aim to automate the urgent comments identification process, based on fine-grained learner modelling—to be used for automatic recommendations to instructors. To showcase and compare these models, we apply them to the first gold standard dataset for Urgent iNstructor InTErvention (UNITE), which we created by labelling FutureLearn MOOC data. We implement both benchmark shallow classifiers and deep learning. Importantly, we not only compare, for the first time for the unbalanced problem, several data balancing techniques, comprising text augmentation, text augmentation with undersampling, and undersampling, but also propose several new pipelines for combining different augmenters for text augmentation. Results show that models with undersampling can predict most urgent cases; and 3X augmentation + undersampling usually attains the best performance. We additionally validate the best models via a generic benchmark dataset (Stanford). As a case study, we showcase how the naïve Bayes with count vector can adaptively support instructors in answering learner questions/comments, potentially saving time or increasing efficiency in supporting learners. Finally, we show that the errors from the classifier mirrors the disagreements between annotators. Thus, our proposed algorithms perform at least as well as a ‘super-diligent’ human instructor (with the time to consider all comments)

Education in the age of Generative AI: Context and Recent Developments

With the emergence of generative artificial intelligence, an increasing number of individuals and organizations have begun exploring its potential to enhance productivity and improve product quality across various sectors. The field of education is no exception. However, it is vital to notice that artificial intelligence adoption in education dates back to the 1960s. In light of this historical context, this white paper serves as the inaugural piece in a four-part series that elucidates the role of AI in education. The series delves into topics such as its potential, successful applications, limitations, ethical considerations, and future trends. This initial article provides a comprehensive overview of the field, highlighting the recent developments within the generative artificial intelligence sphere

The engage taxonomy: SDT-based measurable engagement indicators for MOOCs and their evaluation

Massive Online Open Course (MOOC) platforms are considered a distinctive way to deliver a modern educational experience, open to a worldwide public. However, student engagement in MOOCs is a less explored area, although it is known that MOOCs suffer from one of the highest dropout rates within learning environments in general, and in e-learning in particular. A special challenge in this area is finding early, measurable indicators of engagement. This paper tackles this issue with a unique blend of data analytics and NLP and machine learning techniques together with a solid foundation in psychological theories. Importantly, we show for the first time how Self-Determination Theory (SDT) can be mapped onto concrete features extracted from tracking student behaviour on MOOCs. We map the dimensions of Autonomy, Relatedness and Competence, leading to methods to characterise engaged and disengaged MOOC student behaviours, and exploring what triggers and promotes MOOC students’ interest and engagement. The paper further contributes by building the Engage Taxonomy, the first taxonomy of MOOC engagement tracking parameters, mapped over 4 engagement theories: SDT, Drive, ET, Process of Engagement. Moreover, we define and analyse students’ engagement tracking, with a larger than usual body of content (6 MOOC courses from two different universities with 26 runs spanning between 2013 and 2018) and students (initially around 218.235). Importantly, the paper also serves as the first large-scale evaluation of the SDT theory itself, providing a blueprint for large-scale theory evaluation. It also provides for the first-time metrics for measurable engagement in MOOCs, including specific measures for Autonomy, Relatedness and Competence; it evaluates these based on existing (and expanded) measures of success in MOOCs: Completion rate, Correct Answer ratio and Reply ratio. In addition, to further illustrate the use of the proposed SDT metrics, this study is the first to use SDT constructs extracted from the first week, to predict active and non-active students in the following week

Automatic subject-based contextualisation of programming assignment lists.

As programming must be learned by doing, introductory programming course learners need to solve many problems, e.g., on systems such as ’Online Judges’. However, as such courses are often compulsory for non-Computer Science (nonCS) undergraduates, this may cause difficulties to learners that do not have the typical intrinsic motivation for programming as CS students do. In this sense, contextualised assignment lists, with programming problems related to the students’ major, could enhance engagement in the learning process. Thus, students would solve programming problems related to their academic context, improving their comprehension of the applicability and importance of programming. Nonetheless, preparing these contextually personalised programming assignments for classes for different courses is really laborious and would increase considerably the instructors’/monitors’ workload. Thus, this work aims, for the first time, to the best of our knowledge, to automatically classify the programming assignments in Online Judges based on students’ academic contexts by proposing a new context taxonomy, as well as a comprehensive pipeline evaluation methodology of cutting edge competitive Natural Language Processing (NLP). Our comprehensive methodology pipeline allows for comparing state of the art data augmentation, classifiers, beside NLP approaches. The context taxonomy created contains 23 subject matters related to the non-CS majors, representing thus a challenging multi-classification problem. We show how even on this problem, our comprehensive pipeline evaluation methodology allows us to achieve an accuracy of 95.2%, which makes it possible to automatically create contextually personalised program assignments for non-CS with a minimal error rate (4.8%)

A Good Classifier is Not Enough: A XAI Approach for Urgent Instructor-Intervention Models in MOOCs

Deciding upon instructor intervention based on learners’ comments that need an urgent response in MOOC environments is a known challenge. The best solutions proposed used automatic machine learning (ML) models to predict the urgency. These are ‘black-box’-es, with results opaque to humans. EXplainable artificial intelligence (XAI) is aiming to understand these, to enhance trust in artificial intelligence (AI)-based decision-making. We propose to apply XAI techniques to interpret a MOOC intervention model, by analysing learner comments. We show how pairing a good predictor with XAI results and especially colour-coded visualisation could be used to support instructors making decisions on urgent intervention

Early Performance Prediction for CS1 Course Students using a Combination of Machine Learning and an Evolutionary Algorithm

Many researchers have started extracting student behaviour by cleaning data collected from web environments and using it as features in machine learning (ML) models. Using log data collected from an online judge, we have compiled a set of successful features correlated with the student grade and applying them on a database representing 486 CS1 students. We used this set of features in ML pipelines which were optimised, featuring a combination of an automated approach with an evolutionary algorithm and hyperparameter-tuning with random search. As a result, we achieved an accuracy of 75.55%, using data from only the first two weeks to predict the student final grades. We show how our pipeline outperforms state-of-the-art work on similar scenarios