MOOCs Meet Measurement Theory: A Topic-Modelling Approach

This paper adapts topic models to the psychometric testing of MOOC students based on their online forum postings. Measurement theory from education and psychology provides statistical models for quantifying a person's attainment of intangible attributes such as attitudes, abilities or intelligence. Such models infer latent skill levels by relating them to individuals' observed responses on a series of items such as quiz questions. The set of items can be used to measure a latent skill if individuals' responses on them conform to a Guttman scale. Such well-scaled items differentiate between individuals and inferred levels span the entire range from most basic to the advanced. In practice, education researchers manually devise items (quiz questions) while optimising well-scaled conformance. Due to the costly nature and expert requirements of this process, psychometric testing has found limited use in everyday teaching. We aim to develop usable measurement models for highly-instrumented MOOC delivery platforms, by using participation in automatically-extracted online forum topics as items. The challenge is to formalise the Guttman scale educational constraint and incorporate it into topic models. To favour topics that automatically conform to a Guttman scale, we introduce a novel regularisation into non-negative matrix factorisation-based topic modelling. We demonstrate the suitability of our approach with both quantitative experiments on three Coursera MOOCs, and with a qualitative survey of topic interpretability on two MOOCs by domain expert interviews.Comment: 12 pages, 9 figures; accepted into AAAI'201

Predicting the Need for Urgent Instructor Intervention in MOOC Environments

In recent years, massive open online courses (MOOCs) have become universal knowledge resources and arguably one of the most exciting innovations in e-learning environments. MOOC platforms comprise numerous courses covering a wide range of subjects and domains. Thousands of learners around the world enrol on these online platforms to satisfy their learning needs (mostly) free of charge. However, the retention rates of MOOC courses (i.e., those who successfully complete a course of study) are low (around 10% on average); dropout rates tend to be very high (around 90%). The principal channel via which MOOC learners can communicate their difficulties with the learning content and ask for assistance from instructors is by posting in a dedicated MOOC forum. Importantly, in the case of learners who are suffering from burnout or stress, some of these posts require urgent intervention. Given the above, urgent instructor intervention regarding learner requests for assistance via posts made on MOOC forums has become an important topic for research among researchers. Timely intervention by MOOC instructors may mitigate dropout issues and make the difference between a learner dropping out or staying on a course. However, due to the typically extremely high learner-to-instructor ratio in MOOCs and the often-huge numbers of posts on forums, while truly urgent posts are rare, managing them can be very challenging –– if not sometimes impossible. Instructors can find it challenging to monitor all existing posts and identify which posts require immediate intervention to help learners, encourage retention, and reduce the current high dropout rates. The main objective of this research project, therefore, was thus to mine and analyse learners’ MOOC posts as a fundamental step towards understanding their need for instructor intervention. To achieve this, the researcher proposed and built comprehensive classification models to predict the need for instructor intervention. The ultimate goal is to help instructors by guiding them to posts, topics, and learners that require immediate interventions. Given the above research aim the researcher conducted different experiments to fill the gap in literature based on different platform datasets (the FutureLearn platform and the Stanford MOOCPosts dataset) in terms of the former, three MOOC corpora were prepared: two of them gold-standard MOOC corpora to identify urgent posts, annotated by selected experts in the field; the third is a corpus detailing learner dropout. Based in these datasets, different architectures and classification models based on traditional machine learning, and deep learning approaches were proposed. In this thesis, the task of determining the need for instructor intervention was tackled from three perspectives: (i) identifying relevant posts, (ii) identifying relevant topics, and (iii) identifying relevant learners. Posts written by learners were classified into two categories: (i) (urgent) intervention and (ii) (non-urgent) intervention. Also, learners were classified into: (i) requiring instructor intervention (at risk of dropout) and (ii) no need for instructor intervention (completer). In identifying posts, two experiments were used to contribute to this field. The first is a novel classifier based on a deep learning model that integrates novel MOOC post dimensions such as numerical data in addition to textual data; this represents a novel contribution to the literature as all available models at the time of writing were based on text-only. The results demonstrate that the combined, multidimensional features model proposed in this project is more effective than the text-only model. The second contribution relates to creating various simple and hybrid deep learning models by applying plug & play techniques with different types of inputs (word-based or word-character-based) and different ways of representing target input words as vector representations of a particular word. According to the experimental findings, employing Bidirectional Encoder Representations from Transformers (BERT) for word embedding rather than word2vec as the former is more effective at the intervention task than the latter across all models. Interestingly, adding word-character inputs with BERT does not improve performance as it does for word2vec. Additionally, on the task of identifying topics, this is the first time in the literature that specific language terms to identify the need for urgent intervention in MOOCs were obtained. This was achieved by analysing learner MOOC posts using latent Dirichlet allocation (LDA) and offers a visualisation tool for instructors or learners that may assist them and improve instructor intervention. In addition, this thesis contributes to the literature by creating mechanisms for identifying MOOC learners who may need instructor intervention in a new context, i.e., by using their historical online forum posts as a multi-input approach for other deep learning architectures and Transformer models. The findings demonstrate that using the Transformer model is more effective at identifying MOOC learners who require instructor intervention. Next, the thesis sought to expand its methodology to identify posts that relate to learner behaviour, which is also a novel contribution, by proposing a novel priority model to identify the urgency of intervention building based on learner histories. This model can classify learners into three groups: low risk, mid risk, and high risk. The results show that the completion rates of high-risk learners are very low, which confirms the importance of this model. Next, as MOOC data in terms of urgent posts tend to be highly unbalanced, the thesis contributes by examining various data balancing methods to spot situations in which MOOC posts urgently require instructor assistance. This included developing learner and instructor models to assist instructors to respond to urgent MOOCs posts. The results show that models with undersampling can predict the most urgent cases; 3x augmentation + undersampling usually attains the best performance. Finally, for the first time, this thesis contributes to the literature by applying text classification explainability (eXplainable Artificial Intelligence (XAI)) to an instructor intervention model, demonstrating how using a reliable predictor in combination with XAI and colour-coded visualisation could be utilised to assist instructors in deciding when posts require urgent intervention, as well as supporting annotators to create high-quality, gold-standard datasets to determine posts cases where urgent intervention is required

COGNITIVE PRESENCE IN PEER FACILITATED ASYNCHRONOUS ONLINE DISCUSSION: THE PATTERNS AND HOW TO FACILITATE

This study, in the context of peer-facilitated asynchronous online discussion, explored the characteristics and patterns of students’ cognitive presence, and examined the practices that aim to enhance cognitive presence development. Participants were 53 students from a graduate-level online course that focused on the integration of educational technologies. Data were collected from discussion transcripts, student survey, student artifacts, and researcher’s observations. Results demonstrated four phases of students’ cognitive presence: Triggering event, Exploration, Integration, and Resolution. Among the four phases, students’ cognitive presence tended to aggregate at the middle phases: Integration and Exploration. Percentage of the Resolution was very low. The distribution of students’ discussion behaviors further revealed: a) the hierarchical relationship between the four phases: Integration and Resolution involved a higher-level of cognitive engagement, and Triggering event and Exploration involved a lower-level of cognitive engagement; b) the phase of Resolution heavily relied on experiment, while the other three phases heavily relied on making use of personal experience; c) creating of cognitive presence occurred in both the private space of individual activities and the shared space of having dialogues. The conversation analysis of threads and episodes explored the temporal evolvement of cognitive presence. The results showed that, in an ongoing discussion, students’ cognitive presence evolved in a non-linear way, rather than strictly phase by phase as suggested by the PI model. Experiments were designed and conducted to determine the effects of two pedagogical interventions – 1) providing guidance on peer facilitation techniques; 2) asking students to label their posts. The results showed that the Intervention 1 and the combination of two interventions credibly improved students’ cognitive presence. They were especially effective in improving Integration, a higher level of cognitive presence. After having added Intervention 2, cognitive presence increased from the first-half to the second-half semester, although the improvement was not found to be statistically credible. This study confirmed the close association between and among cognitive presence, social interaction, and peer facilitation. The results clearly showed that Intervention 1 – providing guidance on peer facilitation credibly improved students’ social interaction and peer facilitation. However, Mixed findings were obtained for Intervention 2 – asking students to label their posts. It was found that Intervention 2 positively increased students’ social interaction. However, it did not show any impact on students’ peer facilitation behaviors. It is also worth noting that the effect of the combination of two interventions was much larger than any single one of them. Conversation analysis was conducted to zoom in on the dynamic process of discussion. The cases revealed that when students were provided with the guidance on peer facilitation techniques, they tended to use a variety of facilitation techniques in a strategic way to help peers to achieve a sustained and deeper-level conversation. Compared to the control group, the students in the treatment group showed more peer facilitation behaviors, which led to more conversations and more higher-level cognitive presence. This study has unpacked the complexity of students’ cognitive presence in a peer-facilitated discussion environment, especially when students are coached in performing teaching presence. The results shed light on the pedagogical practices and strategies of creating an online learning community that incubates rich cognitive presence. Finally, implications are discussed for the research and practices in online instruction and discussion analytics

An exploration of the use of MOOCs for Malaysian teachers' professional development

This thesis is a presentation of a project undertaken to explore the contribution that Massive Open Online Courses (MOOCs) can make to the professional development of Malaysian teachers who teach English as a second language in school and higher education institutions. MOOCs are seen as a platform with huge potential in this field, but their application is subject to the accessibility of teacher networks, suitability of course structures and organisation, challenges in the transferability of knowledge and skills, teachers’ readiness in using MOOCs and the opportunities and constraints which the work and family context provides. This thesis includes observations of ten newly available MOOCs and interviews with 14 Malaysian MOOC participants. The ten MOOCs observed are from existing providers: NovoEd, Coursera, FutureLearn and Canvas. They were analysed and compared using a matrix with three main focuses: pedagogy, content materials, and assessment. 14 participants were later recruited as volunteer participants in these MOOCs. All 14 were currently teaching or planning to teach ESL in Malaysia. Their responses were coded and analysed using qualitative analysis software, Atlas.ti. Realising how important it was to find out the context where the participants worked, a series of ethnographic style observations were also carried out. The data gained was coded and analysed using Atlas.ti too. The findings revealed that all ten courses corresponded to the idea of an xMOOC, in that they were run on a model of instructional design. However, in terms of the degree of openness, the MOOCs differed. They were more or less open, or simply contained, depending on how they were pedagogically organised, the materials provided, and the way the assessment was conducted. Reflecting on the participants’ responses, all the MOOCs used had the potential to be an effective medium for teachers’ CPD because they had the characteristic of an ideal CPD programme. Further, they offered teachers certification, rooms for informal learning and the flexibility that a CPD programme cannot offer. However, MOOCs are found to be difficult to fit into the Malaysian EFL teachers’ context because the participants had other things going on in their lives. Further investigation revealed that the participants valued the physical presence of others and thought they learned more by having the face-to-face conversation. Thus, a hybrid of MOOCs and face-to-face interaction seemed to be a logical solution to promote CPD. The study succeeded in showing the value and variation in MOOCs and the opportunity for providing teachers with a desirable CPD programme, thus demonstrating the range of possibilities open to course designers and providers. Through careful consideration of the key characteristics of an ideal CPD programme, along with an opportunity to create a community of practice online and offline, a hybrid MOOC could be designed and implemented to best meet teachers' local and contextual needs. Such an approach could generate positive change among schools, students, and even the education system at large