Dropout Model Evaluation in MOOCs

The field of learning analytics needs to adopt a more rigorous approach for predictive model evaluation that matches the complex practice of model-building. In this work, we present a procedure to statistically test hypotheses about model performance which goes beyond the state-of-the-practice in the community to analyze both algorithms and feature extraction methods from raw data. We apply this method to a series of algorithms and feature sets derived from a large sample of Massive Open Online Courses (MOOCs). While a complete comparison of all potential modeling approaches is beyond the scope of this paper, we show that this approach reveals a large gap in dropout prediction performance between forum-, assignment-, and clickstream-based feature extraction methods, where the latter is significantly better than the former two, which are in turn indistinguishable from one another. This work has methodological implications for evaluating predictive or AI-based models of student success, and practical implications for the design and targeting of at-risk student models and interventions

Trusting the Explainers: Teacher Validation of Explainable Artificial Intelligence for Course Design

Deep learning models for learning analytics have become increasingly popular over the last few years; however, these approaches are still not widely adopted in real-world settings, likely due to a lack of trust and transparency. In this paper, we tackle this issue by implementing explainable AI methods for black-box neural networks. This work focuses on the context of online and blended learning and the use case of student success prediction models. We use a pairwise study design, enabling us to investigate controlled differences between pairs of courses. Our analyses cover five course pairs that differ in one educationally relevant aspect and two popular instance-based explainable AI methods (LIME and SHAP). We quantitatively compare the distances between the explanations across courses and methods. We then validate the explanations of LIME and SHAP with 26 semi-structured interviews of university-level educators regarding which features they believe contribute most to student success, which explanations they trust most, and how they could transform these insights into actionable course design decisions. Our results show that quantitatively, explainers significantly disagree with each other about what is important, and qualitatively, experts themselves do not agree on which explanations are most trustworthy. All code, extended results, and the interview protocol are provided at https://github.com/epfl-ml4ed/trusting-explainers.Comment: Accepted as a full paper at LAK 2023: The 13th International Learning Analytics and Knowledge Conference, March 13-17, 2023, Arlington, Texas, US

Differences in Profile and Personal Learning of Massive Open Online Courses (MOOCs) Participants in Utilizing Open Educational Resources (OER) and Parenting Programs

Massive Open Online Courses (MOOCs) at the Indonesian Open University are independent learning tools that provide the opportunity to introduce the University to the wider community by providing quality knowledge for free. This research aims to describe the Indonesian Open University's efforts to provide solutions to limited access and quality of education by providing various free courses to the wider community through MOOCs which are open and can be attended by an unlimited number of participants. This survey study uses a modified Self-Regulated Learning (SRL) questionnaire, adapted to the profile of the community as participants and the characteristics of the Open University (UT') MOOCs program. In 2023, UT offers 23 MOOC titles, but this research limited it to only two MOOCs, Open Educational Resources (OER) with 56 participants and Parenting with 73 participants (129 teachers and parents of early childhood). This research revealed two MOOCs in real life which were analyzed using the Mann-Whitney U test to see whether there were differences in the answers between respondents who took part in the OER MOOCS and the answers of respondents who took part in the MOOCS parenting program. Apart from that, UT's MOOCs currently still use Indonesian considering that currently they are specifically for Indonesian people. UT MOOCs were developed using an open-source platform (Moodle) considering that previously lecturers and tutors had used similar platforms for their formal courses. Keywords: self-directed learning, Massive Open Online Courses, parenting program, early childhood References: Aldowah, H., Al-Samarraie, H., Alzahrani, A. I., & Alalwan, N. (2020). Factors affecting student dropout in MOOCs: A cause and effect decision‐making model. Journal of Computing in Higher Education, 32(2), 429–454. https://doi.org/10.1007/s12528-019-09241-y Al-Emran, M., Mezhuyev, V., Kamaludin, A., & Shaalan, K. (2018). The impact of knowledge management processes on information systems: A systematic review. International Journal of Information Management, 43, 173–187. https://doi.org/10.1016/j.ijinfomgt.2018.08.001 Ali, Z., Gongbing, B., & Mehreen, A. (2018). Understanding and predicting academic performance through cloud computing adoption: A perspective of technology acceptance model. Journal of Computers in Education, 5(3), 297–327. https://doi.org/10.1007/s40692-018-0114-0 Al-Rahmi, W. M., Yahaya, N., Alamri, M. M., Alyoussef, I. Y., Al-Rahmi, A. M., & Kamin, Y. B. (2021). Integrating innovation diffusion theory with technology acceptance model: Supporting students’ attitude towards using a massive open online course (MOOCs) systems. Interactive Learning Environments, 29(8), 1380–1392. https://doi.org/10.1080/10494820.2019.1629599 Arpaci, I. (2019). A hybrid modeling approach for predicting the educational use of mobile cloud computing services in higher education. Computers in Human Behavior, 90, 181–187. https://doi.org/10.1016/j.chb.2018.09.005 Burd, E. L., Smith, S. P., & Reisman, S. (2015). Exploring Business Models for MOOCs in Higher Education. Innovative Higher Education, 40(1), 37–49. https://doi.org/10.1007/s10755-014-9297-0 Chong, A. Y.-L., Chan, F. T. S., Goh, M., & Tiwari, M. K. (2013). Do interorganisational relationships and knowledge-management practices enhance collaborative commerce adoption? International Journal of Production Research, 51(7), 2006–2018. https://doi.org/10.1080/00207543.2012.701776 Cohen, L., & Magen-Nagar, N. (2016). Self-Regulated Learning and a Sense of Achievement in MOOCs Among High School Science and Technology Students. American Journal of Distance Education, 30(2), 68–79. https://doi.org/10.1080/08923647.2016.1155905 Corrado, R., Pretorius, E., & van der Westhuizen, G. (2021). Undergraduate Students’ Experiences of the Use of MOOCs for Learning at a Cambodian University. Education Sciences, 11(7). https://doi.org/10.3390/educsci11070336 Dai, H. M., Teo, T., & Rappa, N. A. (2020). Understanding continuance intention among MOOC participants: The role of habit and MOOC performance. Computers in Human Behavior, 112, 106455. https://doi.org/10.1016/j.chb.2020.106455 de Freitas, S. I., Morgan, J., & Gibson, D. (2015). Will MOOCs transform learning and teaching in higher education? Engagement and course retention in online learning provision. British Journal of Educational Technology, 46(3), 455–471. https://doi.org/10.1111/bjet.12268 Gardner, J., & Brooks, C. (2018). Student success prediction in MOOCs. User Modeling and User-Adapted Interaction, 28(2), 127–203. https://doi.org/10.1007/s11257-018-9203-z Hsu, R. L.-W. (2021). A Grounded Theory Exploration of Language Massive Open Online Courses (LMOOCs): Understanding Students’ Viewpoints. Sustainability, 13(5). https://doi.org/10.3390/su13052577 Janelli, M., & Lipnevich, A. A. (2021). Effects of pre-tests and feedback on performance outcomes and persistence in Massive Open Online Courses. Computers & Education, 161, 104076. https://doi.org/10.1016/j.compedu.2020.104076 Ma, L., & Lee, C. S. (2020). Drivers and barriers to MOOC adoption: Perspectives from adopters and non-adopters. Online Information Review, 44(3), 671–684. https://doi.org/10.1108/OIR-06-2019-0203 Magen-Nagar, N., & Cohen, L. (2017). Learning strategies as a mediator for motivation and a sense of achievement among students who study in MOOCs. Education and Information Technologies, 22(3), 1271–1290. https://doi.org/10.1007/s10639-016-9492-y Moore, R. L., & Wang, C. (2021). Influence of learner motivational dispositions on MOOC completion. Journal of Computing in Higher Education, 33(1), 121–134. https://doi.org/10.1007/s12528-020-09258-8 Rambe, P., & Moeti, M. (2017). Disrupting and democratising higher education provision or entrenching academic elitism: Towards a model of MOOCs adoption at African universities. Educational Technology Research and Development, 65(3), 631–651. https://doi.org/10.1007/s11423-016-9500-3 Reparaz, C., Aznárez-Sanado, M., & Mendoza, G. (2020). Self-regulation of learning and MOOC retention. Computers in Human Behavior, 111, 106423. https://doi.org/10.1016/j.chb.2020.106423 Singh, A., & Sharma, A. (2021). Acceptance of MOOCs as an alternative for internship for management students during COVID-19 pandemic: An Indian perspective. International Journal of Educational Management, 35(6), 1231–1244. https://doi.org/10.1108/IJEM-03-2021-0085 Wei, X., Saab, N., & Admiraal, W. (2021). Assessment of cognitive, behavioral, and affective learning outcomes in massive open online courses: A systematic literature review. Computers & Education, 163, 104097. https://doi.org/10.1016/j.compedu.2020.10409

Predicting Certification in MOOCs based on Students’ Weekly Activities

Massive Open Online Courses (MOOCs) have been growing rapidly, offering low-cost knowledge for both learners and content providers. However, currently there is a very low level of course purchasing (less than 1% of the total number of enrolled students on a given online course opt to purchase its certificate). This can impact seriously the business model of MOOCs. Nevertheless, MOOC research on learners’ purchasing behaviour on MOOCs remains limited. Thus, the umbrella question that this work tackles is if learner’s data can predict their purchasing decision (certification). Our fine-grained analysis attempts to uncover the latent correlation between learner activities and their decision to purchase. We used a relatively large dataset of 5 courses of 23 runs obtained from the less studied MOOC platform of FutureLearn to: (1) statistically compare the activities of non-paying learners with course purchasers, (2) predict course certification using different classifiers, optimising for this naturally strongly imbalanced dataset. Our results show that learner activities are good predictors of course purchasibility; still, the main challenge was that of early prediction. Using only student number of step accesses, attempts, correct and wrong answers, our model achieve promising accuracies, ranging between 0.81 and 0.95 across the five courses. The outcomes of this study are expected to help design future courses and predict the profitability of future runs; it may also help determine what personalisation features could be provided to increase MOOC revenu