THE ROLE OF SIMULATION IN SUPPORTING LONGER-TERM LEARNING AND MENTORING WITH TECHNOLOGY

Mentoring is an important part of professional development and longer-term learning. The nature of longer-term mentoring contexts means that designing, developing, and testing adaptive learning sys-tems for use in this kind of context would be very costly as it would require substantial amounts of fi-nancial, human, and time resources. Simulation is a cheaper and quicker approach for evaluating the impact of various design and development decisions. Within the Artificial Intelligence in Education (AIED) research community, however, surprisingly little attention has been paid to how to design, de-velop, and use simulations in longer-term learning contexts. The central challenge is that adaptive learning system designers and educational practitioners have limited guidance on what steps to consider when designing simulations for supporting longer-term mentoring system design and development deci-sions. My research work takes as a starting point VanLehn et al.’s [1] introduction to applications of simulated students and Erickson et al.’s [2] suggested approach to creating simulated learning envi-ronments. My dissertation presents four research directions using a real-world longer-term mentoring context, a doctoral program, for illustrative purposes. The first direction outlines a framework for guid-ing system designers as to what factors to consider when building pedagogical simulations, fundamen-tally to answer the question: how can a system designer capture a representation of a target learning context in a pedagogical simulation model? To illustrate the feasibility of this framework, this disserta-tion describes how to build, the SimDoc model, a pedagogical model of a longer-term mentoring learn-ing environment – a doctoral program. The second direction builds on the first, and considers the issue of model fidelity, essentially to answer the question: how can a system designer determine a simulation model’s fidelity to the desired granularity level? This dissertation shows how data from a target learning environment, the research literature, and common sense are combined to achieve SimDoc’s medium fidelity model. The third research direction explores calibration and validation issues to answer the question: how many simulation runs does it take for a practitioner to have confidence in the simulation model’s output? This dissertation describes the steps taken to calibrate and validate the SimDoc model, so its output statistically matches data from the target doctoral program, the one at the university of Saskatchewan. The fourth direction is to demonstrate the applicability of the resulting pedagogical model. This dissertation presents two experiments using SimDoc to illustrate how to explore pedagogi-cal questions concerning personalization strategies and to determine the effectiveness of different men-toring strategies in a target learning context. Overall, this dissertation shows that simulation is an important tool in the AIED system design-ers’ toolkit as AIED moves towards designing, building, and evaluating AIED systems meant to support learners in longer-term learning and mentoring contexts. Simulation allows a system designer to exper-iment with various design and implementation decisions in a cost-effective and timely manner before committing to these decisions in the real world

Intellectualization of Moodle will allow to prolong its maintenance in universities

In the report notes that Moodle which is applied in Universities of the Russian Federation is become outdated morally. Abroad scientists have been directing the main efforts on creation of intellectual tutoring systems with a natural language dialogue since 2000. Maintenance of Moodle can be prolonged by intellectualization of the system. The list of tasks which need to be solved to increase intellectuality of this system is specified. The functional chart of intellectual part of Moodle is considered

How Simulation can Illuminate Pedagogical and System Design Issues in Dynamic Open Ended Learning Environments

A Dynamic Open-Ended Learning Environment (DOELE) is a collection of learners and learning objects (LOs) that could be constantly changing. In DOELEs, learners need the support of Advanced Learning Technology (ALT), but most ALT is not designed to run in such environments. An architecture for designing advanced learning technology that is compatible with DOELEs is the ecological approach (EA). This thesis looks at how to test and develop ALT based on the EA, and argues that this process would benefit from the use of simulation. The essential components of an EA-based simulation are: simulated learners, simulated LOs, and their simulated interactions. In this thesis the value of simulation is demonstrated with two experiments. The first experiment focuses on the pedagogical issue of peer impact, how learning is impacted by the performance of peers. By systematically varying the number and type of learners and LOs in a DOELE, the simulation uncovers behaviours that would otherwise go unseen. The second experiment shows how to validate and tune a new instructional planner built on the EA, the Collaborative Filtering based on Learning Sequences planner (CFLS). When the CFLS planner is configured appropriately, simulated learners achieve higher performance measurements that those learners using the baseline planners. Simulation results lead to predictions that ultimately need to be proven in the real world, but even without real world validation such predictions can be useful to researchers to inform the ALT system design process. This thesis work shows that it is not necessary to model all the details of the real world to come to a better understanding of a pedagogical issue such as peer impact. And, simulation allowed for the design of the first known instructional planner to be based on usage data, the CFLS planner. The use of simulation for the design of EA-based systems opens new possibilities for instructional planning without knowledge engineering. Such systems can find niche learning paths that may have never been thought of by a human designer. By exploring pedagogical and ALT system design issues for DOELEs, this thesis shows that simulation is a valuable addition to the toolkit for ALT researchers

A Closer Look into Recent Video-based Learning Research: A Comprehensive Review of Video Characteristics, Tools, Technologies, and Learning Effectiveness

People increasingly use videos on the Web as a source for learning. To support this way of learning, researchers and developers are continuously developing tools, proposing guidelines, analyzing data, and conducting experiments. However, it is still not clear what characteristics a video should have to be an effective learning medium. In this paper, we present a comprehensive review of 257 articles on video-based learning for the period from 2016 to 2021. One of the aims of the review is to identify the video characteristics that have been explored by previous work. Based on our analysis, we suggest a taxonomy which organizes the video characteristics and contextual aspects into eight categories: (1) audio features, (2) visual features, (3) textual features, (4) instructor behavior, (5) learners activities, (6) interactive features (quizzes, etc.), (7) production style, and (8) instructional design. Also, we identify four representative research directions: (1) proposals of tools to support video-based learning, (2) studies with controlled experiments, (3) data analysis studies, and (4) proposals of design guidelines for learning videos. We find that the most explored characteristics are textual features followed by visual features, learner activities, and interactive features. Text of transcripts, video frames, and images (figures and illustrations) are most frequently used by tools that support learning through videos. The learner activity is heavily explored through log files in data analysis studies, and interactive features have been frequently scrutinized in controlled experiments. We complement our review by contrasting research findings that investigate the impact of video characteristics on the learning effectiveness, report on tasks and technologies used to develop tools that support learning, and summarize trends of design guidelines to produce learning video

Technology-supported personalised learning: Rapid Evidence Review

This Rapid Evidence Review (RER) provides an overview of existing research on the use of technology to support personalised learning in low- and middle-income countries (LMICs). The RER has been produced in response to the widespread global shutdown of schools resulting from the outbreak of COVID-19. It therefore emphasises transferable insights that may be applicable to educational responses resulting from the limitations caused by COVID-19. In the current context, lessons learnt from the use of technology-supported personalised learning — in which technology enables or supports learning based upon particular characteristics of relevance or importance to learners — are particularly salient given this has the potential to adapt to learners’ needs by ‘teaching at the right level’

Integrating knowledge tracing and item response theory: A tale of two frameworks

Traditionally, the assessment and learning science commu-nities rely on different paradigms to model student performance. The assessment community uses Item Response Theory which allows modeling different student abilities and problem difficulties, while the learning science community uses Knowledge Tracing, which captures skill acquisition. These two paradigms are complementary - IRT cannot be used to model student learning, while Knowledge Tracing assumes all students and problems are the same. Recently, two highly related models based on a principled synthesis of IRT and Knowledge Tracing were introduced. However, these two models were evaluated on different data sets, using different evaluation metrics and with different ways of splitting the data into training and testing sets. In this paper we reconcile the models' results by presenting a unified view of the two models, and by evaluating the models under a common evaluation metric. We find that both models are equivalent and only differ in their training procedure. Our results show that the combined IRT and Knowledge Tracing models offer the best of assessment and learning sciences - high prediction accuracy like the IRT model, and the ability to model student learning like Knowledge Tracing

Automatic question generation about introductory programming code

Many students who learn to program end up writing code they do not understand. Most of the available code evaluation systems evaluate the submitted solution functionally and not the knowledge of the person who submitted it. This dissertation proposes a system that generates questions about the code submitted by the student, analyses their answers and returns the correct answers. In this way, students reflect about the code they have written and the teachers of the programming courses can better pinpoint their difficulties. We carried out an experiment with undergraduate and master's students in Computer Science degrees in order to understand their difficulties and test the prototype's robustness. We concluded that most students, although understanding simple details of the code they write, do not understand the behaviour of the program entirely, especially with respect to program state. Improvements to the prototype and how to conduct future experiments are also suggested.Muitos alunos que aprendem a programar acabam por escrever código que não entendem. A maior parte dos sistemas de avaliação de código disponíveis avaliam a solução submetida funcionalmente e não o conhecimento da pessoa que o submeteu. Esta dissertação propõe um sistema que gera questões sobre o código submetido pelo aluno, analisa as suas respostas e devolve as respostas corretas. Desta forma, os alunos refletem sobre o código que escreveram e os professores das unidades curriculares de programação conseguem identificar melhor as suas dificuldades. Conduzimos uma experiência com alunos de licenciatura e mestrado em Engenharia Informática e cursos relacionados de forma a perceber quais as suas dificuldades e testar a robustez do protótipo. Concluímos que a maior parte dos alunos, embora entendam detalhes simples do código que escrevem, não entendem o comportamento do programa na sua totalidade e o estado que este possui num determinado momento. São também sugeridas melhorias ao protótipo e à condução de futuras experiências

Automatic question generation about introductory programming code

Many students who learn to program end up writing code they do not understand. Most of the available code evaluation systems evaluate the submitted solution functionally and not the knowledge of the person who submitted it. This dissertation proposes a system that generates questions about the code submitted by the student, analyses their answers and returns the correct answers. In this way, students reflect about the code they have written and the teachers of the programming courses can better pinpoint their difficulties. We carried out an experiment with undergraduate and master’s students in Computer Science degrees in order to understand their difficulties and test the prototype’s robustness. We concluded that most students, although understanding simple details of the code they write, do not understand the behaviour of the program entirely, especially with respect to program state. Improvements to the prototype and how to conduct future experiments are also suggested.Muitos alunos que aprendem a programar acabam por escrever código que não entendem. A maior parte dos sistemas de avaliação de código disponíveis avaliam a solução submetida funcionalmente e não o conhecimento da pessoa que o submeteu. Esta dissertação propõe um sistema que gera questões sobre o código submetido pelo aluno, analisa as suas respostas e devolve as respostas corretas. Desta forma, os alunos refletem sobre o código que escreveram e os professores das unidades curriculares de programação conseguem identificar melhor as suas dificuldades. Conduzimos uma experiência com alunos de licenciatura e mestrado em Engenharia Informática e cursos relacionados de forma a perceber quais as suas dificuldades e testar a robustez do protótipo. Concluímos que a maior parte dos alunos, embora entendam detalhes simples do código que escrevem, não entendem o comportamento do programa na sua totalidade e o estado que este possui num determinado momento. São também sugeridas melhorias ao protótipo e à condução de futuras experiências

Adapting Collaborative Learning Tools to Support Group Peer Mentorship

Group peer mentorship is a relatively new addition to the area of collaborative learning. We see an untapped potential in supporting this model of mentorship with the existing collaborative learning tools like peer review and wiki. Therefore, we proposed to use a modified peer review system and a modified wiki system. From our preliminary studies using both peer review and wiki systems, we found that participants preferred the peer-review system to the wiki system in supporting them for mentorship. Therefore, this dissertation specifically addresses how to adapt the peer review system to support group peer mentorship. We proposed a modified peer review system, which comprises seven stages – initial submission of the first draft of the paper by the author, the review of author’s paper by peer reviewers, release of review feedback to the author, back-evaluation of their reviews by the authors, modification of the paper by the author, submission of the final paper and the final stage where both authors and reviewers provide an evaluation of the peer review process with respect to their learning, their perception of the helpfulness of the process, and their satisfaction with the process. We also proposed to use our group matching algorithm, based on some constraints and the principles of the Hungarian algorithm, to achieve a diversified grouping of peers for each peer review session. With these, we conducted six peer review studies with the graduate and undergraduate students at the University of Saskatchewan and teachers in Chile. This dissertation reports on the findings from these studies. We found that peer review, with some modifications, is a good tool to facilitate group peer mentorship. An evaluation of the performance of our group matching algorithm showed an improvement over three other algorithms, with respect to three metrics – knowledge gain of peers, time and space consumption of the algorithm. Finally, this dissertation also shows that wiki has the potential to support group peer mentorship, but needs further research

Efficiency of Automated Detectors of Learner Engagement and Affect Compared with Traditional Observation Methods

This report investigates the costs of developing automated detectors of student affect and engagement and applying them at scale to the log files of students using educational software. We compare these costs and the accuracy of the computer-based observations with those of more traditional observation methods for detecting student engagement and affect. We discuss the potential for automated detectors to contribute to the development of adaptive and responsive educational software