Qualitative, quantitative, and data mining methods for analyzing log data to characterize students' learning strategies and behaviors [discussant]

This symposium addresses how different classes of research methods, all based upon the use of log data from educational software, can facilitate the analysis of students’ learning strategies and behaviors. To this end, four multi-method programs of research are discussed, including the use of qualitative, quantitative-statistical, quantitative-modeling, and educational data mining methods. The symposium presents evidence regarding the applicability of each type of method to research questions of different grain sizes, and provides several examples of how these methods can be used in concert to facilitate our understanding of learning processes, learning strategies, and behaviors related to motivation, meta-cognition, and engagement

Does gamification work for boys and girls?: An exploratory study with a virtual learning environment

SIGAPP ACM Special Interest Group on Applied ComputingThe development and use of Virtual Learning Environments (VLE) has increased considerably over the past decades. Following that trend, many research findings have shown the benefits of using VLE during the learning process. Nevertheless, there are important problems that hinder their use requiring further investigation. Among them, one of the main problems is the inappropriate use of these systems by students. The boredom, lack of interest, monotony, lack of motivation, among other factors, ultimately causes students to behave inappropriately and lead them to a lower performance. In this context, the proposed study investigates whether it is possible to reduce undesirable behaviors and increase performance of students through the use of game mechanics (i.e. gamification). We develop a VLE, E-Game, that can turn on/off several game mechanics, such as points, badges, levels and so on. A case study was conducted with two groups of students to investigate their behavior during their interaction with E-Game with and without gamification. The results indicate that the gamification implemented by E-Game contributed to improve student performance in the case of boys. Yet, improvement was not observed in the case of girls. Furthermore, it was not possible to conclude whether the use of gamification helps to prevent inappropriate student behavior, and therefore, further studies and experiments are needed

Improving fairness in machine learning systems: What do industry practitioners need?

The potential for machine learning (ML) systems to amplify social inequities and unfairness is receiving increasing popular and academic attention. A surge of recent work has focused on the development of algorithmic tools to assess and mitigate such unfairness. If these tools are to have a positive impact on industry practice, however, it is crucial that their design be informed by an understanding of real-world needs. Through 35 semi-structured interviews and an anonymous survey of 267 ML practitioners, we conduct the first systematic investigation of commercial product teams' challenges and needs for support in developing fairer ML systems. We identify areas of alignment and disconnect between the challenges faced by industry practitioners and solutions proposed in the fair ML research literature. Based on these findings, we highlight directions for future ML and HCI research that will better address industry practitioners' needs.Comment: To appear in the 2019 ACM CHI Conference on Human Factors in Computing Systems (CHI 2019

Towards Scalable Assessment of Performance-Based Skills: Generalizing a Detector of Systematic Science Inquiry to a Simulation with a Complex Structure

Abstract. There are well-acknowledged challenges to scaling computerized performance-based assessments. One such challenge is reliably and validly identifying ill-defined skills. We describe an approach that leverages a data mining framework to build and validate a detector that evaluates an ill-defined inquiry process skill, designing controlled experiments. The detector was originally built and validated for use with physical science simulations that have a simpler, linear causal structure. In this paper, we show that the detector can be used to identify demonstration of skill within a life science simulation on Ecosystems that has a complex underlying causal structure. The detector is evaluated in three ways: 1) identifying skill demonstration for a new student cohort, 2) handling the variability in how students conduct experiments, and 3) using it to determine when students are off-track before they finish collecting data

When Does Disengagement Correlate with Performance in Spoken Dialog Computer Tutoring?

In this paper we investigate how student disengagement relates to two performance metrics in a spoken dialog computer tutoring corpus, both when disengagement is measured through manual annotation by a trained human judge, and also when disengagement is measured through automatic annotation by the system based on a machine learning model. First, we investigate whether manually labeled overall disengagement and six different disengagement types are predictive of learning and user satisfaction in the corpus. Our results show that although students’ percentage of overall disengaged turns negatively correlates both with the amount they learn and their user satisfaction, the individual types of disengagement correlate differently: some negatively correlate with learning and user satisfaction, while others don’t correlate with eithermetric at all. Moreover, these relationships change somewhat depending on student prerequisite knowledge level. Furthermore, using multiple disengagement types to predict learning improves predictive power. Overall, these manual label-based results suggest that although adapting to disengagement should improve both student learning and user satisfaction in computer tutoring, maximizing performance requires the system to detect and respond differently based on disengagement type. Next, we present an approach to automatically detecting and responding to user disengagement types based on their differing correlations with correctness. Investigation of ourmachine learningmodel of user disengagement shows that its automatic labels negatively correlate with both performance metrics in the same way as the manual labels. The similarity of the correlations across the manual and automatic labels suggests that the automatic labels are a reasonable substitute for the manual labels. Moreover, the significant negative correlations themselves suggest that redesigning ITSPOKE to automatically detect and respond to disengagement has the potential to remediate disengagement and thereby improve performance, even in the presence of noise introduced by the automatic detection process

LMS assessment: using IRT analysis to detect defective multiple-choice test items

Due to the computerisation of assessment tests, the use of Item Response Theory (IRT) has become commonplace for educational assessment development, evaluation and refinement. When used appropriately by a Learning Management System (LMS), IRT can improve the assessment quality, increase the efficiency of the testing process and provide in-depth descriptions of item properties. This paper introduces a methodological and architectural framework which embeds an IRT analysis tool in an LMS so as to extend its functionality with assessment optimisation support. By applying a set of validity rules to the statistical indices produced by the IRT analysis, the enhanced LMS is able to detect several defective items from an item pool which are then reported for reviewing of their content. Assessment refinement is achieved by repeatedly employing this process until all flawed items are eliminated

Let's Set Up Some Subgoals: Understanding Human-Pedagogical Agent Collaborations and Their Implications for Learning and Prompt and Feedback Compliance

International audienceResearch on collaborative learning between humans and virtual pedagogical agents represents a necessary extension to recent research on the conceptual, theoretical, methodological, analytical, and educational issues behind co-and socially-shared regulated learning between humans. This study presents a novel coding framework that was developed and used to describe collaborations between learners and a pedagogical agent (PA) during a subgoal setting activity with MetaTutor, an intelligent tutoring system. Learner-PA interactions were examined across two scaffolding conditions: prompt and feedback (PF), and control. Learners' compliance to follow the PA's prompts and feedback in the PF condition were also examined. Results demonstrated that learners followed the PA's prompts and feedback to help them set more appropriate subgoals for their learning session the majority of the time. Descriptive statistics revealed that when subgoals were set collaboratively between learners and the PA, they generally lead to higher proportional learning gains when compared to less collaboratively set goals. Taken together, the results provide preliminary evidence that learners are both willing to engage in and benefit from collaborative interactions with PAs when immediate, directional feedback and the opportunity to try again are provided. Implications and future directions for extending co-and socially-shared regulated learning theories to include learner-PA interactions are proposed

Discovery with Models: A Case Study on Carelessness in Computer-based Science Inquiry

In recent years, an increasing number of analyses in Learning Analytics and Educational Data Mining (EDM) have adopted a "Discovery with Models" approach, where an existing model is used as a key component in a new EDM/analytics analysis. This article presents a theoretical discussion on the emergence of discovery with models, its potential to enhance research on learning and learners, and key lessons learned in how discovery with models can be conducted validly and effectively. We illustrate these issues through discussion of a case study where discovery with models was used to investigate a form of disengaged behavior, i.e., carelessness, in the context of middle school computer-based science inquiry. This behavior has been acknowledged as a problem in education as early as the 1920s. With the increasing use of high-stakes testing, the cost of student carelessness can be higher. For instance, within computer-based learning environments careless errors can result in reduced educational effectiveness, with students continuing to receive material they have already mastered. Despite the importance of this problem, it has received minimal research attention, in part due to difficulties in operationalizing carelessness as a construct. Building from theory on carelessness and a Bayesian framework for knowledge modeling, we use machine-learned detectors to predict carelessness within authentic use of a computer-based learning environment. We then use a discovery with models approach to link these validated carelessness measures to survey data, to study the correlations between the prevalence of carelessness and student goal orientation. The second construct, carelessness, refers to incorrect answers given by a student on material that the student should be able to answer correctly Rodriguez-Fornells & Maydeu-Olivares, 2000). The application of discovery with models involves two main phases. First, a model of a construct is developed using machine learning or knowledge engineering techniques, and is then validated, as discussed below. Second, this validated model is applied to data and used as a component in another analysis: For example, for identifying outliers through model predictions; examining which variables best predict the modeled construct; finding relationships between the construct and other variables using correlations, predictions, associations rules, causal relationships or other methods; or studying the contexts where the construct occurs, including its prevalence across domains, systems, or populations. For example, in One essential question to pose prior to a discovery with model analysis is whether the model adopted is valid, both overall, and for the specific situation in which it is being used. Ideally, a model should be validated using an approach such as cross-validation, where the model is repeatedly trained on one portion of the data and tested on a different portion, with model predictions compared to appropriate external measures, for example assessments made by humans with acceptably high inter-rater reliability, such as field observations of student behavior for gaming the system (cf. Even after validating in this fashion, validity should be re-considered if the model is used for a substantially different population or context than was used when developing the model.. An alternative approach is to use a simpler knowledge-engineered definition, rationally deriving a function/rule that is then applied to the data. In this case, the model can be inferred to have face validity. However, knowledge-engineered models often DISCOVERY WITH MODELS: A CASE STUDY ON CARELESSNESS 6 produce different results than machine learning-based models, for example in the case of gaming the system. Research studying whether student or content is a better predictor of gaming the system identified different results, depending on which model was applied (cf. Baker, 2007a

O papel dos jogos no envolvimento dos estudantes

O presente estudo tem como objetivo fornecer uma revisão abrangente da literatura sobre gamificação, jogos sérios e envolvimento dos alunos, a fim de sintetizar a informação disponível e determinar o papel da gamificação e da implementação de jogos no processo de aprendizagem para aumentar o envolvimento dos alunos. O estudo examinará o contexto da gamificação e dos jogos sérios na educação, explorará o problema do desinteresse dos alunos nas abordagens de aprendizagem tradicionais e investigará a forma como a gamificação e a aprendizagem baseada em jogos podem resolver este problema. A questão de investigação que orienta este estudo é: "Qual é o papel da gamificação e da implementação de jogos no processo de aprendizagem no envolvimento e motivação dos alunos?" Para responder à questão de investigação, será efetuada uma revisão narrativa da literatura. Estudos, artigos e publicações relevantes serão reunidos e analisados para identificar as principais conclusões e tendências relativas à eficácia da gamificação e da aprendizagem baseada em jogos no reforço do envolvimento dos alunos. A metodologia envolverá a realização de um questionário abrangente que avaliará as perceções dos alunos e dos professores sobre a gamificação e a implementação de jogos no processo de aprendizagem, bem como o impacto dos jogos na aprendizagem, e analisará os dados extraídos. Espera-se que os principais resultados deste estudo forneçam informações sobre os benefícios e desafios da utilização da gamificação e da aprendizagem baseada em jogos na educação. Prevê-se que os resultados demonstrem o impacto positivo da gamificação no envolvimento, motivação e resultados de aprendizagem dos alunos. Além disso, o estudo explorará fatores que contribuem para a implementação bem-sucedida de estratégias de gamificação, como a conceção dos elementos do jogo, o alinhamento da mecânica do jogo com os objetivos de aprendizagem e a importância de incorporar feedback e recompensas. A investigação também irá lançar luz sobre potenciais limitações e barreiras à utilização eficaz da gamificação na sala de aula. Este estudo tem implicações para educadores, designers instrucionais e decisores políticos no domínio da educação. Os resultados informarão o desenvolvimento de melhores práticas e diretrizes para a incorporação da gamificação e da aprendizagem baseada em jogos em contextos educativos, com o objetivo de aumentar o envolvimento dos alunos e promover experiências de aprendizagem significativas.The present study aims to provide a comprehensive review of the literature on gamification, serious games, and student engagement in order to synthesize the available information and determine the role of gamification and game implementation in the learning process to increase student engagement. The study will examine the context of gamification and serious games in education, explore the problem of student disengagement in traditional learning approaches, and investigate how gamification and game-based learning can address this issue. The research question that guides this study is: "What is the role of gamification and the implementation of games in the learning process on student engagement and motivation?" To address the research question, a systematic review of the literature will be conducted. Relevant studies, articles, and publications will be gathered and analyzed to identify key findings and trends regarding the effectiveness of gamification and game-based learning in enhancing student engagement. The methodology will involve conducting a comprehensive questionaire evaluating student and teacher perceptions on gamification and the implementation of games in the learning process as well as the impact of games on learning and analyzing the data extracted from these. The main results of this study are expected to provide insights into the benefits and challenges of using gamification and game-based learning in education. It is anticipated that the findings will demonstrate the positive impact of gamification on student engagement, motivation, and learning outcomes. Additionally, the study will explore factors that contribute to the successful implementation of gamification strategies, such as the design of game elements, the alignment of game mechanics with learning objectives, and the importance of incorporating feedback and rewards. The research will also shed light on potential limitations and barriers to the effective use of gamification in the classroom. This study has implications for educators, instructional designers, and policymakers in the field of education. The findings will inform the development of best practices and guidelines for incorporating gamification and game-based learning into educational settings, with the aim of enhancing student engagement and promoting meaningful learning experiences