Improving and Scaling Mobile Learning via Emotion and Cognitive-state Aware Interfaces

Massive Open Online Courses (MOOCs) provide high-quality learning materials at low cost to millions of learners. Current MOOC designs, however, have minimal learner-instructor communication channels. This limitation restricts MOOCs from addressing major challenges: low retention rates, frequent distractions, and little personalization in instruction. Previous work enriched learner-instructor communication with physiological signals but was not scalable because of the additional hardware requirement. Large MOOC providers, such as Coursera, have released mobile apps providing more flexibility with “on-the-go” learning environments. This thesis reports an iterative process for the design of mobile intelligent interfaces that can run on unmodified smartphones, implicitly sense multiple modalities from learners, infer learner emotions and cognitive states, and intervene to provide gains in learning. The first part of this research explores the usage of photoplethysmogram (PPG) signals collected implicitly on the back-camera of unmodified smartphones. I explore different deep neural networks, DeepHeart, to improve the accuracy (+2.2%) and robustness of heart rate sensing from noisy PPG signals. The second project, AttentiveLearner, infers mind-wandering events via the collected PPG signals at a performance comparable to systems relying on dedicated physiological sensors (Kappa = 0.22). By leveraging the fine-grained cognitive states, the third project, AttentiveReview, achieves significant (+17.4%) learning gains by providing personalized interventions based on learners’ perceived difficulty. The latter part of this research adds real-time facial analysis from the front camera in addition to the PPG sensing from the back camera. AttentiveLearner2 achieves more robust emotion inference (average accuracy = 84.4%) in mobile MOOC learning. According to a longitudinal study with 28 subjects for three weeks, AttentiveReview2, with the multimodal sensing component, improves learning gain by 28.0% with high usability ratings (average System Usability Scale = 80.5). Finally, I show that technologies in this dissertation not only benefit MOOC learning, but also other emerging areas such as computational advertising and behavior targeting. AttentiveVideo, building on top of the sensing architecture in AttentiveLearner2, quantifies emotional responses to mobile video advertisements. In a 24-participant study, AttentiveVideo achieved good accuracy on a wide range of emotional measures (best accuracy = 82.6% across 9 measures)

Improving Mobile MOOC Learning via Implicit Physiological Signal Sensing

Massive Open Online Courses (MOOCs) are becoming a promising solution for delivering high- quality education on a large scale at low cost in recent years. Despite the great potential, today’s MOOCs also suffer from challenges such as low student engagement, lack of personalization, and most importantly, lack of direct, immediate feedback channels from students to instructors. This dissertation explores the use of physiological signals implicitly collected via a "sensorless" approach as a rich feedback channel to understand, model, and improve learning in mobile MOOC contexts. I first demonstrate AttentiveLearner, a mobile MOOC system which captures learners' physiological signals implicitly during learning on unmodified mobile phones. AttentiveLearner uses on-lens finger gestures for video control and monitors learners’ photoplethysmography (PPG) signals based on the fingertip transparency change captured by the back camera. Through series of usability studies and follow-up analyses, I show that the tangible video control interface of AttentiveLearner is intuitive to use and easy to operate, and the PPG signals implicitly captured by AttentiveLearner can be used to infer both learners’ cognitive states (boredom and confusion levels) and divided attention (multitasking and external auditory distractions). Building on top of AttentiveLearner, I design, implement, and evaluate a novel intervention technology, Context and Cognitive State triggered Feed-Forward (C2F2), which infers and responds to learners’ boredom and disengagement events in real time via a combination of PPG-based cognitive state inference and learning topic importance monitoring. C2F2 proactively reminds a student of important upcoming content (feed-forward interventions) when disengagement is detected. A 48-participant user study shows that C2F2 on average improves learning gains by 20.2% compared with a non-interactive baseline system and is especially effective for bottom performers (improving their learning gains by 41.6%). Finally, to gain a holistic understanding of the dynamics of MOOC learning, I investigate the temporal dynamics of affective states of MOOC learners in a 22 participant study. Through both a quantitative analysis of the temporal transitions of affective states and a qualitative analysis of subjective feedback, I investigate differences between mobile MOOC learning and complex learning activities in terms of affect dynamics, and discuss pedagogical implications in detail

Scalable Teaching and Learning via Intelligent User Interfaces

The increasing demand for higher education and the educational budget cuts lead to large class sizes. Learning at scale is also the norm in Massive Open Online Courses (MOOCs). While it seems cost-effective, the massive scale of class challenges the adoption of proven pedagogical approaches and practices that work well in small classes, especially those that emphasize interactivity, active learning, and personalized learning. As a result, the standard teaching approach in today’s large classes is still lectured-based and teacher-centric, with limited active learning activities, and with relatively low teaching and learning effectiveness. This dissertation explores the usage of Intelligent User Interfaces (IUIs) to facilitate the efficient and effective adoption of the tried-and-true pedagogies at scale. The first system is MindMiner, an instructor-side data exploration and visualization system for peer review understanding. MindMiner helps instructors externalize and quantify their subjective domain knowledge, interactively make sense of student peer review data, and improve data exploration efficiency via distance metric learning. MindMiner also helps instructors generate customized feedback to students at scale. We then present BayesHeart, a probabilistic approach for implicit heart rate monitoring on smartphones. When integrated with MOOC mobile clients, BayesHeart can capture learners’ heart rates implicitly when they watch videos. Such information is the foundation of learner attention/affect modeling, which enables a ‘sensorless’ and scalable feedback channel from students to instructors. We then present CourseMIRROR, an intelligent mobile system integrated with Natural Language Processing (NLP) techniques that enables scalable reflection prompts in large classrooms. CourseMIRROR 1) automatically reminds and collects students’ in-situ written reflections after each lecture; 2) continuously monitors the quality of a student’s reflection at composition time and generates helpful feedback to scaffold reflection writing; 3) summarizes the reflections and presents the most significant ones to both instructors and students. Last, we present ToneWars, an educational game connecting Chinese as a Second Language (CSL) learners with native speakers via collaborative mobile gameplay. We present a scalable approach to enable authentic competition and skill comparison with native speakers by modeling their interaction patterns and language skills asynchronously. We also prove the effectiveness of such modeling in a longitudinal study

PhysioKit: An Open-Source, Low-Cost Physiological Computing Toolkit for Single- and Multi-User Studies

The proliferation of physiological sensors opens new opportunities to explore interactions, conduct experiments and evaluate the user experience with continuous monitoring of bodily functions. Commercial devices, however, can be costly or limit access to raw waveform data, while low-cost sensors are efforts-intensive to setup. To address these challenges, we introduce PhysioKit, an open-source, low-cost physiological computing toolkit. PhysioKit provides a one-stop pipeline consisting of (i) a sensing and data acquisition layer that can be configured in a modular manner per research needs, and (ii) a software application layer that enables data acquisition, real-time visualization and machine learning (ML)-enabled signal quality assessment. This also supports basic visual biofeedback configurations and synchronized acquisition for co-located or remote multi-user settings. In a validation study with 16 participants, PhysioKit shows strong agreement with research-grade sensors on measuring heart rate and heart rate variability metrics data. Furthermore, we report usability survey results from 10 small-project teams (44 individual members in total) who used PhysioKit for 4–6 weeks, providing insights into its use cases and research benefits. Lastly, we discuss the extensibility and potential impact of the toolkit on the research community

Bayesian Knowledge Tracing for Navigation through Marzano’s Taxonomy

In this paper we propose a theoretical model of an ITS (Intelligent Tutoring Systems) capable of improving and updating computer-aided navigation based on Bloom’s taxonomy. For this we use the Bayesian Knowledge Tracing algorithm, performing an adaptive control of the navigation among different levels of cognition in online courses. These levels are defined by a taxonomy of educational objectives with a hierarchical order in terms of the control that some processes have over others, called Marzano’s Taxonomy, that takes into account the metacognitive system, responsible for the creation of goals as well as strategies to fulfill them. The main improvements of this proposal are: 1) An adaptive transition between individual assessment questions determined by levels of cognition. 2) A student model based on the initial response of a group of learners which is then adjusted to the ability of each learner. 3) The promotion of metacognitive skills such as goal setting and self-monitoring through the estimation of attempts required to pass the levels. One level of Marzano's taxonomy was left in the hands of the human teacher, clarifying that a differentiation must be made between the tasks in which an ITS can be an important aid and in which it would be more difficult

O USO DE COMPUTAÇÃO AFETIVA EM MOOCS: UM MAPEAMENTO SISTEMÁTICO

The current growth of technologies reflected in different areas, with regard to education, the growth of virtual teaching modalities and the interest in training is remarkable. In this context, MOOCs bring new teaching and learning methods, but still with a high dropout rate. Studies report that affective states play an important role in learning, affectivity is highlighted here, as, applied to education, it allows the teacher to design the course according to the student's affectivity. Therefore, this paper investigate how the use of affective can help in the creation of MOOCs, verifying its influence on the engagement of students and without the number of graduates. The review was carried out following the methodology of systematic mapping and the results show that the use of affective in MOOCs is an area that is still little addressed in Brazil, and the conduction of researches that investigate this experience allows for an understanding of courses and students.El crecimiento actual de las tecnologías se refleja en diferentes áreas, en lo que respecta a la educación, es destacable el crecimiento de las modalidades de enseñanza virtual y el interés por la formación. En este contexto, los MOOC (Massive Open Online Courses) traen nuevos métodos de enseñanza y aprendizaje, pero aún con una alta tasa de deserción. Los estudios informan que los estados afectivos juegan un papel importante en el aprendizaje. Se pone énfasis en la computación afectiva aplicada a la educación, que permite al docente diseñar el curso de acuerdo a la afectividad del alumno. Por lo tanto, este trabajo busca investigar cómo el uso de la computación afectiva puede ayudar a crear MOOCs, verificando su influencia en el engagement de los estudiantes y el número de egresados. La revisión se llevó a cabo siguiendo la metodología de mapeo sistemático y los resultados muestran que el uso de la computación afectiva en los MOOC es un área aún poco abordada en Brasil, y la conducción de investigaciones que investiguen esta experiencia permite comprender los cursos y estudiantes.O atual crescimento das tecnologias reflete-se em diferentes âmbitos, com relação à educação é notável o crescimento das modalidades de ensino virtuais e o interesse em capacitações. Nesse contexto, os MOOCs (Massive Open Online Courses) trazem novos métodos de ensino e aprendizagem, porém, ainda com um alto índice de evasão. Estudos relatam que os estados afetivos têm papel importante na aprendizagem. Destaca-se aqui a computação afetiva, pois aplicada à educação, possibilita ao professor projetar o curso conforme a afetividade do aluno. Portanto, este trabalho busca investigar como o uso de computação afetiva pode auxiliar na criação de MOOCs, verificando sua influência no engajamento dos estudantes e no número de concluintes. A revisão foi realizada seguindo a metodologia do mapeamento sistemático e os resultados mostram que o uso de computação afetiva em MOOCs é uma área ainda pouco abordada no Brasil, e a realização de pesquisas que investiguem essa experiência possibilita entendimento sobre os cursos e estudantes

Multimodal data as a means to understand the learning experience

publishedVersionPaid Open AccessUNIT-agreemen

Experimental Studies in Learning Technology and Child–Computer Interaction

This book is about the ways in which experiments can be employed in the context of research on learning technologies and child–computer interaction (CCI). It is directed at researchers, supporting them to employ experimental studies while increasing their quality and rigor. The book provides a complete and comprehensive description on how to design, implement, and report experiments, with a focus on and examples from CCI and learning technology research. The topics covered include an introduction to CCI and learning technologies as interdisciplinary fields of research, how to design educational interfaces and visualizations that support experimental studies, the advantages and disadvantages of a variety of experiments, methodological decisions in designing and conducting experiments (e.g. devising hypotheses and selecting measures), and the reporting of results. As well, a brief introduction on how contemporary advances in data science, artificial intelligence, and sensor data have impacted learning technology and CCI research is presented. The book details three important issues that a learning technology and CCI researcher needs to be aware of: the importance of the context, ethical considerations, and working with children. The motivation behind and emphasis of this book is helping prospective CCI and learning technology researchers (a) to evaluate the circumstances that favor (or do not favor) the use of experiments, (b) to make the necessary methodological decisions about the type and features of the experiment, (c) to design the necessary “artifacts” (e.g., prototype systems, interfaces, materials, and procedures), (d) to operationalize and conduct experimental procedures to minimize potential bias, and (e) to report the results of their studies for successful dissemination in top-tier venues (such as journals and conferences). This book is an open access publication