Managing Learner’s Affective States in Intelligent Tutoring Systems

Abstract. Recent works in Computer Science, Neurosciences, Education, and Psychology have shown that emotions play an important role in learning. Learner’s cognitive ability depends on his emotions. We will point out the role of emotions in learning, distinguishing the different types and models of emotions which have been considered until now. We will address an important issue con-cerning the different means to detect emotions and introduce recent approaches to measure brain activity using Electroencephalograms (EEG). Knowing the influ-ence of emotional events on learning it becomes important to induce specific emo-tions so that the learner can be in a more adequate state for better learning or memorization. To this end, we will introduce the main components of an emotion-ally intelligent tutoring system able to recognize, interpret and influence learner’s emotions. We will talk about specific virtual agents that can influence learner’s emotions to motivate and encourage him and involve a more cooperative work, particularly in narrative learning environments. Pushing further this paradigm, we will present the advantages and perspectives of subliminal learning which inter

Amorçage cognitif pour amélioration de l’acquisition de la connaissance dans un système tutoriel intelligent

Cette thèse vise à définir une nouvelle méthode d’enseignement pour les systèmes tutoriels intelligents dans le but d’améliorer l’acquisition des connaissances. L’apprentissage est un phénomène complexe faisant intervenir des mécanismes émotionnels et cognitifs de nature consciente et inconsciente. Nous nous intéressons à mieux comprendre les mécanismes inconscients du raisonnement lors de l’acquisition des connaissances. L’importance de ces processus inconscients pour le raisonnement est bien documentée en neurosciences, mais demeure encore largement inexplorée dans notre domaine de recherche. Dans cette thèse, nous proposons la mise en place d’une nouvelle approche pédagogique dans le domaine de l’éducation implémentant une taxonomie neuroscientifique de la perception humaine. Nous montrons que cette nouvelle approche agit sur le raisonnement et, à tour de rôle, améliore l’apprentissage général et l’induction de la connaissance dans un environnement de résolution de problème. Dans une première partie, nous présentons l’implémentation de notre nouvelle méthode dans un système tutoriel visant à améliorer le raisonnement pour un meilleur apprentissage. De plus, compte tenu de l’importance des mécanismes émotionnels dans l’apprentissage, nous avons également procédé dans cette partie à la mesure des émotions par des capteurs physiologiques. L’efficacité de notre méthode pour l’apprentissage et son impact positif observé sur les émotions a été validée sur trente et un participants. Dans une seconde partie, nous allons plus loin dans notre recherche en adaptant notre méthode visant à améliorer le raisonnement pour une meilleure induction de la connaissance. L’induction est un type de raisonnement qui permet de construire des règles générales à partir d’exemples spécifiques ou de faits particuliers. Afin de mieux comprendre l’impact de notre méthode sur les processus cognitifs impliqués dans ce type de raisonnement, nous avons eu recours à des capteurs cérébraux pour mesurer l’activité du cerveau des utilisateurs. La validation de notre approche réalisée sur quarante-trois volontaires montre l’efficacité de notre méthode pour l’induction de la connaissance et la viabilité de mesurer le raisonnement par des mesures cérébrales suite à l’application appropriée d’algorithmes de traitement de signal. Suite à ces deux parties, nous clorons la thèse par une discussion applicative en décrivant la mise en place d’un nouveau système tutoriel intelligent intégrant les résultats de nos travaux.This thesis aims at defining a new learning method to improve knowledge acquisition for intelligent tutoring systems. Learning is a complex phenomenon interlinking both emotional and cognitive mechanisms on conscious and unconscious levels. We are interested in understanding the unconscious mechanisms involved in human reasoning for knowledge acquisition. The importance of these unconscious processes is well documented in neuroscience, but remains largely unexplored in our research field. In this thesis, we put forward a new pedagogical approach in the field of education based on a taxonomy of human perception in neuroscience. We show that this method improves on reasoning which in turn enhances overall learning and inductive capabilities for knowledge acquisition in a problem solving environment. In a first part, we present the implementation of our new method in a tutorial system to improve reasoning hence leading to better learning. Furthermore, acknowledging the importance of emotional mechanisms in learning, we therefore recorded, in this first part, the emotional reactions of users using physiological sensors. The effectiveness of our method for learning and its positive impact on emotions has been validated on 31 participants. In a second part, we go further in our research and enhance our approach in order to improve reasoning for a better induction of knowledge. Induction in a bottom-up logical reasoning approach where one constructs general rules from observed examples. To better understand the impact of our method on the cognitive processes involved in this type of reasoning, we used sensors to record the users’ brain electrical activity. The validation of our approach was carried out on a cohort of 43 volunteers. We showed the effectiveness of our method on the induction of knowledge and sustainability of measuring user’s reasoning by brain recordings after applying proper signal processing algorithms to the data. Following the two parts, we finish the thesis by presenting the implementation of a new intelligent tutoring system incorporating the results found throughout this work

Détection et amélioration de l'état cognitif de l'apprenant

Cette thèse vise à détecter et améliorer l’état cognitif de l’apprenant. Cet état est défini par la capacité d’acquérir de nouvelles connaissances et de les stocker dans la mémoire. Nous nous sommes essentiellement intéressés à améliorer le raisonnement des apprenants, et ceci dans trois environnements : environnement purement cognitif Logique, jeu sérieux LewiSpace et jeu sérieux intelligent Inertia. La détection de cet état se fait essentiellement par des mesures physiologiques (en particulier les électroencéphalogrammes) afin d’avoir une idée sur les interactions des apprenants et l’évolution de leurs états mentaux. L’amélioration des performances des apprenants et de leur raisonnement est une clé pour la réussite de l’apprentissage. Dans une première partie, nous présentons l’implémentation de l’environnement cognitif logique. Nous décrivons des statistiques faites sur cet environnement. Nous avons collecté durant une étude expérimentale les données sur l’engagement, la charge cognitive et la distraction. Ces trois mesures se sont montrées efficaces pour la classification et la prédiction des performances des apprenants. Dans une deuxième partie, nous décrivons le jeu Lewispace pour l’apprentissage des diagrammes de Lewis. Nous avons mené une étude expérimentale et collecté les données des électroencéphalogrammes, des émotions et des traceurs de regard. Nous avons montré qu’il est possible de prédire le besoin d’aide dans cet environnement grâce à ces mesures physiologiques et des algorithmes d’apprentissage machine. Dans une troisième partie, nous clôturons la thèse en présentant des stratégies d’aide intégrées dans un jeu virtuel Inertia (jeu de physique). Cette dernière s’adapte selon deux mesures extraites des électroencéphalogrammes (l’engagement et la frustration). Nous avons montré que ce jeu permet d’augmenter le taux de réussite dans ses missions, la performance globale et par conséquent améliorer l’état cognitif de l’apprenant.This thesis aims at detecting and enhancing the cognitive state of a learner. This state is measured by the ability to acquire new knowledge and store it in memory. Focusing on three types of environments to enhance reasoning: environment Logic, serious game LewiSpace and intelligent serious game Inertia. Physiological measures (in particular the electroencephalograms) have been taken in order to measure learners’ engagement and mental states. Improving learners’ reasoning is key for successful learning process. In a first part, we present the implementation of logic environment. We present statistics on this environment, with data collected during an experimental study. Three types of data: engagement, workload and distraction, these measures were effective and can predict and classify learner’s performance. In a second part, we describe the LewiSpace game, aimed at teaching Lewis diagrams. We conducted an experimental study and collected data from electroencephalograms, emotions and eye-tracking software. Combined with machine learning algorithms, it is possible to anticipate a learner’s need for help using these data. In a third part, we finish by presenting some assistance strategies in a virtual reality game called Inertia (to teach Physics). The latter adapts according to two measures extracted from electroencephalograms (frustration and engagement). Based on our study, we were able to enhance the learner’s success rate on game missions, by improving its cognitive state

Seven HCI Grand Challenges

This article aims to investigate the Grand Challenges which arise in the current and emerging landscape of rapid technological evolution towards more intelligent interactive technologies, coupled with increased and widened societal needs, as well as individual and collective expectations that HCI, as a discipline, is called upon to address. A perspective oriented to humane and social values is adopted, formulating the challenges in terms of the impact of emerging intelligent interactive technologies on human life both at the individual and societal levels. Seven Grand Challenges are identified and presented in this article: Human-Technology Symbiosis; Human-Environment Interactions; Ethics, Privacy and Security; Well-being, Health and Eudaimonia; Accessibility and Universal Access; Learning and Creativity; and Social Organization and Democracy. Although not exhaustive, they summarize the views and research priorities of an international interdisciplinary group of experts, reflecting different scientific perspectives, methodological approaches and application domains. Each identified Grand Challenge is analyzed in terms of: concept and problem definition; main research issues involved and state of the art; and associated emerging requirements

Affective Computing

This book provides an overview of state of the art research in Affective Computing. It presents new ideas, original results and practical experiences in this increasingly important research field. The book consists of 23 chapters categorized into four sections. Since one of the most important means of human communication is facial expression, the first section of this book (Chapters 1 to 7) presents a research on synthesis and recognition of facial expressions. Given that we not only use the face but also body movements to express ourselves, in the second section (Chapters 8 to 11) we present a research on perception and generation of emotional expressions by using full-body motions. The third section of the book (Chapters 12 to 16) presents computational models on emotion, as well as findings from neuroscience research. In the last section of the book (Chapters 17 to 22) we present applications related to affective computing

Designing Embodied Interactive Software Agents for E-Learning: Principles, Components, and Roles

Embodied interactive software agents are complex autonomous, adaptive, and social software systems with a digital embodiment that enables them to act on and react to other entities (users, objects, and other agents) in their environment through bodily actions, which include the use of verbal and non-verbal communicative behaviors in face-to-face interactions with the user. These agents have been developed for various roles in different application domains, in which they perform tasks that have been assigned to them by their developers or delegated to them by their users or by other agents. In computer-assisted learning, embodied interactive pedagogical software agents have the general task to promote human learning by working with students (and other agents) in computer-based learning environments, among them e-learning platforms based on Internet technologies, such as the Virtual Linguistics Campus (www.linguistics-online.com). In these environments, pedagogical agents provide contextualized, qualified, personalized, and timely assistance, cooperation, instruction, motivation, and services for both individual learners and groups of learners. This thesis develops a comprehensive, multidisciplinary, and user-oriented view of the design of embodied interactive pedagogical software agents, which integrates theoretical and practical insights from various academic and other fields. The research intends to contribute to the scientific understanding of issues, methods, theories, and technologies that are involved in the design, implementation, and evaluation of embodied interactive software agents for different roles in e-learning and other areas. For developers, the thesis provides sixteen basic principles (Added Value, Perceptible Qualities, Balanced Design, Coherence, Consistency, Completeness, Comprehensibility, Individuality, Variability, Communicative Ability, Modularity, Teamwork, Participatory Design, Role Awareness, Cultural Awareness, and Relationship Building) plus a large number of specific guidelines for the design of embodied interactive software agents and their components. Furthermore, it offers critical reviews of theories, concepts, approaches, and technologies from different areas and disciplines that are relevant to agent design. Finally, it discusses three pedagogical agent roles (virtual native speaker, coach, and peer) in the scenario of the linguistic fieldwork classes on the Virtual Linguistics Campus and presents detailed considerations for the design of an agent for one of these roles (the virtual native speaker)

Design and Development of a Real-Time Bio-Sensing System Assessing Student Mental Workload and Engagement

Ο εντοπισμός του επακριβούς επιπέδου προσήλωσης και εμπλοκής των μαθητών με το περιεχόμενο διδασκαλίας στην τάξη είναι ένας από τους πιο μεγαλεπήβολους στόχους των ερευνητών της εκπαιδευτικής και επιστημονικής κοινότητας. (Lang, 1995, Grossberg, 1987). Σχετικές διεπιστημονικές ερευνητικές προσπάθειες προσαύξησης ενδιαφέροντος και εντοπισμού της αποτελεσματικότητας των διδακτικών πρακτικών βασίζονται σε τυπικές μελέτες από τον χώρο της ψυχολογίας, της παιδαγωγικής, της παιδοψυχολογίας και της ψυχοφυσιολογίας. Νέες τεχνολογίες έχουν εισάγει διαγνωστικές συσκευές δανεισμένες από τον χώρο της ιατρικής με σκοπό να εκμεταλλευτούν τις δυνατότητες μετρήσεων βιολογικών σημάτων τα οποία αποτελούν επιβεβαιωμένες εκφράσεις ψυχοφυσιολογικών καταστάσεων οι οποίες μπορούν να μεταφραστούν σε εκδηλώσεις διέγερσης και διάθεσης. Οι ιατρικές συσκευές απαιτούν εργαστηριακό περιβάλλον λόγω των αναγκών χρήσης ηλεκτροδίων, κινητικών περιορισμών, συγχρονισμού και ομοιομορφίας των στοιχείων που προκύπτουν και γι’ αυτό τον λόγο δεν μπόρεσαν ποτέ να αποδόσουν μια προσιτή λύση εφαρμόσιμη ευρύτερα σε εκπαιδευτικό περιβάλλον. Στην παρούσα μελέτη, αναλύονται οι επιδόσεις μιας ειδικά κατασκευασμένης ηλεκτρονικής συσκευής, σχεδιασμένης ώστε να εξεταστούν οι δυνατότητες να εξαχθούν δείκτες ψυχοσωματικών εκφράσεων του χρήστη, με την δυνατότητα να χρησιμοποιείται εύχρηστα στην τάξη χωρίς ηλεκτρόδια και επηρεασμούς από προσαρτήσεις. Το ολοκληρωμένο σύστημα μέτρησης και αποτύπωσης συμπερασμάτων είναι βασισμένο σε μοντελοποίηση συμπεριφορών αλλαγής του καρδιακού παλμού και της ειδικής διηλεκτρικής αγωγιμότητας του δέρματος σε πραγματικό χρόνο. Η συσκευή χρησιμοποιεί οπτικούς και διηλεκτρικούς αισθητήρες επαφής και έχει μελετηθεί σε αντιπαραβολή με διαβαθμισμένα περιβάλλοντα προκλητών καταστάσεων νοητικής φόρτισης. Σειρές πειραματικών διαδικασιών εφαρμοσμένες σε διαβαθμισμένα σενάρια πρόκλησης ψυχοσωματικών διεγέρσεων έχουν ολοκληρωθεί για επικύρωση, μελέτη επιδόσεων και λειτουργία του συστήματος ακόμη και σε σύγκριση με εμπορικό προϊόν. Πειραματικά αποτελέσματα δείχνουν αξιόλογους συσχετισμούς του μοντέλου και των επιδόσεων του συστήματος με τις αναμενόμενες αποκρίσεις με ενθαρρυντικά ποσοστά ακρίβειας.Facing the challenge of improving adaptive interaction in educational technologies scientists and educators have turned their focal point in diverse areas ranging from educational, teaching and behavioural psychology to cognitive, affective and perceptual neuroscience. The introduction of digital technologies and interactive media tools in education has shown improved learning efficiency, much higher memory activation and assimilation than verbal teaching, notably due to enhancing motivation achieved by employing approaches attracting student’s attention. Excelling aspects of audio visual presentation proved highly valuable particularly in classes with multi ethnic groups of students, as for example consistency between definitions and objects which were verbally and visually defined, eliminating possible misconceptions caused by mishearing or misinterpretation by the learner. Taking it all one step further as to how an educational system could be even more efficient, a new element would be needed revealing a credible judgment of learning scores and effectiveness of the learning process instantaneously as for example inner levels of activation and satisfaction. In fact, this could be made possible using existing technologies if subconscious neurophysiological responses of a learner could be ascertained and inferred to psycho-somatic conditions as they occur. A system including bio-sensing, data analysis and processing in real time able to provide quantified markers of psychosomatic states of a learner would help enormously in next generations of educational practice. Incorporating data of student engagement and active involvement could help to deduce the interest of a learner, which is known to improve sensitisation in implicit, incidental and also in classical learning. Experimental settings used in previous studies attempting to incorporate physiological responses and interpretations into responsive educational settings have faced major obstacles. Operational issues caused by the requirements of the devices used for the acquisition of physiological signals such as electrodes and movement restrictions have reduced the progress of such settings to laboratory environments. In such settings as described above, the effects of wiring harnesses and sensory components produced an additional psychological burden on the participants. Consequently, the need to approach the physiological data acquisition from a new angle with seamless and unnoticeable operation is apparent. The challenge to design, develop and validate a system that being minimally obstructive and literally unnoticed by the user would uncover combined subconscious expressions of a learner was the primary objective of this research. Physiological data of Heart Rate and Skin Trans-Conductance (Electro-dermal Response) elected as vitally important and highly appropriate to produce the input of data required to evaluate a behavioural concept model. The behavioural assessment model entailed vector classifiers producing directional interpretations of measurements. Directional information (Gradient response) has been derived by comparison of measurements to previously measured values in real time. Assessing the effectiveness and accuracy of the adopted model to deduce attention and engagement of a learner in real time formed the second major objective. For this purpose, a series of relevant experimental methodologies have been employed. Data produced using formal personality assessments have also been investigated in conjunction with those derived from physiological responses in order to identify personality related particularities. The final part of this work has been supplemented by propositions and suggestions with regards to various applications of the system in accomplishment of the initial aims