Managing Learner’s Affective States in Intelligent Tutoring Systems

Abstract. Recent works in Computer Science, Neurosciences, Education, and Psychology have shown that emotions play an important role in learning. Learner’s cognitive ability depends on his emotions. We will point out the role of emotions in learning, distinguishing the different types and models of emotions which have been considered until now. We will address an important issue con-cerning the different means to detect emotions and introduce recent approaches to measure brain activity using Electroencephalograms (EEG). Knowing the influ-ence of emotional events on learning it becomes important to induce specific emo-tions so that the learner can be in a more adequate state for better learning or memorization. To this end, we will introduce the main components of an emotion-ally intelligent tutoring system able to recognize, interpret and influence learner’s emotions. We will talk about specific virtual agents that can influence learner’s emotions to motivate and encourage him and involve a more cooperative work, particularly in narrative learning environments. Pushing further this paradigm, we will present the advantages and perspectives of subliminal learning which inter

An investigation of the effect of music upon the academic, affective, and attendance profiles of selected fourth grade students

The purpose of this study was to determine if there is an effect on the academic, affective and attendance profiles of selected fourth grade students when baroque music is introduced subliminally into the classroom intermittently or continuously throughout the school day. Students were randomly assigned to three classrooms with one of three treatments: music continuously, music intermittently, or a no treatment, control group. Fourteen measures from the Children\u27s Personality Questionnaire, two measures from the Stanford Achievement Test, and attendance and discipline records were used to assess outcomes of this study. Analysis of variance with repeated measures revealed a significant post test score in the sub-test for tension on the children\u27s Personality Questionnaire, showing students in the classroom with no music becoming more tense than those in either classroom with music. Analysis of Variance revealed significantly more absences in the classroom with music continuously than in that with music intermittently or with no music. If questions about the possibility of increasing absences can be addressed, this study might be replicated with a larger population for further investigation of significant results

Processus cérébraux adaptés aux systèmes tutoriels intelligents

Le module de l'apprenant est l'une des composantes les plus importantes d’un Système Tutoriel Intelligent (STI). L'extension du modèle de l'apprenant n'a pas cessé de progresser. Malgré la définition d’un profil cognitif et l’intégration d’un profil émotionnel, le module de l’apprenant demeure non exhaustif. Plusieurs senseurs physiologiques sont utilisés pour raffiner la reconnaissance des états cognitif et émotionnel de l’apprenant mais l’emploi simultané de tous ces senseurs l’encombre. De plus, ils ne sont pas toujours adaptés aux apprenants dont les capacités sont réduites. Par ailleurs, la plupart des stratégies pédagogiques exécutées par le module du tuteur ne sont pas conçues à la base d’une collecte dynamique de données en temps réel, cela diminue donc de leur efficacité. L’objectif de notre recherche est d’explorer l’activité électrique cérébrale et de l’utiliser comme un nouveau canal de communication entre le STI et l’apprenant. Pour ce faire nous proposons de concevoir, d’implémenter et d’évaluer le système multi agents NORA. Grâce aux agents de NORA, il est possible d’interpréter et d’influencer l’activité électrique cérébrale de l’apprenant pour un meilleur apprentissage. Ainsi, NORA enrichit le module apprenant d’un profile cérébral et le module tuteur de quelques nouvelles stratégies neuropédagogiques efficaces. L’intégration de NORA à un STI donne naissance à une nouvelle génération de systèmes tutoriels : les STI Cérébro-sensibles (ou STICS) destinés à aider un plus grand nombre d’apprenants à interagir avec l’ordinateur pour apprendre à gérer leurs émotions, maintenir la concentration et maximiser les conditions favorable à l’apprentissage.The learner module is the most important component within an Intelligent Tutoring System (ITS). The extension of the learner module is still in progress, despite the integration of the cognitive profile and the emotional profile, it is not yet exhaustive. To improve the prediction of the learner’s emotional and cognitive states, many physiological sensors have been used, but all of these sensors are cumbersome. In addition, they are not always adapted to the learners with reduced capacities. Beside, most of the pedagogical strategies that are executed by the tutor module are based on no-live collections of data. This fact reduces their efficiency. The objective of our research is to explore the electrical brain activity and use it as a communication channel between a learner and an ITS. To reach this aim, we suggest to conceive, to implement and to evaluate the multi-agent system NORA. Integrated to an ITS, this one became a Brain Sensitive Intelligent Tutoring System (BS-ITS). Agents of NORA interpret the learner’s brain electrical signal and react to it. The new BS-ITS is the extension of an ITS and enrich the learner module with the brain profile and the tutor module with a new Neuropedagogical Strategies. We aim to reach more categories of learners and help them to manage their stress, anxiety and maintain the concentration, the attention and the interest

Recent Applications in Graph Theory

Graph theory, being a rigorously investigated field of combinatorial mathematics, is adopted by a wide variety of disciplines addressing a plethora of real-world applications. Advances in graph algorithms and software implementations have made graph theory accessible to a larger community of interest. Ever-increasing interest in machine learning and model deployments for network data demands a coherent selection of topics rewarding a fresh, up-to-date summary of the theory and fruitful applications to probe further. This volume is a small yet unique contribution to graph theory applications and modeling with graphs. The subjects discussed include information hiding using graphs, dynamic graph-based systems to model and control cyber-physical systems, graph reconstruction, average distance neighborhood graphs, and pure and mixed-integer linear programming formulations to cluster networks

Brain-Computer Interface

Brain-computer interfacing (BCI) with the use of advanced artificial intelligence identification is a rapidly growing new technology that allows a silently commanding brain to manipulate devices ranging from smartphones to advanced articulated robotic arms when physical control is not possible. BCI can be viewed as a collaboration between the brain and a device via the direct passage of electrical signals from neurons to an external system. The book provides a comprehensive summary of conventional and novel methods for processing brain signals. The chapters cover a range of topics including noninvasive and invasive signal acquisition, signal processing methods, deep learning approaches, and implementation of BCI in experimental problems

Modélisation des émotions de l’apprenant et interventions implicites pour les systèmes tutoriels intelligents

La modélisation de l’expérience de l’utilisateur dans les Interactions Homme-Machine est un enjeu important pour la conception et le développement des systèmes adaptatifs intelligents. Dans ce contexte, une attention particulière est portée sur les réactions émotionnelles de l’utilisateur, car elles ont une influence capitale sur ses aptitudes cognitives, comme la perception et la prise de décision. La modélisation des émotions est particulièrement pertinente pour les Systèmes Tutoriels Émotionnellement Intelligents (STEI). Ces systèmes cherchent à identifier les émotions de l’apprenant lors des sessions d’apprentissage, et à optimiser son expérience d’interaction en recourant à diverses stratégies d’interventions. Cette thèse vise à améliorer les méthodes de modélisation des émotions et les stratégies émotionnelles utilisées actuellement par les STEI pour agir sur les émotions de l’apprenant. Plus précisément, notre premier objectif a été de proposer une nouvelle méthode pour détecter l’état émotionnel de l’apprenant, en utilisant différentes sources d’informations qui permettent de mesurer les émotions de façon précise, tout en tenant compte des variables individuelles qui peuvent avoir un impact sur la manifestation des émotions. Pour ce faire, nous avons développé une approche multimodale combinant plusieurs mesures physiologiques (activité cérébrale, réactions galvaniques et rythme cardiaque) avec des variables individuelles, pour détecter une émotion très fréquemment observée lors des sessions d’apprentissage, à savoir l’incertitude. Dans un premier lieu, nous avons identifié les indicateurs physiologiques clés qui sont associés à cet état, ainsi que les caractéristiques individuelles qui contribuent à sa manifestation. Puis, nous avons développé des modèles prédictifs permettant de détecter automatiquement cet état à partir des différentes variables analysées, à travers l’entrainement d’algorithmes d’apprentissage machine. Notre deuxième objectif a été de proposer une approche unifiée pour reconnaître simultanément une combinaison de plusieurs émotions, et évaluer explicitement l’impact de ces émotions sur l’expérience d’interaction de l’apprenant. Pour cela, nous avons développé une plateforme hiérarchique, probabiliste et dynamique permettant de suivre les changements émotionnels de l'apprenant au fil du temps, et d’inférer automatiquement la tendance générale qui caractérise son expérience d’interaction à savoir : l’immersion, le blocage ou le décrochage. L’immersion correspond à une expérience optimale : un état dans lequel l'apprenant est complètement concentré et impliqué dans l’activité d’apprentissage. L’état de blocage correspond à une tendance d’interaction non optimale où l'apprenant a de la difficulté à se concentrer. Finalement, le décrochage correspond à un état extrêmement défavorable où l’apprenant n’est plus du tout impliqué dans l’activité d’apprentissage. La plateforme proposée intègre trois modalités de variables diagnostiques permettant d’évaluer l’expérience de l’apprenant à savoir : des variables physiologiques, des variables comportementales, et des mesures de performance, en combinaison avec des variables prédictives qui représentent le contexte courant de l’interaction et les caractéristiques personnelles de l'apprenant. Une étude a été réalisée pour valider notre approche à travers un protocole expérimental permettant de provoquer délibérément les trois tendances ciblées durant l’interaction des apprenants avec différents environnements d’apprentissage. Enfin, notre troisième objectif a été de proposer de nouvelles stratégies pour influencer positivement l’état émotionnel de l’apprenant, sans interrompre la dynamique de la session d’apprentissage. Nous avons à cette fin introduit le concept de stratégies émotionnelles implicites : une nouvelle approche pour agir subtilement sur les émotions de l’apprenant, dans le but d’améliorer son expérience d’apprentissage. Ces stratégies utilisent la perception subliminale, et plus précisément une technique connue sous le nom d’amorçage affectif. Cette technique permet de solliciter inconsciemment les émotions de l’apprenant, à travers la projection d’amorces comportant certaines connotations affectives. Nous avons mis en œuvre une stratégie émotionnelle implicite utilisant une forme particulière d’amorçage affectif à savoir : le conditionnement évaluatif, qui est destiné à améliorer de façon inconsciente l’estime de soi. Une étude expérimentale a été réalisée afin d’évaluer l’impact de cette stratégie sur les réactions émotionnelles et les performances des apprenants.Modeling the user’s experience within Human-Computer Interaction is an important challenge for the design and development of intelligent adaptive systems. In this context, a particular attention is given to the user’s emotional reactions, as they decisively influence his cognitive abilities, such as perception and decision-making. Emotion modeling is particularly relevant for Emotionally Intelligent Tutoring Systems (EITS). These systems seek to identify the learner’s emotions during tutoring sessions, and to optimize his interaction experience using a variety of intervention strategies. This thesis aims to improve current methods on emotion modeling, as well as the emotional strategies that are presently used within EITS to influence the learner’s emotions. More precisely, our first objective was to propose a new method to recognize the learner’s emotional state, using different sources of information that allow to measure emotions accurately, whilst taking account of individual characteristics that can have an impact on the manifestation of emotions. To that end, we have developed a multimodal approach combining several physiological measures (brain activity, galvanic responses and heart rate) with individual variables, to detect a specific emotion, which is frequently observed within computer tutoring, namely : uncertainty. First, we have identified the key physiological indicators that are associated to this state, and the individual characteristics that contribute to its manifestation. Then, we have developed predictive models to automatically detect this state from the analyzed variables, trough machine learning algorithm training. Our second objective was to propose a unified approach to simultaneously recognize a combination of several emotions, and to explicitly evaluate the impact of these emotions on the learner’s interaction experience. For this purpose, we have developed a hierarchical, probabilistic and dynamic framework, which allows one to track the learner’s emotional changes over time, and to automatically infer the trend that characterizes his interaction experience namely : flow, stuck or off-task. Flow is an optimal experience : a state in which the learner is completely focused and involved within the learning activity. The state of stuck is a non-optimal trend of the interaction where the learner has difficulty to maintain focused attention. Finally, the off-task behavior is an extremely unfavorable state where the learner is not involved anymore within the learning session. The proposed framework integrates three-modality diagnostic variables that sense the learner’s experience including : physiology, behavior and performance, in conjunction with predictive variables that represent the current context of the interaction and the learner’s personal characteristics. A human-subject study was conducted to validate our approach through an experimental protocol designed to deliberately elicit the three targeted trends during the learners’ interaction with different learning environments. Finally, our third objective was to propose new strategies to positively influence the learner’s emotional state, without interrupting the dynamics of the learning session. To this end, we have introduced the concept of implicit emotional strategies : a novel approach to subtly impact the learner’s emotions, in order to improve his learning experience. These strategies use the subliminal perception, and more precisely a technique known as affective priming. This technique aims to unconsciously solicit the learner’s emotions, through the projection of primes charged with specific affective connotations. We have implemented an implicit emotional strategy using a particular form of affective priming namely : the evaluative conditioning, which is designed to unconsciously enhance self-esteem. An experimental study was conducted in order to evaluate the impact of this strategy on the learners’ emotional reactions and performance

Fusion of musical contents, brain activity and short term physiological signals for music-emotion recognition

In this study we propose a multi-modal machine learning approach, combining EEG and Audio features for music emotion recognition using a categorical model of emotions. The dataset used consists of film music that was carefully created to induce strong emotions. Five emotion categories were adopted: Fear, Anger, Happy, Tender and Sad. EEG data was obtained from three male participants listening to the labeled music excerpts. Feature level fusion was adopted to combine EEG and Audio features. The results show that the multimodal system outperformed the EEG mono modal system. Additionally, we evaluated the contribution of each audio feature in the classification performance of the multimodal system. Preliminary results indicate a significant contribution of individual audio features in the classification accuracy, we also found that various audio features that noticeably contributed in the classification accuracy were also reported in previous research studying the correlation between audio features and emotion ratings using the same dataset.

Fatigue detection system to aid in remote work

During the Covid-19 pandemic there was a noticeable surge in the amount of remote workers. In the aftermath of the pandemic working from home still remains a reality for many workers with noticeable impacts on the mental health of people. With the increased stress caused by current situation and the harder time establishing boundaries there was an increase in the overall stress and fatigue in workers, leading to burnouts. Fatigue detection systems are used in several areas, mainly in the automotive industry as a mean to decrease the number of accidents. This research started by approaching the Artificial Intelligence (AI) area and its domains, followed by a study of the current techniques used in order to predict fatigue. With the main ones utilising eye state, facial landmarks, electrocardiogram or heart rate. After a research into existing Fatigue detection systems was done in order to identify the strengths of solutions currently in the market, whether in the automotive industry or other applications. This thesis proposes the creation of a system able to detect fatigue in a user as well as warn him when fatigue levels increase. This system incorporates a webcam analysing the users face and performing eye state detection in order to calculate the percentage of the time the eyes are closed (PERCLOS). Heart rate data was also analysed and a model was developed in order to incorporate this data, the percentage of time the eyes are closed, the program the user has open and time of day in order to predict the level of fatigue. By combining these two different techniques this system can be more effective and more accurate in giving predictions of the level of fatigue. The review of literature showed that the conjunction of these two techniques in predicting fatigue is novelty. The developed system also contains integration with smartwatch technology in order to both harness heart rate data as well as communicate with the user via pop up notifications to inform him when fatigue levels get too high. The conclusion of this work is that eye state detection using Artificial Intelligence can achieve a high accuracy and be a reliable tool in identifying fatigue in an user. The combination of Heart Rate and PERCLOS allows the system to have a higher accuracy as well as not being completely reliant on one sensor. The creation of a fatigue prediction model was hindered by the lack of existent data in order to train a model, a problem that could be fixed with the adoption of the system in a broader scope.Durante a pandemia de Covid-19, houve um aumento notável na quantidade de trabalhadores remotos. No rescaldo da pandemia, trabalhar a partir de casa continua a ser uma realidade para muitos trabalhadores, com impactos visíveis na saúde mental das pessoas. Com o aumento do stresse causado pela situação atual e a dificuldade de estabelecer limites, houve um aumento do stresse geral e da fadiga dos trabalhadores, levando ao esgotamento. Os sistemas de detecção de fadiga são utilizados em diversas áreas, principalmente na indústria automobilística como forma de diminuir o número de acidentes. Este estudo começou por abordar a área de Inteligência Artificial (IA) e os seus domínios, seguida de um estudo das técnicas atuais utilizadas para prever a fadiga. Com os principais utilizando o estado dos olhos, pontos de referência faciais, eletrocardiograma ou frequência cardíaca. Depois foi feita uma pesquisa sobre os sistemas de detecção de fadiga existentes de forma a identificar os pontos fortes das soluções actualmente no mercado, quer seja na indústria automóvel ou outras aplicações. Esta dissertação propõe a criação de um sistema capaz de detectar fadiga num utilizador, bem como alertar quando os níveis de fadiga aumentam. Este sistema incorpora uma webcam que analisa a face do utilizador e realiza a detecção do estado dos olhos para calcular a percentagem de tempo em que os olhos estão fechados (PERCLOS). Os dados de frequência cardíaca também foram analisados e um modelo foi desenvolvido para incorporar estes dados, a percentagem de tempo que os olhos ficam fechados, o programa que o utilizador tem aberto e a hora do dia para prever o nível de fadiga. Ao combinar essas duas técnicas diferentes, este sistema pode ser mais eficaz e mais preciso em fornecer previsões do nível de fadiga. A revisão da literatura mostrou que a conjunção dessas duas técnicas na previsão da fadiga é novidade. O sistema desenvolvido também contém integração com a tecnologia smartwatch para aproveitar os dados da frequência cardíaca e comunicar com o utilizador por meio de notificações pop-up para informá-lo quando os níveis de fadiga se encontrarem altos. A conclusão deste trabalho é que a detecção do estado ocular usando Inteligência Artificial pode alcançar uma alta precisão e ser uma ferramenta confiável na identificação de fadiga num utilizador. A combinação da frequência cardíaca e PERCLOS permite que o sistema tenha maior precisão, além de não depender completamente de um unico sensor. A criação de um modelo de previsão de fadiga foi dificultada pela falta de dados existentes para treinar um modelo, problema que poderia ser colmatado com a adoção do sistema numa população maior

Comparing EEG-neurofeedback visual modalities between screen-based and immersive head-mounted VR

Tese de Mestrado Integrado, Engenharia Biomédica e Biofísica (Sinais e Imagens Médicas), 2022, Universidade de Lisboa, Faculdade de CiênciasNeurofeedback (NF) can be defined as a form of biofeedback that trains subjects to have self-control over brain their functions, by providing real-time feedback of their own cerebral activity. This activity can be presented in various forms, with auditory and visual feedback being the most common. Recently, NF has been investigated as a potential treatment for various clinical conditions associated with abnormal brain activity or cognitive capacities. However, the greater research focus is not discussing how the feedback should be presented. The chosen modality for any NF training system may strongly influence the training protocol and consequently the outcome of the experiment. In this thesis, a systematical comparison between two different type of visual modalities (ScreenBased vs. immersive-virtual reality (VR) ) was performed with the goal to evaluate the effectiveness of each modality on the NF training results. Data from two previous studies, recorded on healthy participants, in protocols that targeted the increase in the upper alpha (UA) band power measured at the EEG electrode Cz was used. This was then divided into two modality groups: Screen-Based modality group (N = 8) and the Immersive-VR group (N = 4). An extensive data processing and cleaning protocol was applied to both groups and the training effectiveness was measured through band power calculation, the definition of learning ability indexes and the application of statistical tests. Results showed that, both groups had a generally positive training effect within sessions, however data regarding different sessions is inconclusive and does not show clear evidence of up-regulation of the target feature. Additionally, when only considering within-session evolution, only the Immersive-VR modality group was able to maintain an increasing trend in all sessions. One of the main limitations of this study was the sample size, which was too small to determine the precise effect of NF training. Future work requires, not only an increase in sample size but also, the definition and incorporation of learning predictors that allow the pre-selection of subjects before the training sessions, in order to prevent high number of non-learners