Communication skills training exploiting multimodal emotion recognition

The teaching of communication skills is a labour-intensive task because of the detailed feedback that should be given to learners during their prolonged practice. This study investigates to what extent our FILTWAM facial and vocal emotion recognition software can be used for improving a serious game (the Communication Advisor) that delivers a web-based training of communication skills. A test group of 25 participants played the game wherein they were requested to mimic specific facial and vocal emotions. Half of the assignments included direct feedback and the other half included no feedback. It was investigated whether feedback on the mimicked emotions would lead to better learning. The results suggest the facial performance growth was found to be positive, particularly significant in the feedback condition. The vocal performance growth was significant in both conditions. The results are a significant indication that the automated feedback from the software improves learners’ communication performances.The Netherlands Laboratory for Lifelong Learning (NELLL) of the Open University Netherland

Propuesta arquitectural para la interoperabilidad entre sistemas multiagente y mundos virtuales 3D

El término Web 3D hace referencia a lenguajes de programación, protocolos, formatos de archivos y tecnologías utilizadas para la creación y presentación de universos tridimensionales interactivos a través de Internet. Por medio de interfaces tridimensionales se logra una aproximación a la realidad virtual, permitiendo la creación de escenarios de simulación, que facilitan la interacción de los seres humanos con ambientes de aprendizaje enriquecidos similares al mundo real. Sin embargo, un entorno grafico llamativo en sí, no garantiza el aprendizaje efectivo, se necesita un seguimiento, evaluación y retroalimentación. Por lo tanto, se propone la integración de entornos tridimensionales con sistemas multi-agentes, que permitan personalizar elproceso de aprendizaje a las necesidades individuales de los estudiantes, organizar y distribuir contenidos de manera eficiente y apoyar reflexiones de los estudiantes sobre su aprendizaje. Diferentes estudios han realizado implementaciones específicas de agentes en mundos virtuales a través de lenguajes de marcas de hipertexto; sin embargo, los modelos propuestos no permiten implementaciones de propósito general. Nuestro modelo dearquitectura integra los comportamientos de los sistemas multi-agente y los mundos virtuales 3D. La validación se realizó a través de una aplicación que integra Java Development Framework Agent desarrollado por Telecom Italia Lab y OpenSimulator

Measuring Human Comprehension from Nonverbal Behaviour using Artificial Neural Networks

This paper presents the adaptation and application of Silent Talker, a psychological profiling system in the measurement of human comprehension through the monitoring of multiple channels of facial nonverbal behaviour using Artificial Neural Networks (ANN). Everyday human interactions are abundant with almost unconscious nonverbal behaviours accounting for approximately 93% of communication, providing a potentially rich source of information once decoded. Existing comprehension assessments techniques are inhibited by inconsistencies, limited to the verbal communication dimension and are often time-consuming with feedback delay. Major weaknesses hinder humans as accurate decoders of nonverbal behaviour with being error prone, inconsistent and poor at simultaneously focusing on multiple channels. Furthermore, human decoders are susceptible to fatigue and require training resulting in a costly, time-consuming process. ANNs are powerful, adaptable, scalable computational models that are able to overcome human decoder and pattern classification weaknesses. Therefore, the neural networks computer-based Silent Talker system has been trained and validated in the measurement of human comprehension using videotaped participant nonverbal behaviour from an informed consent field study. A series of experiments on training backpropagation ANNs with different topologies were conducted. The results show that comprehension and non comprehension patterns exist within the monitored multichannels of facial NVB with both experiments consistently yielding classification accuracies above 80%

Quantification of human operator skill in a driving simulator for applications in human adaptive mechatronics

Nowadays, the Human Machine System (HMS) is considered to be a proven technology, and now plays an important role in various human activities. However, this system requires that only a human has an in-depth understanding of the machine operation, and is thus a one-way relationship. Therefore, researchers have recently developed Human Adaptive Mechatronics (HAM) to overcome this problem and balance the roles of the human and machine in any HMS. HAM is different compared to ordinary HMS in terms of its ability to adapt to changes in its surroundings and the changing skill level of humans. Nonetheless, the main problem with HAM is in quantifying the human skill level in machine manipulation as part of human recognition. Therefore, this thesis deals with a proposed formula to quantify and classify the skill of the human operator in driving a car as an example application between humans and machines. The formula is evaluated using the logical conditions and the definition of skill in HAM in terms of time and error. The skill indices are classified into five levels: Very Highly Skilled, Highly Skilled, Medium Skilled, Low Skilled and Very Low Skilled. Driving was selected because it is considered to be a complex mechanical task that involves skill, a human and a machine. However, as the safety of the human subjects when performing the required tasks in various situations must be considered, a driving simulator was used. The simulator was designed using Microsoft Visual Studio, controlled using a USB steering wheel and pedals, as was able to record the human ii path and include the desired effects on the road. Thus, two experiments involving the driving simulator were performed; 20 human subjects with a varying numbers of years experience in driving and gaming were used in the experiments. In the first experiment, the subjects were asked to drive in Expected and Guided Conditions (EGC). Five guided tracks were used to show the variety of driving skill: straight, circular, elliptical, square and triangular. The results of this experiment indicate that the tracking error is inversely proportional to the elapsed time. In second experiment, the subjects experienced Sudden Transitory Conditions (STC). Two types of unexpected situations in driving were used: tyre puncture and slippery surface. This experiment demonstrated that the tracking error is not directly proportional to the elapsed time. Both experiments also included the correlation between experience and skill. For the first time, a new skill index formula is proposed based on the logical conditions and the definition of skill in HAM

Design emocional e expressão de emoção em agentes tutores

Mestrado em Comunicação MultimédiaA presente investigação visa compreender como é possível estimular e estabelecer relações a longo prazo entre humanos e companheiros virtuais. Neste sentido, pretende-se utilizar a emoção como factor preponderante no estabelecimento desta relação, nomeadamente através da sua expressão e averiguar da sua pertinência neste processo. Para testar a viabilidade deste propósito, foi estabelecido um protocolo com um agrupamento de escolas onde se pretende interagir directamente com os potenciais utilizadores. Com o objectivo de comprovar a hipótese traçada foram entrevistados os alunos de duas escolas de Coimbra que participam na investigação, onde apuramos a sua visão sobre o aspecto físico do companheiro virtual e a correcta associação de expressões faciais com o estado emocional do companheiro. Os dados recolhidos revelam que o aspecto humano do companheiro, as cores do vestuário e as expressões faciais são elementos-chave na interacção para os seus utilizadores, uma vez que oferecem familiarização e reconhecimento de características humanas no agente virtual, o que, por sua vez, proporciona uma interacção mais natural e motivante. ABSTRACT: This investigation aims to understand how it is possible to establish long term relationships between humans and virtual companions. Therefore we intend to use emotion as a key element in the establishment of this relationship, namely through its expression and determine its relevance in this process. To test the viability of this purpose it has been established a protocol with a group of schools where we intend to interact directly with potential users. To validate the hypotheses we interviewed children from two schools of Coimbra who participate in the investigation. With the interviews we tried to understand their vision over our virtual companion’s body and if they could correctly identify the companion’s emotional state through its facial expression. The collected data suggest that the companion’s human body, the costume’s colour and its facial expression are vital issues in the interaction, they provide the user with a sense of familiarization and recognition of human features in the virtual agent which allows the user to have a more natural and engaging experience

Enhancing electronic intelligent tutoring systems by responding to affective states

The overall aim of this research is the exploration mechanisms which allow an understanding of the emotional state of students and the selection of an appropriate cognitive and affective feedback for students on the basis of students' emotional state and cognitive state in an affective learning environment. The learning environment in which this research is based is one in which students learn by watching an instructional video. The main contributions in the thesis include: - A video study was carried out to gather data in order to construct the emotional models in this research. This video study adopted a methodology in qualitative research called “Quick and Dirty Ethnography”(Hughes et al., 1995). In the video study, the emotional states, including boredom, frustration, confusion, flow, happiness, interest, were identified as being the most important to a learner in learning. The results of the video study indicates that blink frequencies can reflect the learner's emotional states and it is necessary to intervene when students are in self-learning through watching an instructional video in order to ensure that attention levels do not decrease. - A novel emotional analysis model for modeling student’s cognitive and emotional state in an affective learning system was constructed. It is an appraisal model which is on the basis of an instructional theory called Gagne’s theory (Gagne, 1965). - A novel emotion feedback model for producing appropriate feedback tactics in affective learning system was developed by Ontology and Influence Diagram ii approach. On the basis of the tutor-remediation hypothesis and the self-remediation hypothesis (Hausmann et al., 2013), two feedback tactic selection algorithms were designed and implemented. The evaluation results show: the emotion analysis model can be used to classify negative emotion and hence deduce the learner’s cognitive state; the degree of satisfaction with the feedback based on the tutor-remediation hypothesis is higher than the feedback based on self-remediation hypothesis; the results indicated a higher degree of satisfaction with the combined cognitive and emotional feedback than cognitive feedback on its own

The potential of a classroom network to support teacher feedback:a study in statistics education.

In het wiskundeonderwijs ervaren docenten voortdurend een gebrek aan tijd om hun leerlingen goed te instrueren. In Nederland is daarbij de contacttijd voor wiskunde in de afgelopen vijftien jaar nog eens afgenomen. Wiskunde wordt door leerlingen bovendien als moeilijk ervaren. Dit onderzoek richt zich op de vraag: hoe kunnen we de contacttijd in het wiskundeonderwijs beter benutten? Meta-analyses van leeropbrengsten, zoals die beschreven door Hattie (2009), laten zien dat feedback één van de krachtigste enkelvoudige middelen is om die leeropbrengst te verhogen. In dit onderzoek benutten we de mogelijkheid van grafische rekenmachines (GR), verbonden met de computer van de docent via een draadloos netwerk, om de feedback in wiskundeonderwijs te verbeteren. Enerzijds kregen de leerlingen via hun GR onmiddellijke feedback op bepaalde opgaven en anderzijds gaf de docent, meestal in de volgende les, feedback op het werk van de leerlingen, daarbij ondersteund door een analyse van dat werk door het systeem. Het onderzoek richtte zich in eerste instantie op het ontwikkelen van zogenaamde 'gegevens geletterdheid' bij de leerlingen, waarbij de 'algoritmische vaardigheden' niet vergeten werden. Gedurende vier empirische rondes is deze wijze van werken in negen klassen ontworpen, getest, geëvalueerd en bijgesteld. De wiskundedocenten en hun leerlingen waren over het algemeen enthousiast over het resultaat. Zo adviseren zij bijvoorbeeld om de helft van de lessen aan deze werkvorm te besteden. De docenten geven daarbij aan dat ze een hoge werkdruk hebben ervaren om deze manier van doceren onder de knie te krijgen. De studie expliciteert de voorwaarden waaraan moet worden voldaan voordat de werkwijze succesvol kan zijn. In mathematics education teachers experience a constant lack of time to properly instruct their students. In the Netherlands the contact time for mathematics in secondary education during the last fifteen years again declined. Mathematics is also perceived as difficult by students. This research focuses on the question: how can we better utilize contact time in mathematics education? Meta-analyses of learning outcomes, such as those described by Hattie (2009), show that feedback is one of the most powerful single tools for improving learning achievements. In this study we explore the possibility of graphing calculators (GR), connected to the teacher computer through the use of a wireless network, to improve the feedback in mathematics education. First, students received immediate feedback on their worked out mathematics assignments GR and second, the teacher, usually in the next lesson, gave feedback on the work of the students, supported by an analysis of that work through the system. This study focused primarily on the development of 'data literacy' among students, while the 'algorithmic skills' were not forgotten. In four stages, a prototype of the intervention designed, tested, evaluated and adjusted in nine groups of students. The mathematics teachers and their students are generally enthusiastic about the results. They for instance recommend to spend half of each lesson working this way. Though, the teachers explicitly state that they have experienced a tough workload while mastering this way of teaching. The study makes the conditions to be met before the method can be successful explicit.

Detecting human comprehension from nonverbal behaviour using artificial neural networks

Every day, communication between humans is abundant with an array of nonverbal behaviours. Nonverbal behaviours are signals emitted without using words such as facial expressions, eye gaze and body movement. Nonverbal behaviours have been used to identify a person’s emotional state in previous research. With nonverbal behaviour being continuously available and almost unconscious, it provides a potentially rich source of knowledge once decoded. Humans are weak decoders of nonverbal behaviour due to being error prone, susceptible to fatigue and poor at simultaneously monitoring numerous nonverbal behaviours. Human comprehension is primarily assessed from written and spoken language. Existing comprehension assessments tools are inhibited by inconsistencies and are often time-consuming with feedback delay. Therefore, there is a niche for attempting to detect human comprehension from nonverbal behaviour using artificially intelligent computational models such as Artificial Neural Networks (ANN), which are inspired by the structure and behaviour of biological neural networks such as those found within the human brain. This Thesis presents a novel adaptable system known as FATHOM, which has been developed to detect human comprehension and non-comprehension from monitoring multiple nonverbal behaviours using ANNs. FATHOM’s Comprehension Classifier ANN was trained and validated on human comprehension detection using the errorbackpropagation learning algorithm and cross-validation in a series of experiments with nonverbal datasets extracted from two independent comprehension studies where each participant was digitally video recorded: (1) during a mock informed consent field study and (2) in a learning environment. The Comprehension Classifier ANN repeatedly achieved averaged testing classification accuracies (CA) above 84% in the first phase of the mock informed consent field study. In the learning environment study, the optimised Comprehension Classifier ANN achieved a 91.385% averaged testing CA. Overall, the findings revealed that human comprehension and noncomprehension patterns can be automatically detected from multiple nonverbal behaviours using ANNs