Data Fusion for Real-time Multimodal Emotion Recognition through Webcams and Microphones in E-Learning

The original article is available on the Taylor & Francis Online website in the following link: http://www.tandfonline.com/doi/abs/10.1080/10447318.2016.1159799?journalCode=hihc20This paper describes the validation study of our software that uses combined webcam and microphone data for real-time, continuous, unobtrusive emotion recognition as part of our FILTWAM framework. FILTWAM aims at deploying a real time multimodal emotion recognition method for providing more adequate feedback to the learners through an online communication skills training. Herein, timely feedback is needed that reflects on their shown intended emotions and which is also useful to increase learners’ awareness of their own behaviour. At least, a reliable and valid software interpretation of performed face and voice emotions is needed to warrant such adequate feedback. This validation study therefore calibrates our software. The study uses a multimodal fusion method. Twelve test persons performed computer-based tasks in which they were asked to mimic specific facial and vocal emotions. All test persons’ behaviour was recorded on video and two raters independently scored the showed emotions, which were contrasted with the software recognition outcomes. A hybrid method for multimodal fusion of our multimodal software shows accuracy between 96.1% and 98.6% for the best-chosen WEKA classifiers over predicted emotions. The software fulfils its requirements of real-time data interpretation and reliable results.The Netherlands Laboratory for Lifelong Learning (NELLL) of the Open University Netherlands

MAPEAMENTO SISTEMÁTICO DE LITERATURA SOBRE ESTUDOS DE INTERFACES DE USUÁRIO EM TECNOLOGIA ASSISTIVA

The article presents, through a Systematic Literature Review, the state-of-the-art of research on User Interfaces (UI) applied in Assistive Technology (TA) for users with motor and /or visual impairment, presenting patterns, trends and new gaps of research in this field of study of human-computer interactionO artigo apresenta, através de um Mapeamento Sistemático de Literatura, o estado da arte da pesquisa sobre as diversas Interfaces de Usuário (IU) aplicadas em Tecnologia Assistiva (TA) para usuários com deficiência motora e/ou visual, apresentando padrões, tendências e novos espaços de pesquisa nesse campo de estudo da interação humano-computador

Estimating pilots’ cognitive load from ocular parameters through simulation and in-flight studies

Eye tracking is the process of measuring either the point of gaze (where one is looking) or the motion of an eye relative to the head. This paper investigated use of eye gaze trackers in military aviation environment to automatically estimate pilot’s cognitive load from ocular parameters. We used a fixed base variable stability flight simulator with longitudinal tracking task and collected data from 14 military pilots. In a second study, we undertook three test flights with a BAES Hawk Trainer aircraft doing air to ground attack training missions and constant G level turn maneuvers up to +5G. Our study found that ocular parameters like rate of fixation is significantly different in different flying conditions and significantly correlate with altitude gradient during air to ground dive training task, normal load factor (G) of the aircraft during constant G level turn maneuvers and pilot’s control inceptor and tracking error in simulation tasks. Results from our studies can be used for real time estimation of pilots’ cognitive load, providing suitable warnings and alerts to the pilot in cockpit and training of military pilots on cognitive load management during operational missions

Cognitve Load Estimation in Drivers for Advanced Driver Assistance System

In this thesis, we investigate some physiological metrics to estimate the cognitive load experienced by drivers in a driving simulated environment and how their cognitive load is varies and affects certain physiological metrics.It is important to better understand the cognitive status of the human beings inorder to design efficient machines for automation. For instance, an advanced driver assistance system (ADAS) with the ability to understand the cognitive state of human will enhance the overall safety of the roads. In order to achieve such an intelligent automation system involving humans we need to develop approaches that can better estimate cognitive load through non-invasive means and to develop control strategies for real-time system adaptation with humans. The focus of this thesis is to present some hypotheses for cognitive load estimation based on some physiological measures, such as, pupil diameter, heart rate, and response-time and how each of these metrics change for varying cognitive difficulty levels. The variation in these metrics is proved using statistical analysis techniques

Um Estudo de Mapeamento Sistemático sobre Metodologias de Avaliação em Interação Humano-Computador voltadas à Tecnologia Assistiva com foco em Pessoas com Deficiência Motora

A proposição de uma Tecnologia Assistiva (TA) para a interação com o computador é ainda um grande desafio, uma vez que os dispositivos de interação precisam estar adaptados às necessidades e habilidades dos usuários. Este desafio é atualmente abordado pela área de Interação Humano-Computador (IHC), que explora o projeto, implementação e avaliação de sistemas informáticos computacionais interativos. No caso da avaliação de um dispositivo voltado para TA é ela que, além de outros fatores de performance, irá validar se a TA é realmente voltada para o público alvo. Este trabalho explora metodologias de avaliação em IHC com foco em pessoas com deficiência motora nos membros superiores, resultado de um mapeamento sistemático da literatura. Por fim, este trabalho incluí uma proposta de taxonomia de como estes dispositivos de TA são classificados quanto às suas formas de captação de dados.