Assessing Visual Attention Using Eye Tracking Sensors in Intelligent Cognitive Therapies Based on Serious Games

This study examines the use of eye tracking sensors as a means to identify children's behavior in attention-enhancement therapies. For this purpose, a set of data collected from 32 children with different attention skills is analyzed during their interaction with a set of puzzle games. The authors of this study hypothesize that participants with better performance may have quantifiably different eye-movement patterns from users with poorer results. The use of eye trackers outside the research community may help to extend their potential with available intelligent therapies, bringing state-of-the-art technologies to users. The use of gaze data constitutes a new information source in intelligent therapies that may help to build new approaches that are fully-customized to final users' needs. This may be achieved by implementing machine learning algorithms for classification. The initial study of the dataset has proven a 0.88 (±0.11) classification accuracy with a random forest classifier, using cross-validation and hierarchical tree-based feature selection. Further approaches need to be examined in order to establish more detailed attention behaviors and patterns among children with and without attention problems

Eye movement analysis and cognitive assessment: the use of comparative visual search tasks in a non-immersive vr application

Background: An adequate behavioral response depends on attentional and mnesic processes. When these basic cognitive functions are impaired, the use of non-immersive Virtual Reality Applications (VRAs) can be a reliable technique for assessing the level of impairment. However, most non-immersive VRAs use indirect measures to make inferences about visual attention and mnesic processes (e.g., time to task completion, error rate). Objectives: To examine whether the eye movement analysis through eye tracking (ET) can be a reliable method to probe more effectively where and how attention is deployed and how it is linked with visual working memory during comparative visual search tasks (CVSTs) in non-immersive VRAs. Methods: The eye movements of 50 healthy participants were continuously recorded while CVSTs, selected from a set of cognitive tasks in the Systemic Lisbon Battery (SLB). Then a VRA designed to assess of cognitive impairments were randomly presented. Results: The total fixation duration, the number of visits in the areas of interest and in the interstimulus space, along with the total execution time was significantly different as a function of the Mini Mental State Examination (MMSE) scores. Conclusions: The present study demonstrates that CVSTs in SLB, when combined with ET, can be a reliable and unobtrusive method for assessing cognitive abilities in healthy individuals, opening it to potential use in clinical samples.info:eu-repo/semantics/submittedVersio

Motion-based technology to support motor skills screening in developing children: A scoping review

Background. Acquiring motor skills is fundamental for children's development since it is linked to cognitive development. However, access to early detection of motor development delays is limited. Aim. This review explores the use and potential of motion-based technology (MBT) as a complement to support and increase access to motor screening in developing children. Methods. Six databases were searched following the PRISMA guidelines to search, select, and assess relevant works where MBT recognised the execution of children's motor skills. Results. 164 studies were analysed to understand the type of MBT used, the motor skills detected, the purpose of using MBT and the age group targeted. Conclusions. There is a gap in the literature aiming to integrate MBT in motor skills development screening and assessment processes. Depth sensors are the prevailing technology offering the largest detection range for children from age 2. Nonetheless, the motor skills detected by MBT represent about half of the motor skills usually observed to screen and assess motor development. Overall, research in this field is underexplored. The use of multimodal approaches, combining various motion-based sensors, may support professionals in the health domain and increase access to early detection programmes.Funding for open access charge: Universidad de Málaga / CBUA

A survey of Alzheimer's disease early diagnosis methods for cognitive assessment

Dementia is a syndrome that is characterised by the decline of different cognitive abilities. A high rate of deaths and high cost for detection, treatments, and patients care count amongst its consequences. Although there is no cure for dementia, a timely diagnosis helps in obtaining necessary support, appropriate medication, and maintenance, as far as possible, of engagement in intellectual, social, and physical activities. The early detection of Alzheimer Disease (AD) is considered to be of high importance for improving the quality of life of patients and their families. In particular, Virtual Reality (VR) is an expanding tool that can be used in order to assess cognitive abilities while navigating through a Virtual Environment (VE). The paper summarises common AD screening and diagnosis techniques focusing on the latest approaches that are based on Virtual Environments, behaviour analysis, and emotions recognition, aiming to provide more reliable and non-invasive diagnostics at home or in a clinical environment. Furthermore, different AD diagnosis evaluation methods and metrics are presented and discussed together with an overview of the different datasets

Virtual Reality Games for Motor Rehabilitation

This paper presents a fuzzy logic based method to track user satisfaction without the need for devices to monitor users physiological conditions. User satisfaction is the key to any product’s acceptance; computer applications and video games provide a unique opportunity to provide a tailored environment for each user to better suit their needs. We have implemented a non-adaptive fuzzy logic model of emotion, based on the emotional component of the Fuzzy Logic Adaptive Model of Emotion (FLAME) proposed by El-Nasr, to estimate player emotion in UnrealTournament 2004. In this paper we describe the implementation of this system and present the results of one of several play tests. Our research contradicts the current literature that suggests physiological measurements are needed. We show that it is possible to use a software only method to estimate user emotion

Investigating Visual Perception Impairments through Serious Games and Eye Tracking to Anticipate Handwriting Difficulties

Dysgraphia is a learning disability that causes handwritten production below expectations. Its diagnosis is delayed until the completion of handwriting development. To allow a preventive training program, abilities not directly related to handwriting should be evaluated, and one of them is visual perception. To investigate the role of visual perception in handwriting skills, we gamified standard clinical visual perception tests to be played while wearing an eye tracker at three difficulty levels. Then, we identified children at risk of dysgraphia through the means of a handwriting speed test. Five machine learning models were constructed to predict if the child was at risk, using the CatBoost algorithm with Nested Cross-Validation, with combinations of game performance, eye-tracking, and drawing data as predictors. A total of 53 children participated in the study. The machine learning models obtained good results, particularly with game performances as predictors (F1 score: 0.77 train, 0.71 test). SHAP explainer was used to identify the most impactful features. The game reached an excellent usability score (89.4 +/- 9.6). These results are promising to suggest a new tool for dysgraphia early screening based on visual perception skills

Upper-limb Kinematic Analysis and Artificial Intelligent Techniques for Neurorehabilitation and Assistive Environments

Stroke, one of the leading causes of death and disability around the world, usually affects the motor cortex causing weakness or paralysis in the limbs of one side of the body. Research efforts in neurorehabilitation technology have focused on the development of robotic devices to restore motor and cognitive function in impaired individuals, having the potential to deliver high-intensity and motivating therapy. End-effector-based devices have become an usual tool in the upper- limb neurorehabilitation due to the ease of adapting to patients. However, they are unable to measure the joint movements during the exercise. Thus, the first part of this thesis is focused on the development of a kinematic reconstruction algorithm that can be used in a real rehabilitation environment, without disturbing the normal patient-clinician interaction. On the basis of the algorithm found in the literature that presents some instabilities, a new algorithm is developed. The proposed algorithm is the first one able to online estimate not only the upper-limb joints, but also the trunk compensation using only two non-invasive wearable devices, placed onto the shoulder and upper arm of the patient. This new tool will allow the therapist to perform a comprehensive assessment combining the range of movement with clinical assessment scales. Knowing that the intensity of the therapy improves the outcomes of neurorehabilitation, a ‘self-managed’ rehabilitation system can allow the patients to continue the rehabilitation at home. This thesis proposes a system to online measure a set of upper-limb rehabilitation gestures, and intelligently evaluates the quality of the exercise performed by the patients. The assessment is performed through the study of the performed movement as a whole as well as evaluating each joint independently. The first results are promising and suggest that this system can became a a new tool to complement the clinical therapy at home and improve the rehabilitation outcomes. Finally, severe motor condition can remain after rehabilitation process. Thus, a technology solution for these patients and people with severe motor disabilities is proposed. An intelligent environmental control interface is developed with the ability to adapt its scan control to the residual capabilities of the user. Furthermore, the system estimates the intention of the user from the environmental information and the behavior of the user, helping in the navigation through the interface, improving its independence at home.El accidente cerebrovascular o ictus es una de las causas principales de muerte y discapacidad a nivel mundial. Normalmente afecta a la corteza motora causando debilidad o parálisis en las articulaciones del mismo lado del cuerpo. Los esfuerzos de investigación dentro de la tecnología de neurorehabilitación se han centrado en el desarrollo de dispositivos robóticos para restaurar las funciones motoras y cognitivas en las personas con esta discapacidad, teniendo un gran potencial para ofrecer una terapia de alta intensidad y motivadora. Los dispositivos basados en efector final se han convertido en una herramienta habitual en la neurorehabilitación de miembro superior ya que es muy sencillo adaptarlo a los pacientes. Sin embargo, éstos no son capaces de medir los movimientos articulares durante la realización del ejercicio. Por tanto, la primera parte de esta tesis se centra en el desarrollo de un algoritmo de reconstrucción cinemática que pueda ser usado en un entorno de rehabilitación real, sin perjudicar a la interacción normal entre el paciente y el clínico. Partiendo de la base que propone el algoritmo encontrado en la literatura, el cual presenta algunas inestabilidades, se ha desarrollado un nuevo algoritmo. El algoritmo propuesto es el primero capaz de estimar en tiempo real no sólo las articulaciones del miembro superior, sino también la compensación del tronco usando solamente dos dispositivos no invasivos y portátiles, colocados sobre el hombro y el brazo del paciente. Esta nueva herramienta permite al terapeuta realizar una valoración más exhaustiva combinando el rango de movimiento con las escalas de valoración clínicas. Sabiendo que la intensidad de la terapia mejora los resultados de la recuperación del ictus, un sistema de rehabilitación ‘auto-gestionado’ permite a los pacientes continuar con la rehabilitación en casa. Esta tesis propone un sistema para medir en tiempo real un conjunto de gestos de miembro superior y evaluar de manera inteligente la calidad del ejercicio realizado por el paciente. La valoración se hace a través del estudio del movimiento ejecutado en su conjunto, así como evaluando cada articulación independientemente. Los primeros resultados son prometedores y apuntan a que este sistema puede convertirse en una nueva herramienta para complementar la terapia clínica en casa y mejorar los resultados de la rehabilitación. Finalmente, después del proceso de rehabilitación pueden quedar secuelas motoras graves. Por este motivo, se propone una solución tecnológica para estas personas y para personas con discapacidades motoras severas. Así, se ha desarrollado una interfaz de control de entorno inteligente capaz de adaptar su control a las capacidades residuales del usuario. Además, el sistema estima la intención del usuario a partir de la información del entorno y el comportamiento del usuario, ayudando en la navegación a través de la interfaz, mejorando su independencia en el hogar

Analyse visuelle et cérébrale de l’état cognitif d’un apprenant

Un état cognitif peut se définir comme étant l’ensemble des processus cognitifs inférieurs (par exemple : perception et attention) et supérieurs (par exemple : prise de décision et raisonnement), nécessitant de la part de l’être humain toutes ses capacités mentales en vue d’utiliser des connaissances existantes pour résoudre un problème donné ou bien d’établir de nouvelles connaissances. Dans ce contexte, une attention particulière est portée par les environnements d’apprentissage informatisés sur le suivi et l’analyse des réactions émotionnelles de l’apprenant lors de l’activité d’apprentissage. En effet, les émotions conditionnent l’état mental de l’apprenant qui a un impact direct sur ses capacités cognitives tel que le raisonnement, la prise de décision, la mémorisation, etc. Dans ce contexte, l’objectif est d’améliorer les capacités cognitives de l’apprenant en identifiant et corrigeant les états mentaux défavorables à l’apprentissage en vue d’optimiser les performances des apprenants. Dans cette thèse, nous visons en particulier à examiner le raisonnement en tant que processus cognitif complexe de haut niveau. Notre objectif est double : en premier lieu, nous cherchons à évaluer le processus de raisonnement des étudiants novices en médecine à travers leur comportement visuel et en deuxième lieu, nous cherchons à analyser leur état mental quand ils raisonnent afin de détecter des indicateurs visuels et cérébraux permettant d’améliorer l’expérience d’apprentissage. Plus précisément, notre premier objectif a été d’utiliser les mouvements des yeux de l’apprenant pour évaluer son processus de raisonnement lors d’interactions avec des jeux sérieux éducatifs. Pour ce faire, nous avons analysé deux types de mesures oculaires à savoir : des mesures statiques et des mesures dynamiques. Dans un premier temps, nous avons étudié la possibilité d’identifier automatiquement deux classes d’apprenants à partir des différentes mesures statiques, à travers l’entrainement d’algorithmes d’apprentissage machine. Ensuite, en utilisant les mesures dynamiques avec un algorithme d’alignement de séquences issu de la bio-informatique, nous avons évalué la séquence logique visuelle suivie par l’apprenant en cours de raisonnement pour vérifier s’il est en train de suivre le bon processus de raisonnement ou non. Notre deuxième objectif a été de suivre l’évolution de l’état mental d’engagement d’un apprenant à partir de son activité cérébrale et aussi d’évaluer la relation entre l’engagement et les performances d’apprentissage. Pour cela, une étude a été réalisée où nous avons analysé la distribution de l’indice d’engagement de l’apprenant à travers tout d’abord les différentes phases de résolution du problème donné et deuxièmement, à travers les différentes régions qui composent l’interface de l’environnement. L’activité cérébrale de chaque participant a été mesurée tout au long de l’interaction avec l’environnement. Ensuite, à partir des signaux obtenus, un indice d’engagement a été calculé en se basant sur les trois bandes de fréquences α, β et θ. Enfin, notre troisième objectif a été de proposer une approche multimodale à base de deux senseurs physiologiques pour permettre une analyse conjointe du comportement visuel et cérébral de l’apprenant. Nous avons à cette fin enregistré les mouvements des yeux et l’activité cérébrale de l’apprenant afin d’évaluer son processus de raisonnement durant la résolution de différents exercices cognitifs. Plus précisément, nous visons à déterminer quels sont les indicateurs clés de performances à travers un raisonnement clinique en vue de les utiliser pour améliorer en particulier, les capacités cognitives des apprenants novices et en général, l’expérience d’apprentissage.A cognitive state can be defined as a set of inferior (e.g. perception and attention) and superior (e.g. perception and attention) cognitive processes, requiring the human being to have all of his mental abilities in an effort to use existing knowledge to solve a given problem or to establish new knowledge. In this context, a particular attention is paid by computer-based learning environments to monitor and assess learner’s emotional reactions during a learning activity. In fact, emotions govern the learner’s mental state that has in turn a direct impact on his cognitive abilities such as reasoning, decision-making, memory, etc. In this context, the objective is to improve the cognitive abilities of the learner by identifying and redressing the mental states that are unfavorable to learning in order to optimize the learners’ performances. In this thesis, we aim in particular to examine the reasoning as a high-level cognitive process. Our goal is two-fold: first, we seek to evaluate the reasoning process of novice medical students through their visual behavior and second, we seek to analyze learners’ mental states when reasoning to detect visual and cerebral indicators that can improve learning outcomes. More specifically, our first objective was to use the learner’s eye movements to assess his reasoning process while interacting with educational serious games. For this purpose, we have analyzed two types of ocular metrics namely, static metrics and dynamic metrics. First of all, we have studied the feasibility of using static metrics to automatically identify two groups of learners through the training of machine learning algorithms. Then, we have assessed the logical visual sequence followed by the learner when reasoning using dynamic metrics and a sequence alignment method from bio-informatics to see if he/she performed the correct reasoning process or not. Our second objective was to analyze the evolution of the learner’s engagement mental state from his brain activity and to assess the relationship between engagement and learning performance. An experimental study was conducted where we analyzed the distribution of the learner engagement index through first, the different phases of the problem-solving task and second, through the different regions of the environment interface. The cerebral activity of each participant was recorded during the whole game interaction. Then, from the obtained signals, an engagement index was computed based on the three frequency bands α, β et θ. Finally, our third objective was to propose a multimodal approach based on two physiological sensors to provide a joint analysis of the learner’s visual and cerebral behaviors. To this end, we recorded eye movements and brain activity of the learner to assess his reasoning process during the resolution of different cognitive tasks. More precisely, we aimed to identify key indicators of reasoning performance in order to use them to improve the cognitive abilities of novice learners in particular, and the learning experience in general

Movement Pattern Recognition in Physical Rehabilitation - Cognitive Motivation-based IT Method and Algorithms

In this paper, a solution is presented to support both existing and future movement rehabilitation applications. The presented method combines the advantages of human-computer interaction-based movement therapy, with the cognitive property of intelligent decision-making systems. With this solution, therapy could be fully adapted to the needs of the patients and conditions while maintaining a sense of success in them, thereby motivating them. In our modern digital age, the development of HCI interfaces walks together with the growth of users’ needs. The available technologies have limitations, which can reduce the effectiveness of modern input devices, such as the Kinect sensor or any other similar sensors. In this article, multiple newly developed and modified methods are introduced with the aim to overcome these limitations. These methods can fully adapt the movement pattern recognition to the users' skills. The main goals are to apply this method in movement rehabilitation, where the supervisor, a therapist can personalize the rehabilitation exercises due to the Distance Vector-based Gesture Recognition (DVGR), Reference Distance-based Synchronous/Asynchronous Movement Recognition (RDSMR/RDAMR) and the Real-Time Adaptive Movement Pattern Classification (RAMPC) methods