Augmented Reality Action Assistance and Learning for Cognitively Impaired People. A Systematic Literature Review

Blattgerste J, Renner P, Pfeiffer T. Augmented Reality Action Assistance and Learning for Cognitively Impaired People. A Systematic Literature Review. In: The 12th PErvasive Technologies Related to Assistive Environments Conference (PETRA ’19). New York, NY, USA: ACM; 2019.Augmented reality (AR) is a promising tool for many situations in which assistance is needed, as it allows for instructions and feedback to be contextualized. While research and development in this area have been primarily driven by industry, AR could also have a huge impact on those who need assistance the most: cognitively impaired people of all ages. In recent years some primary research on applying AR for action assistance and learning in the context of this target group has been conducted. However, the research field is sparsely covered and contributions are hard to categorize. An overview of the current state of research is missing. We contribute to filling this gap by providing a systematic literature review covering 52 publications. We describe the often rather technical publications on an abstract level and quantitatively assess their usage purpose, the targeted age group and the type of AR device used. Additionally, we provide insights on the current challenges and chances of AR learning and action assistance for people with cognitive impairments. We discuss trends in the research field, including potential future work for researchers to focus on

Modélisation d'une interaction système-résident contextuelle, personnalisée et adaptative pour l'assistance cognitive à la réalisation des activités de la vie quotidienne dans les maisons connectées

Alors que le nombre de personnes vivant avec des déficits cognitifs qui découlent d’un traumatisme craniocérébral (TCC) va en croissant, les technologies d’assistance sont de plus en plus développées pour résoudre les problèmes qu’ils induisent dans la réalisation des activités de la vie quotidienne. L’Internet des objets et l’intelligence ambiante offrent un cadre pour fournir des services d’assistance sensibles au contexte, adaptatifs, autonomes et personnalisés pour ces personnes ayant des besoins particuliers. Une revue de la littérature sur le sujet permet de constater que les systèmes existants offrent très souvent une assistance excessive, quand l’aide contient plus d’information que nécessaire ou quand elle est fournie automatiquement à chaque étape de l’activité. Cette assistance, inadaptée aux besoins et aux capacités de la personne, est contraire à certains principes de la réadaptation cognitive qui prônent la fourniture d’une assistance minimale pour encourager la personne à agir au meilleur de ses capacités. Cette thèse propose des modèles pour automatiser l’assistance cognitive sous forme de dialogue contextuel entre une personne ayant des déficits cognitifs dus au TCC et un système lui fournissant l’assistance appropriée qui l’encourage à réaliser ses activités par lui-même. Les principales contributions sont : (1) un modèle ontologique comme support de l’assistance cognitive dans les maisons connectées ; (2) un modèle d’interaction entre l’agent intelligent d’une maison connectée et une personne ayant subi un TCC, dans le cadre de l’assistance cognitive. Le modèle ontologique proposé s’appuie sur les actes de langages et les données probantes de la réadaptation cognitive afin que l’assistance reflète la pratique clinique. Il vise à fournir aux maisons intelligentes la sémantique des données nécessaires pour caractériser les situations où il y a besoin d’assistance, les messages d’assistance de gradations différentes et les réactions de la personne. Informé par le modèle ontologique, le modèle d’interaction basé sur des arbres de comportement (« behaviour trees ») permet alors à un agent intelligent de planifier dynamiquement la diffusion de messages d’assistance progressifs avec des ajustements si nécessaire, en fonction du profil et du comportement du résident de la maison connectée lors de l’accomplissement de ses activités. Une validation préliminaire montre l’applicabilité des modèles dans l’implémentation de scénarios relatifs à l’utilisation sécuritaire d’une cuisinière connectée dédiée aux personnes ayant subi un TCC

Kinematic assessment for stroke patients in a stroke game and a daily activity recognition and assessment system

Stroke is the leading cause of serious, long-term disabilities among which deficits in motor abilities in arms or legs are most common. Those who suffer a stroke can recover through effective rehabilitation which is delicately personalized. To achieve the best personalization, it is essential for clinicians to monitor patients' health status and recovery progress accurately and consistently. Traditionally, rehabilitation involves patients performing exercises in clinics where clinicians oversee the procedure and evaluate patients' recovery progress. Following the in-clinic visits, additional home practices are tailored and assigned to patients. The in-clinic visits are important to evaluate recovery progress. The information collected can then help clinicians customize home practices for stroke patients. However, as the number of in-clinic sessions is limited by insurance policies, the recovery information collected in-clinic is often insufficient. Meanwhile, the home practice programs report low adherence rates based on historic data. Given that clinicians rely on patients to self-report adherence, the actual adherence rate could be even lower. Despite the limited feedback clinicians could receive, the measurement method is subjective as well. In practice, classic clinical scales are mostly used for assessing the qualities of movements and the recovery status of patients. However, these clinical scales are evaluated subjectively with only moderate inter-rater and intra-rater reliabilities. Taken together, clinicians lack a method to get sufficient and accurate feedback from patients, which limits the extent to which clinicians can personalize treatment plans. This work aims to solve this problem. To help clinicians obtain abundant health information regarding patients' recovery in an objective approach, I've developed a novel kinematic assessment toolchain that consists of two parts. The first part is a tool to evaluate stroke patients' motions collected in a rehabilitation game setting. This kinematic assessment tool utilizes body-tracking in a rehabilitation game. Specifically, a set of upper body assessment measures were proposed and calculated for assessing the movements using skeletal joint data. Statistical analysis was applied to evaluate the quality of upper body motions using the assessment outcomes. Second, to classify and quantify home activities for stroke patients objectively and accurately, I've developed DARAS, a daily activity recognition and assessment system that evaluates daily motions in a home setting. DARAS consists of three main components: daily action logger, action recognition part, and assessment part. The logger is implemented with a Foresite system to record daily activities using depth and skeletal joint data. Daily activity data in a realistic environment were collected from sixteen post-stroke participants. The collection period for each participant lasts three months. An ensemble network for activity recognition and temporal localization was developed to detect and segment the clinically relevant actions from the recorded data. The ensemble network fuses the prediction outputs from customized 3D Convolutional-De-Convolutional, customized Region Convolutional 3D network and a proposed Region Hierarchical Co-occurrence network which learns rich spatial-temporal features from either depth data or joint data. The per-frame precision and the per-action precision were 0.819 and 0.838, respectively, on the validation set. For the recognized actions, the kinematic assessments were performed using the skeletal joint data, as well as the longitudinal assessments. The results showed that, compared with non-stroke participants, stroke participants had slower hand movements, were less active, and tended to perform fewer hand manipulation actions. The assessment outcomes from the proposed toolchain help clinicians to provide more personalized rehabilitation plans that benefit patients.Includes bibliographical references

Proceedings of the 10th international conference on disability, virtual reality and associated technologies (ICDVRAT 2014)

The proceedings of the conferenc

Proceedings of the 9th international conference on disability, virtual reality and associated technologies (ICDVRAT 2012)

The proceedings of the conferenc