Design of emotion-aware mobile apps for autistic children

Sensor technologies and facial expression recognition are now widely used by mobile devices to sense our environment and our own physical and mental state. With these technologies today, we have the ability to sense emotions and create emotion-aware apps. One target group that would benefit from emotion-aware Apps are autistic children as they have difficulty understanding and expressing emotions and they are keen mobile device users. However, current mobile apps aimed at autistic children are not emotion-aware. This led our team to design a suite of Apps, called CaptureMyEmotion, that uses wireless sensors to capture physiological data together with facial expression recognition to provide a very personalised way to help autistic children and their carers understanding and managing their emotions. This paper describes how we designed CaptureMyEmotion and it discusses our experience while using sensors and facial expression recognition to detect emotion. It presents in more details the first App we developed for Android phone and tablets, called MyMedia. MyMedia enables children to take photos, videos or sounds, and simultaneously attach emotion data to them. The photos can then be reviewed together with a carer providing them a new way to understand emotions and discussing their daily activities. © 2013 IUPESM and Springer-Verlag

An application to improve emotional skills in children with Autism Spectrum Disorder

Dissertação de Mestrado Integrado em Engenharia de Eletrónica Industrial e ComputadoresThis dissertation presents a project developed with the aim of promoting emotional skills in children with Autism Spectrum Disorders (ASD). The project involves a serious game and a playware object, which is a physical component that acts as the game controller and allows the user to interactively play the serious game. The playware object has six pressure buttons, each one showing an emoji with a specific facial expression and communicates wirelessly via Bluetooth with the serious game app installed in an Android device. The facial expressions used are: happiness, sadness, fear, anger, surprise and neutral/normal. They were applied to the three game activities (imitation, recognition and storytelling). The chain of tests started with an online questionnaire to validate the avatars created to represent the previously mentioned facial expressions in the game (with 114 answers and a mean success rate of 96.2%), which was followed by a usability test of the application (serious game and playware object) with six typically developing children (with 94.4% answer accuracy). Finally, the three game activities were tested with six children with ASD in three/four sessions. Due to the small group test and the short number of sessions, the goal was to test the acceptance of the game rather than the users´ improvement in the activity. It is worth referring that both the serious game and the playware object had a high level of approval from the children and they expressed their interest during the activities. With this project it was intended to contribute to the development of pedagogical resources to be used by professionals and families in the support of children with ASD.Esta dissertação apresenta um projeto desenvolvido com o objetivo de promover capacidades emocionais em crianças com Perturbação do Espectro do Autismo. Este projeto envolve um jogo sério e um objeto playware, que é um componente físico que funciona como controlador de jogo e permite que o utilizador jogue o jogo sério de uma forma interativa. O objeto playware tem seis botões de pressão, cada um com um emoji com uma expressão facial específica, e comunica sem fios por Bluetooth com a aplicação do jogo sério instalada no dispositivo Android. As expressões faciais usadas são: felicidade, tristeza, medo, raiva, surpresa e neutro/normal. Estas foram aplicadas às três diferentes atividades de jogo (imitar, reconhecer e contar histórias). A cadeia de testes começou com um questionário online para validar os avatares criados para representar as expressões faciais previamente mencionadas no jogo (com 114 submissões e uma taxa média de sucesso de 96,2%), seguido de um teste de usabilidade da aplicação (jogo sério e objeto playware) com seis crianças tipicamente desenvolvidas (com 94,4% de respostas corretas). Por fim, as três atividades de jogo foram testadas com seis crianças com Perturbação do Espectro do Autismo durante 3 a 4 sessões. Devido à pequena dimensão do grupo de teste e ao baixo número de sessões, o objetivo foi testar a aceitação do jogo em vez da evolução das capacidades dos utilizadores na atividade. É importante referir que tanto o jogo sério como o objeto playware tiveram um alto nível de aprovação por parte das crianças que expressaram o seu interesse durante as atividades. Este projeto pretende contribuir para o desenvolvimento de recursos pedagógicos a serem usados por profissionais e famílias no apoio a crianças com Perturbação do Espectro do Autismo

Feasibility of a smartphone application to identify young children at risk for Autism Spectrum Disorder in a low-income community setting in South Africa

Introduction and aims More than 90% of children with Autism Spectrum Disorder (ASD) live in low- and middle-income countries (LMIC) where there is a great need for culturally appropriate, scalable and effective early identification and intervention tools. Smartphone technology and application (‘apps’) may potentially play an important role in this regard. The Autism&Beyond iPhone App was designed as a potential screening tool for ASD risk in children aged 12-72 months. Here we investigated the technical feasibility and cultural acceptability of a smartphone app to determine risk for ASD in children aged 12-72 months in a naturalistic, low-income South African community setting. Methodology 37 typically-developing African children and their parents/carers were recruited from community centres in Khayelitsha Township, Cape Town, South Africa. We implemented a mixed-methods design, collecting both quantitative and qualitative data from participants in 2 stages. In stage 1, we collected quantitative data. With appropriate ethics and consent, parents completed a short technology questionnaire about their familiarity with and access to smartphones, internet and apps, followed by electronic iPhone-based demographic and ASD-related questionnaires. Next, children were shown 3 short videos of 30s each and a mirror stimulus on a study smartphone. The smartphone front facing (“selfie”) camera recorded video of the child’s facial expressions and head movement. Automated computer algorithms quantified positive emotions and time attending to stimuli. We validated the automatic coding by a) comparing the computer-generated analysis to human coding of facial expressions in a random sample (N=9), and b) comparing automated analysis of the South African data (N=33) with a matched American sample (N=33). In stage 2, a subset of families were invited to participate in focus group discussions to provide qualitative data on accessibility, acceptability, and cultural appropriateness of the app in their local community. Results Most parents (64%) owned a smartphone of which all (100%) were Android based, and many used Apps (45%). Human-automated coding showed excellent correlation for positive emotion (ICC= 0.95, 95% CI 0.81-0.99) and no statistically significant differences were observed between the South African and American sample in % time attending to the video stimuli. South African children, however, smiled less at the Toys&Rhymes (SA mean (SD) = 14% (24); USA mean (SD) = 31% (34); p=0.05) and Bunny video (SA mean (SD) = 12% (17); USA mean (SD) = 30% (0.27); p=0.006). Analysis of focus group data indicated that parents/carers found the App relatively easy to use, and would recommend it to others in their community provided the App and data transfer were free. Conclusion The results from this pilot study suggested the App to be technically accurate, accessible and culturally acceptable to families from a low-resource environment in South Africa. Given the differences in positive emotional response between the groups, careful consideration should be given to identify suitable stimuli if % time smiling is to be used as a global marker for autism risk across cultures and environments

Toward emotional interactive videogames for children with autism spectrum disorder

Technology and videogames have been proven as motivating tools for working attention and complex communication skills, especially in children with autism spectrum disorder (ASD). In this work, we present two experiences that used interactive games for promoting communication and attention. The first game considers emotions in order to measure children’s attention, concentration and satisfaction, while the second uses tangible tabletops for fostering cognitive planning. The analysis of the results obtained allows to propose a new study integrating both, in which the tangible interactive game is complemented with the emotional trainer in a way that allows identifying and classifying children’s emotion with ASD when they collaborate to solve cognitively significant and contextualized challenges. The first application proposed is an emotional trainer application in which the child can work out the seven basic emotions (happiness, sadness, fear, disgust, anger, surprise and neutral). Further, a serious videogame is proposed: a 3D maze where the emotions can be captured. The second case study was carried out in a Special Education Center, where a set of activities for working cognitive planning was proposed. In this case, a tangible interactive tabletop was used to analyze, in students with ASD, how the communication processes with these interfaces affect to the attention, memory, successive and simultaneous processing that compose cognitive planning from the PASS model. The results of the first study, suggest that the autistic children did not act with previous planning, but they used their perception to adjust their actions a posteriori (that explains the higher number of collisions). On the second case study, the successive processing was not explored. The inclusion of the mazes of case study 1 to a semantic rich scenario could allow us to measure the prior planning and the emotions involved in the maze game. The new physiological sensors will also help to validate the emotions felt by the children. The first study has as objective the capability to imitate emotions and resolve a maze without semantic context. The second study organized all the actions from a semantic context close to users. The attention results presented by the second study are coherent with the first study and complement it showing that attention can be receptive or selective. In the first study case, the receptive attention was the focus of analysis. In the second case, both contributed to explain and understand how it can be developed from a videogame

Robots and autistic children: a review

In accordance with the advancement in robotics and the scholarly literature, the extents of utilizing robots for autistic children are widened and could be a promising method for individual with Autism Spectrum Disorder (ASD) treatments, where the different form of robot (humanoid, non-humanoid, animal-like, toy, and kits) can be employed effectively as a support tool to augment the learning skills and rehabilitate of the individual with Autism Spectrum Disorder (ASD). Thus, the robots were exploited for ASD children in different aspects namely; modelling, teaching, and skills practicing; testing, highlighting and evaluating; providing feedback or encouragement; join Attention; eliciting social behaviours; emotion recognition and expression; imitation; vocalization; turn-taking; and diagnostic. The related literature published recently in journals and conferences is taken into account. In this paper, we review the use of robots that help in the therapy of individuals with Autism Spectrum Disorder (ASD). The articles on using robots for autistic children rehabilitation and education which reported results of experiments on a number of participants were implicated. After looking in digital libraries under this criteria, and excluding non-related, and duplicated studies, 39 studies have been found. The findings were focused mainly on the social communication skills of autistic children and how the extent of the robots mitigate their stereotyped behaviours. Deeper research is required in this area to cover all applications of robotic on autistic children in order to design feasible and low-cost robots that ensure provide high validity

Insights from mechanistic and digital intervention approaches

Eine effektive Förderung des Sozialverhaltens autistischer Kinder erfordert ein tiefgehendes Verständnis der Ursachen maladaptiver Reaktionen und die Bereitstellung eines leicht verfügbaren Förderangebots. Hier haben digitale Angebote ein großes Potential. Ebenso wie bei der Ursachenforschung mangelt es jedoch an Forschungsansätzen, welche eine Vielzahl an kognitiven und emotionalen Prozessen in die digitale Förderung integrieren. Dementsprechend untersuchte die Dissertation zunächst das Zusammenspiel verschiedener Ursachen aggressives Sozialverhaltens anhand eines etablierten Modells der sozial-kognitiven Informationsverarbeitung. Durch die Integration verschiedener Facetten der Empathie und deren zugrundeliegende Kompetenzen in ein digitales Förderangebot sollte im zweiten Schritt eine Verbesserung des Sozialverhaltens autistischer Grundschüler:innen erreicht werden. Es zeigte sich, dass Emotionsdysregulation verschiedene Formen aggressiven Sozialverhaltens und damit assoziierte feindselige Attributionen verstärkt. Letztere stand vor allem mit verbalen und verdeckten Aggressionsformen sowie mit guten Emotionserkennungsfertigkeiten im Zusammenhang. Eine Verbesserung des Sozialverhaltens und der Emotionsregulation konnte mittelfristig durch das sechswöchige, eltern-begleitete eLearningprogramm „Zirkus Empathico“ erreicht werden. Die multizentrische, randomisiert kontrollierte Studie ergab zudem kurzfristige und moderate Interventionseffekte für Empathie und Emotionserkennung als primäre Endpunkte. Insgesamt unterstreicht die Dissertation die Validität etablierter Modelle der sozialen Informationsverarbeitung sowie die Relevanz, diese zukünftigen Forschungs- und Interventionsansätzen zugrunde zu legen. Durch die Integration verschiedener sozio-emotionaler Kompetenzen scheint die digitale Intervention Zirkus Empathico prosoziales Verhalten autistischer Kinder auf effektive und praktikable Weise zu fördern.Effective training of autistic children`s social behavior requires an in-depth understanding of the causes of maladaptive responses and the provision of easily accessible support services. In this context, digital interventions have great potential. However, there is a lack of research approaches that integrate a variety of cognitive and emotional processes into both, explanation and digital support. The present dissertation first examined the interplay of different causes of aggressive social behavior by applying an established model of social-cognitive information processing to a sample of autistic elementary school students. Second, by integrating different facets of empathy and their underlying competencies into a digital program, the social behavior of autistic elementary school children should be improved. First, it was shown that emotion dysregulation strengthens various forms of aggressive social behavior and associated hostile attribution biases. The latter was mainly related to verbal and covert forms of aggression and good emotion recognition skills. Second, the parent-assisted eLearning program "Zirkus Empathico" led to a medium-term improvement in social behavior and emotion regulation after a six-week training. In addition, the multicenter randomized controlled trial showed moderate intervention effects on empathy and emotion recognition as primary outcomes, which were no longer detectable three months later. Overall, the dissertation highlights the validity of established models of social information processing and the relevance of using them as a foundation for future research and intervention. By integrating various socio-emotional competencies, the digital intervention Zirkus Empathico seems to strengthen autistic children’s prosocial behavior effectively and feasibly

Designing interactive virtual environments with feedback in health applications.

One of the most important factors to influence user experience in human-computer interaction is the user emotional reaction. Interactive environments including serious games that are responsive to user emotions improve their effectiveness and user satisfactions. Testing and training for user emotional competence is meaningful in healthcare field, which has motivated us to analyze immersive affective games using emotional feedbacks. In this dissertation, a systematic model of designing interactive environment is presented, which consists of three essential modules: affect modeling, affect recognition, and affect control. In order to collect data for analysis and construct these modules, a series of experiments were conducted using virtual reality (VR) to evoke user emotional reactions and monitoring the reactions by physiological data. The analysis results lead to the novel approach of a framework to design affective gaming in virtual reality, including the descriptions on the aspects of interaction mechanism, graph-based structure, and user modeling. Oculus Rift was used in the experiments to provide immersive virtual reality with affective scenarios, and a sample application was implemented as cross-platform VR physical training serious game for elderly people to demonstrate the essential parts of the framework. The measurements of playability and effectiveness are discussed. The introduced framework should be used as a guiding principle for designing affective VR serious games. Possible healthcare applications include emotion competence training, educational softwares, as well as therapy methods

Using Augmented Reality Toward Improving Social Skills:Scoping Review

BackgroundAugmented reality (AR) has emerged as a promising technology in educational settings owing to its engaging nature. However, apart from applications aimed at the autism spectrum disorder population, the potential of AR in social-emotional learning has received less attention. ObjectiveThis scoping review aims to map the range of AR applications that improve social skills and map the characteristics of such applications. MethodsIn total, 2 independent researchers screened 2748 records derived from 3 databases in December 2021—PubMed, IEEE Xplore, and ACM Guide to Computing Literature. In addition, the reference lists of all the included records and existing reviews were screened. Records that had developed a prototype with the main outcome of improving social skills were included in the scoping review. Included records were narratively described for their content regarding AR and social skills, their target populations, and their outcomes. Evaluation studies were assessed for methodological quality. ResultsA total of 17 records met the inclusion criteria for this study. Overall, 10 records describe applications for children with autism, primarily teaching about reading emotions in facial expressions; 7 records describe applications for a general population, targeting both children and adults, with a diverse range of outcome goals. The methodological quality of evaluation studies was found to be weak. ConclusionsMost applications are designed to be used alone, although AR is well suited to facilitating real-world interactions during a digital experience, including interactions with other people. Therefore, future AR applications could endorse social skills in a general population in more complex group settings

Assistive Technology to Improve Collaboration in Children with ASD: State-of-the-Art and Future Challenges in the Smart Products Sector

Within the field of products for autism spectrum disorder, one of the main research areas is focused on the development of assistive technology. Mid and high-tech products integrate interactive and smart functions with multisensory reinforcements, making the user experience more intuitive, adaptable, and dynamic. These products have a very significant impact on improving the skills of children with autism, including collaboration and social skills, which are essential for the integration of these children into society and, therefore, their well-being. This work carried out an exhaustive analysis of the scientific literature, as well as market research and trends, and patent analysis to explore the state-of-the-art of assistive technology and smart products for children with ASD, specifically those aimed at improving social and communication skills. The results show a reduced availability of products that act as facilitators of the special needs of children with ASD, which is even more evident for products aimed at improving collaboration skills. Products that allow the participation of several users simultaneously through multi-user interfaces are required. On top of this, the trend toward virtual environments is leading to a loss of material aspects in the design that are essential for the development of these children

Emotions Recognition in people with Autism using Facial Expressions and Machine Learning Techniques: Survey

في الآونة الأخيرة ، اهتمت الكثير من الدراسات بالتعرف على المشاعر واكتشافها لدى الأشخاص المصابين بالتوحد. الهدف الرئيسي من هذه الورقة هو مسح الدراسات المختلفة التي تتعلق بالحالة العاطفية للأشخاص المصابين بالتوحد. يتضمن الاستطلاع جزأين ، يركز الجزء الأول على الدراسات التي استخدمت تعابير الوجه للتعرف على المشاعر واكتشافها. حيث تعتبر تعبيرات الوجه من التقنيات العاطفية المهمة التي تستخدم للتعبير عن أنماط مختلفة من المشاعر. ركزت الأجزاء الثانية من هذه الدراسة على الأساليب التقنية المختلفة مثل التعلم الآلي والتعلم العميق والخوارزميات الأخرى التي تستخدم لتحليل وتحديد سلوكيات الوجه للأشخاص المصابين بالتوحد. للعثور على الحل الأمثل ، يتم من خلال التحقيق في مقارنة أنظمة الكشف عن المشاعر الحالية في هذه الورقة.Recently, a lot of studies have been interested in recognizing and detection of emotions in people with autism.  The main goal of this paper is to survey different studies which have been concerned emotional state of people with autism.  The survey includes two parts, first one focused on studies which use facial expressions to recognize and detect emotions. As facial expressions are considered the affective and important techniques which is used to express different patterns of emotions.  Second parts of this study, focuses on different technical methods like machine learning, deep learning and other algorithms that are employed to analyze and determine the facial behaviors of people with autism. To find the optimal solution, a comparison of current emotion-detecting systems is investigated in this paper