This paper presents a project developed with the aim of promoting emotional skills in children with Autism Spectrum Disorders (ASD). The project involves a serious game and a playware object, which is a physical component that allows the user to interactively play the serious game. The playware object has six buttons, each one showing an emoji with a specific facial expression and communicates via Bluetooth with the serious game app installed in an Android device. The facial expressions used are: happiness, sadness, fear, anger, surprise and neutral/normal. They were applied to the three game activities (imitation, recognition and storytelling). The chain of tests started with an online questionnaire to validate the avatars created to represent the previously mentioned facial expressions in the game, which was followed by a usability test of the application (serious game and playware object) with six typically developing children. Finally, the three game activities were tested with six children with ASD in three/four sessions. Due to the small test group and reduced number of sessions, the primary objective was to assess if the target group accepted the application. In fact, it had a high level of approval regarding both the serious game and the playware object.
had a high level of approval regarding both the serious game and the playware object.COMPETE:  POCI-01-0145-FEDER-007043  and  FCT –Fundação  para  a  Ciência  e  Tecnologia  within  the  Project Scope:  UID/CEC/00319/2013.  Vinicius  Silva  also  thanks FCT for the PhD scholarship SFRH/BD/SFRH/BD/133314/2017. The authors thank the teachers and students   of   the   Elementary   School   of   Gualtar   (EB1/JI 
Gualtar) in Braga for the participation in the testsinfo:eu-repo/semantics/publishedVersio

Azevedo, José Pedro Coelho

Esteves, João Sena

Pereira, Ana Paula da Silva

Silva, Vinicius Corrêa Alves

Soares, Filomena

English

Universidade do Minho: RepositoriUM

Serious Games Assisted By Playware As A Way To Improve Socio-Emotional Skills In Children With Autism Spectrum Disorder  José Azevedo1, Vinicius Silva2 1Dept. of Industrial Electronics  2Algoritmi Research Centre  University of Minho Guimarães, Portugal [email: a70448, a65312]@alunos.uminho.pt Filomena Soares1,2, João Sena Esteves1,2, and Ana Paula Pereira3  3Research Centre on Education, Institute of Education, University of Minho Braga, Portugal [email: fsoares, sena]@dei.uminho.pt, appereira@ie.uminho.pt   Abstract—This paper presents a project developed with the aim of promoting emotional skills in children with Autism Spectrum Disorders (ASD). The project involves a serious game and a playware object, which is a physical component that allows the user to interactively play the serious game. The playware object has six buttons, each one showing an emoji with a specific facial expression and communicates via Bluetooth with the serious game app installed in an Android device. The facial expressions used are: happiness, sadness, fear, anger, surprise and neutral/normal. They were applied to the three game activities (imitation, recognition and storytelling). The chain of tests started with an online questionnaire to validate the avatars created to represent the previously mentioned facial expressions in the game, which was followed by a usability test of the application (serious game and playware object) with six typically developing children. Finally, the three game activities were tested with six children with ASD in three/four sessions. Due to the small test group and reduced number of sessions, the primary objective was to assess if the target group accepted the application. In fact, it had a high level of approval regarding both the serious game and the playware object. Keywords - Serious Games; Playware; Autism Spectrum Disorder; Emotions. I.  INTRODUCTION Humans are social by nature. Every interaction is emphasized by the participants’ emotions, which are constantly present during their lives. Being unable of “reading” the emotional state of their peers is a very significant deficit for anyone. This is the situation in which children with Autism Spectrum Disorder (ASD) usually find themselves in, since they struggle with emotions interpretation, replication and control. This fact is present throughout their lives but may have a reduced effect if the subject undergoes behavioural intervention and/or receives specialized education. Several works have been developed in areas, such as robotics and software engineering, in the form of serious games [1]–[6], playware technology [7]–[11] and robots [12]–[19], among others, with the aim of facilitating the assimilation and improvement of emotional skills by children with ASD. Playware is the term attributed to the use of intelligent technology that aims at producing playful experiences by combining the use of both hardware and software [7]. This project integrates playware and serious games with the objective of delivering an interactive and appellative serious game to the user, allowing him/her to improve his/her socio-emotional skills.  This article is divided into four sections: Section 2 addresses the application, presenting the playware object and the serious game, Section 3 focus on the obtained results and Section 4 presents the concluding thoughts. II. THE APPLICATION The application is composed of a serious game and a playware object. The goal was creating a serious game to deliver the information in a “funny” way. A playware object was added with the aim of making the game even more attractive and interactive, working as a reinforcement.  A. Playware Object A playware object was created for this project to act as the intermediary between the user and the Android device (Fig. 1).    Figure 1. The playware object that acts as the game controller  158Copyright (c) IARIA, 2018.     ISBN:  978-1-61208-660-6SENSORDEVICES 2018 : The Ninth International Conference on Sensor Device Technologies and ApplicationsThe style chosen for the playware was a panel of six buttons, which is simple, making it easy to assimilate information, acting as an intuitive controller. Furthermore, the fact that all six buttons are visible at once facilitates the task of going through the facial expressions available to choose from and decide upon the one that the user finds more appropriate. The playware object integrates a microcontroller, which detects which button was pressed and sends the respective signal to the serious game wirelessly through a Bluetooth module.  B. Serious Game The serious game is the core of the application, because it is where all the information is centred. It was developed for Android and, during the test phase, it was played in an Android tablet. Upon the start, the serious game opens the main screen, i.e., the menu (Fig. 2, in Portuguese). From the menu, the user can choose one of the three game modes to play (imitation, recognition or storytelling), access the options menu (to adjust the volume, start the Bluetooth connection or access the scoreboard) or quit the game.   Figure 2. The serious game main menu (in Portuguese) – Imitation, Recognition, Storytelling and Options buttons respectively.   Visually, the serious game has a very simplistic and clean design, not only in the main menu, but also throughout the game itself, so the users’ attention will not change its focus towards unnecessary features. Note that the chosen game language is Portuguese, due to the target group.  1) Imitation game mode In this game mode, several different facial expressions displayed by the avatar are presented to the user, one at a time. After stating what emotion is being displayed, the user him/herself replicates that emotion. This set of instructions is given to the user in a pre-game screen, designed for this purpose. The game is played side by side with a therapist/teacher that acts as a reinforcement to motivate the user to show the facial expression that matches each emotion. This agent has also the role of evaluating if the correct facial expression is replicated and sending the respective feedback to the serious game.  2) Recognition game mode The instruction screen tells the user how to perform this game mode, which he/she does using the playware object.  The same set of images (avatars) is presented to the player, but the task is different. In this case, the user looks at the display, identifies the emotion that the avatar is portraying and chooses the emoji present in the playware object that more accurately shows that emotion.  3) Storytelling game mode This is the hardest of all three game activities. Throughout the exhibition of 15 images, each displaying a unique scenario with its own narrative, the user listens to the story and, at the end, he/she is prompted to tell what emotion the main character was feeling in that specific situation. The answer is transmitted to the serious game in the same manner used in the recognition game mode, i.e., by the user, through the playware object. III. RESULTS The first test that was performed had the objective of validating the avatars (Fig. 3) that were created to represent each of the six previously mentioned facial expressions. This was possible to achieve due to an on-line questionnaire that was developed for this purpose. After 114 submissions, all avatars had an accuracy of over 90% for the emotion that they meant to portrait. The values were: fear – 94.7%, surprise – 93%, neutral – 95.6%, happiness – 99.1%, sadness – 98.2% and anger – 99.1%. A usability test of the application was then performed with six typically developing children that were asked to look at the avatars and say what emotion they were representing. After, they looked at the emojis and chose the one that they felt matched that emotion.  It is worth to point out that only two of the six children did not answer correctly to all questions (each failed one emotion).   Figure 3. Avatars used in the serious game 159Copyright (c) IARIA, 2018.     ISBN:  978-1-61208-660-6SENSORDEVICES 2018 : The Ninth International Conference on Sensor Device Technologies and Applications The last round of tests was performed over the course of a maximum of four sessions with children with ASD, with ages ranging between 6 to 10 years old, where data was saved considering the time they took to answer and if that answer was correct or not. For the purpose of anonymity, the children will be referred as subject 1 to 6 (or S1 to S6).  A. Subject 1 The teachers considered that the two activities, recognize and imitate facial expressions, would be easy for Subject 1. On the other hand, the storytelling game would be challenging for her.  In fact, as it is visible in Figures 4 to 6, both the imitation activity, as well as the recognition activity were executed flawlessly, and it is possible to see a significant increase in the percentage of correct answers between both sessions for the storytelling game mode. It is to note that subject 1 was highly participative and stated that she enjoyed these activities very much.     Figure 4. Results for the imitation activity from S1     Figure 5. Results for the recognition activity from S1  Figure 6. Results for the storytelling activity from S1 B. Subject 2 Subject 2 demonstrated a good understanding of the game and it is clearly visible the improvement in both time and percentage of correct answers for the recognition activity (Fig. 7).  The imitation game mode was slightly more difficult, but S2 was able to identify emotions and replicate them, for the most part (Fig. 8). The storytelling activity was by far the hardest (Fig. 9), which was expected. It should also be mentioned that even though the number of correct answers for this game mode improved in the second session, the therapist needed to be more interventive, which did not happen for the two other game modes, in any of the sessions.  Figure 7. Results for the recognition activity from S2   Figure 8. Results for the imitation activity from S2 160Copyright (c) IARIA, 2018.     ISBN:  978-1-61208-660-6SENSORDEVICES 2018 : The Ninth International Conference on Sensor Device Technologies and Applications Figure 9. Results for the storytelling activity from S2 C. Subject 3 This child had already studied emotions before and was very participative, which contributed to the good performance demonstrated and visible in Figures 10 to 12. The main issue was the “fear” of giving wrong answers, something that was more present in the last sessions and even led him to quit the activity. Improvements are very visible, especially in the first two game modes.  The storytelling activity is a particular case, because the data gathered from session 1 is only regarding seven answers. Subject 3 quit the activity at that point and only completed all the fifteen stories on the second session.   Figure 10. Results for the imitation activity from S3  Figure 11. Results for the recognition activity from S3  Figure 12. Results for the storytelling activity from S3  D. Subject 4 Subject 4 was very young and had a lot of difficulties playing the game in a correct manner. Most of the time he was distracted, focusing on the screen of the Android device and trying to touch the avatar face. He could not replicate emotions and did not show any interest in doing so, which led that activity to fail completely.  The recognition game mode followed similar steps. The therapist had to intervene in the session trying to obtain an answer from the child, which was almost always the wrong one.  Since both the imitation and the recognition activities were unsuccessful, the conclusion was that there was no point in keep trying during further sessions, because he would not engage in the activity, due to the lack of attention demonstrated.  E. Subject 5 This child was very participative and autonomous, stating more than once that the game was interesting and funny to play. Her main difficulty was interpreting emotions, especially when those emotions were shown by the emojis. In the imitation game mode, she had an easier time replicating the facial expressions than identifying emotions (Fig. 13).  Similarly to the imitation activity, the recognition game was played during all four sessions. The results are visible in Fig.14 and show an improvement in the number of correct answers, even though interlinked with an increase in response time. The storytelling activity was the one where she showed more difficulties (Fig. 15), what was expected since she would have to interpret specific situations and have a deeper knowledge regarding emotions.    161Copyright (c) IARIA, 2018.     ISBN:  978-1-61208-660-6SENSORDEVICES 2018 : The Ninth International Conference on Sensor Device Technologies and Applications Figure 13. Results for the imitation activity from S5  Figure 14. Results for the recognition activity from S5  Figure 15. Results for the storytelling activity from S5 F. Subject 6 No data was gathered regarding subject 6 because he did not engage in the activity at all.  The lack of attention was the main issue. This child went into the room and shifted his focus between everyone and everything around him. Furthermore, he could not understand the game or even pay attention to the application long enough to start playing.   IV. CONCLUSIONS This paper presented the work developed using a serious game and a playware object, aimed at improving the socio-emotional skills of children with ASD. The main goal was to determine if this application would be well accepted by the target group and its impact in teaching and/or facilitating the learning of emotions. After testing it first with six typically developing children and later with six children with ASD (these ones during four sessions), it was noticeable that both the playware and the serious game are intuitive, attractive, and easy to use. The entire application was well received and even though the sample size was too small, improvements were visible, indicating that this type of tool could act as a mediator in the learning of emotional skills. Future work with the application will be based on more testing, with a larger group and over a larger period. This type of testing allows for a better support and provides more reliability to the results and respective conclusions, consequently determining if this type of tool really acts as a facilitator in the learning of social skills. There is also a high possibility of integrating this application with other works developed by the group Robótica-Autismo (http://robotica-autismo.dei.uminho.pt/) which would involve humanoid robots like ZECA [15] ACKNOWLEDGMENTS The authors would like to express their acknowledgments to COMPETE: POCI-01-0145-FEDER-007043 and FCT – Fundação para a Ciência e Tecnologia within the Project Scope: UID/CEC/00319/2013. Vinicius Silva also thanks FCT for the PhD scholarship SFRH/BD/ SFRH/BD/133314/2017. The authors thank the teachers and students of the Elementary School of Gualtar (EB1/JI Gualtar) in Braga for the participation in the tests.   REFERENCES [1] H. Noor, F. Shahbodin, and N. Pee, “Serious game for autism children: review of literature,” World Acad. Sci., vol. 6, no. 4, pp. 554–559, 2012. [2] A. Z. Hassan et al., “Developing the concept of money by interactive computer games for autistic children,” Proc. - 2011 IEEE Int. Multimedia, ISM 2011, pp. 559–564, 2011. [3] A. Anwar, M. M. Rahman, S. M. Ferdous, S. A. Anik, and S. I. Ahmed, “A computer game based approach for increasing fluency in the speech of the autistic children,” Proc. 2011 11th IEEE Int. Conf. Adv. Learn. Technol. ICALT 2011, pp. 17–18, 2011. [4] M. Frutos, I. Bustos, B. G. Zapirain, and A. M. Zorrilla, “Computer game to learn and enhance speech problems for children with autism,” Proc. CGAMES’2011 USA - 16th Int. Conf. Comput. Games AI, Animat. Mobile, Interact. Multimedia, Educ. Serious Games, pp. 209–216, 2011. [5] S. Bernardini, K. Porayska-Pomsta, and T. J. Smith, “ECHOES: An intelligent serious game for fostering social communication in children with autism,” Inf. Sci. (Ny)., vol. 264, pp. 41–60, 2014. [6] Z. S. S. De Urturi, A. M. M. Zorrilla, and B. G. B. G. Zapirain, “Serious Game based on first aid education for individuals with Autism Spectrum Disorder (ASD) using 162Copyright (c) IARIA, 2018.     ISBN:  978-1-61208-660-6SENSORDEVICES 2018 : The Ninth International Conference on Sensor Device Technologies and Applicationsandroid mobile devices,” Proc. CGAMES’2011 USA - 16th Int. Conf. Comput. Games AI, Animat. Mobile, Interact. Multimedia, Educ. Serious Games, pp. 223–227, 2011. [7] H. H. Lund, T. Klitbo, and C. Jessen, “Playware technology for physically activating play,” Artif. Life Robot., vol. 9, no. 4, pp. 165–174, 2005. [8] G. N. Yannakakis, O. M, J. Hallam, and H. H. Lund, “Comparative Fun Analysis in the Innovative Playware Game Platform,” Proc. 1st World Conf. Fun ’n Games, pp. 64–70, 2006. [9] H. H. Lund and P. Marti, “Designing modular robotic playware,” Proc. - IEEE Int. Work. Robot Hum. Interact. Commun., pp. 115–121, 2009. [10] H. H. Lund, L. S. D. Jensen, Y. Ssessanga, S. Cataldo, and K. I. Yahya-Malima, “An approach for a national eHealth implementation - The case of modular interactive tiles for rehabilitation,” 2015 IST-Africa Conf. IST-Africa 2015, pp. 1–10, 2015. [11] K. Suzuki, “Social Playware: Device-mediated social interaction for therapeutic activities,” Proc. - IEEE Int. Work. Robot Hum. Interact. Commun., vol. 2014–Octob, no. October, pp. 69–72, 2014. [12] S. Costa, F. Soares, and C. Santos, “Facial expressions and gestures to convey emotions with a humanoid robot,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 8239 LNAI, pp. 542–551, 2013. [13] H. Kozima, M. P. Michalowski, and C. Nakagawa, “Keepon: A playful robot for research, therapy, and entertainment,” Int. J. Soc. Robot., vol. 1, no. 1, pp. 3–18, 2009. [14] B. Vanderborght et al., “Using the social robot probo as a social story telling agent for children with ASD,” Interact. Stud., vol. 13, no. 3, pp. 348–372, 2012. [15] S. Costa, F. Soares, A. P. Pereira, C. Santos, and A. Hiolle, “A pilot study using imitation and storytelling scenarios as activities for labelling emotions by children with autism using a humanoid robot,” IEEE ICDL-EPIROB 2014 - 4th Jt. IEEE Int. Conf. Dev. Learn. Epigenetic Robot., pp. 299–304, 2014. [16] P. Chevalier et al., "Dialogue Design for a Robot-Based Face-Mirroring Game to Engage Autistic Children with Emotional Expressions ", “Social Robotics,” vol. 7621, 4th International Conference, ICSR 2012, Chengdu, China, October 29-31, 2012., pp. 546–555, 2012. [17] “Aims &amp; Objectives – DE-ENIGMA.” [Online]. Available: http://de-enigma.eu/features/aims-objectives/. [Retrieved: Feb,2018]. [18] A. Ghorbandaei Pour, A. Taheri, M. Alemi, and A. Meghdari, “Human–Robot Facial Expression Reciprocal Interaction Platform: Case Studies on Children with Autism,” Int. J. Soc. Robot., pp. 179-198, 2018. [19] D. O. David, C. A. Costescu, S. Matu, A. Szentagotai, and A. Dobrean, “Developing Joint Attention for Children with Autism in Robot-Enhanced Therapy,” Int. J. Soc. Robot., pp. 1-11, 2018.   163Copyright (c) IARIA, 2018.     ISBN:  978-1-61208-660-6SENSORDEVICES 2018 : The Ninth International Conference on Sensor Device Technologies and Applications

Serious games assisted by playware as a way to improve socio-emotional skills in children with autism spectrum disorder

http://repositorium.sdum.uminho.pt/bitstream/1822/59059/1/Serious%20games%20assisted%20by%20playware%20as%20a%20way%20to%20improve.pdf

Serious games assisted by playware as a way to improve socio-emotional skills in children with autism spectrum disorder

Abstract

Similar works

Full text

Available Versions

Universidade do Minho: RepositoriUM