Christinaki, Eirini

Triantafyllidis, Georgios

Vidakis, Nikolaos

The recognition of facial expressions is important for the perception of emotions. Understanding emotions is essential in human communication and social interaction. Children with autism have been reported to exhibit deficits in the recognition of affective expressions. With the appropriate intervention, elimination of those deficits can be achieved. Interventions are proposed to start as early as possible. Computer-based programs have been widely used with success to teach people with autism to recognize emotions. However, those computer interventions require considerable skills for interaction. Such abilities are beyond very young children with autism as they have major restriction in their ability to interact with computers. Our approach takes account of the specific characteristics of preschoolers with autism and their physical inabilities. By creating an educational computer game which provides physical interaction, we aim to support early intervention and to foster emotion learning

    Aalborg UniversitetA gesture-controlled Serious Game for teaching emotion recognition skills topreschoolers with autismChristinaki, Eirini; Triantafyllidis, Georgios; Vidakis, NikolaosPublication date:2013Document VersionPeer reviewed versionLink to publication from Aalborg UniversityCitation for published version (APA):Christinaki, E., Triantafyllidis, G., & Vidakis, N. (2013). A gesture-controlled Serious Game for teaching emotionrecognition skills to preschoolers with autism. Poster session presented at Foundations of Digital Games,Chania, Greece.General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright ownersand it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.            ? Users may download and print one copy of any publication from the public portal for the purpose of private study or research.            ? You may not further distribute the material or use it for any profit-making activity or commercial gain            ? You may freely distribute the URL identifying the publication in the public portal ?Take down policyIf you believe that this document breaches copyright please contact us at vbn@aub.aau.dk providing details, and we will remove access tothe work immediately and investigate your claim.Downloaded from vbn.aau.dk on: April 24, 2017A gesture-controlled Serious Game for teaching emotion recognition skills to preschoolers with autism Eirini Christinaki  Dep. Applied Informatics & Multimedia Technological Educational Institute of Crete, 715 00 Heraklion – Crete echrist@ics.forth.gr Georgios Triantafyllidis Medialogy Section, AD:MT  Aalborg University Copenhagen  A.C.Meyers Vænge 15 gt@create.aau.dk Nikolaos Vidakis Dep. Applied Informatics & Multimedia Technological Educational Institute of Crete, 715 00 Heraklion – Crete vidakis@epp.teicrete.gr  ABSTRACT The recognition of facial expressions is important for the perception of emotions. Understanding emotions is essential in human communication and social interaction. Children with autism have been reported to exhibit deficits in the recognition of affective expressions. With the appropriate intervention, elimination of those deficits can be achieved. Interventions are proposed to start as early as possible. Computer-based programs have been widely used with success to teach people with autism to recognize emotions. However, those computer interventions require considerable skills for interaction. Such abilities are beyond very young children with autism as they have major restriction in their ability to interact with computers. Our approach takes account of the specific characteristics of preschoolers with autism and their physical inabilities. By creating an educational computer game which provides physical interaction, we aim to support early intervention and to foster emotion learning. Keywords Serious Games, Gesture-based interaction, Natural User Interface, Autism, Facial emotion recognition 1. INTRODUCTION Social interaction impairments, a core feature of ASD, involve difficulties in understanding and expressing emotions [1]. Children with autism often fail to recognize the qualitative differences and associations between the various expressions of emotions. Those deficits seem to be rather permanent in individuals with autism so intervention tools for improving those impairments are desirable. Also, as the number of children diagnosed with autism increased [2], new methods for educating this population become necessary.     It is claimed that Computer Assistive Technologies (CAT) and in particular serious games [3] can be very effective [4] in the areas of therapy and education for children with autism. Furthermore, educational interventions for teaching emotion recognition from facial expressions should occur as early as possible in order to be successful and to have a positive effect [5]. However, those computer interventions require considerable skills and in particular the ability to control the mouse or to use the keyboard. Before the age of 6, most children with autism experience motor control problems in their hands and demonstrate delays in fine motor skills which cause difficulties in grasping and manipulating objects, such as a mouse [6]. 2. DESIGN AND IMPLEMENTATION  In this paper we present an educational computer-based single player game specially designed for Greek preschoolers with autism. Our aim is to teach them facial emotion recognition so as to enhance their social interaction.   The development of our game was based on the following main principles. As a serious game, it should have an impact on the player in a real life context [7]. A recent study conducted to analyze user needs for serious games for teaching children with ASD emotions, revealed the characteristics of the children’s game play behaviors [8]. The observation showed repetition, matching instead of learning the features, lack of holistic face processing and deliberately incorrect selection. On account of those findings, our game was intentionally designed to avoid those behaviors. Additionally, our design incorporates a theory-driven game design framework supported by learning and developmental theories. The framework is based on the integration of the experiential learning model and the cognitive model [9]. From the above systematic approach, six essential elements (matching, recognition, observation, understanding, generalizing and mimicking) were extracted for designing games to teach children with ASD emotions. These elements were taken into consideration during the design process of our serious game.   2.1 Game Environment  The game environment is simple and less detailed in order to avoid children’s distraction. Individuals with autism are reported to have enhanced perception of details [10] which may cause distraction. For these reasons we have chosen black context presented on a white background and grayscale stimuli. Game begins with an instruction page where the child is informed what is going to happen, what he/she has to do and how he/she can do it. When the child feels ready, he/she can choose to start the game. The game provides a structure learning environment which consists of 3 different levels with increasing difficulty. Breaking the teaching intervention into small learning steps makes the task easier to perform. In the first level children should learn labeling emotions, in the second level they should learn to recognize emotions from facial features and in the third level they should learn to identify the causes of various feelings in different situations, obtained through the use of social stories.  Individuals with autism are usually visual learners, which mean that they understand written words, photos and visual information better than spoken language. For teaching emotions, it is  Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. FDG 2013, May 14-17, 2013, Chania, Crete, Greece. Copyright 2010 ACM 1-58113-000-0/00/0010 …$15.00.  recommended to describe each feeling pictorially by using pictures with clear outline, minimal details and color [11]. For young children it is advisable to keep to the basic emotions (happy, sad, angry, scared and surprised). The face stimuli we used are 15 grayscale photographs of male and female faces, taken from the CAlifornia Facial Expressions (CAFE) dataset [12]. All images used in the game, meet FACS criteria [13] and all faces have been certified as “FACS-correct”. The stimuli are presented on each trial with different pair of photos and the goal is to choose the correct image among the two. 2.2 System Development Our game is implemented with the use of Kinect, which is a motion sensing input device by Microsoft for Windows PCs. Kinect is a cheap and simple device for motion capturing. It offers simple and reliable skeleton tracking as well as an open source SDK. By using Visual studios 2010 we were able to use the Kinect SDK 1.6 released by Microsoft with C# as backend. The advantages of using C# were a) capability to  integrate XNA Game Studio 4.0 to develop our game using Kinect and b) possible to utilize the XNA libraries (provided by Microsoft) to create the graphics. 2.3 Interaction with the System  Interaction may be one of the areas that need to be developed with extreme care. Our gesture-based interaction approach moves the control of computer from a mouse and keyboard, to the motions of the body via new input devices. Our game is designed to use non-touch based NUI and to be controlled by hand gestures. The player has three possible actions in all game states, to choose the left or the right image and to move to the next play area. These basic actions are implemented with efficient and easy to use gestures. Moving to the next play area requires a two-hand gesture which is performed by moving both hands above the head. Selecting the left or the right image (the orientation of the image is decided by looking toward the screen) requires one-hand gesture which is performed by moving the left or the right hand above the head. During the game, if the player selects the correct or incorrect stimuli, the system will provide an audio and a visual feedback such as operation-related sounds and changing the images’ color. A voice telling “Bravo” rewards player for the correct answer and for the incorrect answer a voice telling “Try again” encourages the user to retry. There are no other sound effects because individuals with ASD may suffer from auditory sensitivity and may feel discomfort when exposed to certain sounds [14]. Visual feedback is also provided by changing the image’s color into light green for the correct answer and into light red for incorrect answer. Light colors were selected because in ASD occur a reduced chromatic discrimination that is due to general reduction in sensitivity [15].  3. DISCUSION AND FUTURE WORK Serious games with NUI interaction are a promising intervention strategy because they are appealing and motivating for young children to use as well as convenient to access. When the control is done through natural gestures, the user does not have to learn how to perform the action and how to operate the game.  Future work could include the design of special gestures aiming to improve children fine motor skills or the design of a serious game that will enable children to make facial emotion expressions that the system will be able to detect in order to support the “learn by doing” method. 4. REFERENCES [1] White S. W., Keonig K. and Scahill L. 2007. Social skills development in children with autism spectrum disorders: A review of the intervention research. Journal of Autism and Developmental Disorders. 37, 10, 1858-1868. [2] Baio J. 2012. Prevalence of autism spectrum disorders--Autism and Developmental Disabilities Monitoring Network, 14 sites, United States, 2008. MMWR Surveillance Summaries. 61, 3, 1-19. [3] Djaouti D., et al. 2011. Origins of Serious Games. Serious Games and Edutainment Applications. 25-43. [4] Ploog B. O., et al. 2012. Use of Computer-Assisted Technologies (CAT) to Enhance Social, Communicative, and Language Development in Children with Autism Spectrum Disorders. Journal of Autism and Developmental Disorders. 1-22. [5] Stahmer A. C., Akshoomoff N. and Cunningham A. B. 2011. Inclusion for toddlers with autism spectrum disorders The first ten years of a community program. Autism. 15, 5, 625-641. [6] Jasmin E., et al. 2009. Sensori-motor and daily living skills of preschool children with autism spectrum disorders. Journal of Autism and Developmental Disorders. 39, 2, 231-241. [7] Mitgutsch K. and Alvarado N. 2012. Purposeful by design?: a serious game design assessment framework. Proceedings of the International Conference on the Foundations of Digital Games, 121-128. [8] Abirached B., Zhang Y. and Park J. H. 2012. Understanding User Needs for Serious Games for Teaching Children with Autism Spectrum Disorders Emotions. Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications. 1054-1063. [9] Park J. H., Abirached B. and Zhang Y. 2012. A framework for designing assistive technologies for teaching children with ASDs emotions. Extended Abstracts on Human Factors in Computing Systems. 2423-2428. [10] Ashwin E., et al. 2009. Eagle-eyed visual acuity: an experimental investigation of enhanced perception in autism. Biological Psychiatry, 65, 1, 17-21. [11] Dodd S. 2004. Understanding Autism, Sydney: Elsevier. [12] Dailey M. N., Cottrell G. W. and Reilly J. 2001. CAlifornia Facial Expressions (CAFE). [Online]. Available: http://www.cs.ucsd.edu/users/gary/CAFE/. [13] Ekman P. and Friesen W. V. 1978. Facial Action Coding System, Palo Alto: Consulting Psychologist Press. [14] Tan Y. H., et al. 2012. Auditory abnormalities in children with autism. Open Journal of Psychiatry. 2, 1, 33-37. [15] Franklin A., et al. 2010. Reduced chromatic discrimination in children with autism spectrum disorders. Developmental Science, 13, 1, 188-200. 

A gesture-controlled Serious Game for teaching emotion recognition skills to preschoolers with autism

http://vbn.aau.dk/files/195820996/FDG_2013_paper_185.pdf

A gesture-controlled Serious Game for teaching emotion recognition skills to preschoolers with autism

Abstract

Similar works

Full text

Available Versions

VBN