The work presented in this paper is one of the outcomes of the virtual sign project that aims to assist the
communication with deaf students in the classroom. The project is being developed by Portuguese researchers. The main goal of the virtual sign project is the creation of a bidirectional Portuguese sign language translator. This translator supports the development of a serious game that was developed to facilitate the sign language learning process. In the game, it is possible to gather different gestures and reproduce them. The game includes three different scenarios. These scenarios are linked to increasing levels of proficiency in sign language. The first one
addresses the alphabet, the second one goes a step forward to teach words, and the third one introduces full
sentences. The game experience can be enhanced by using the Kinect to perform the gestures. This game intends to
be of great assistance in the process of learning the Portuguese sign language.This work was supported by Engineering Institute of Oporto and GILT (Graphics, Interaction and Learning
Technologies) under Grant Nos. 60973089. FCT (Fundação para a ciência e Tecnologia) project.info:eu-repo/semantics/publishedVersio

Baltazar, Ana Bela

Barbosa, Maciel

Bidarra, José

Escudeiro, Nuno

Escudeiro, Paula

Lopes, Jorge

Norberto, Marcelo

Reis, Rosa

English

Repositório Aberto da Universidade Aberta

China-USA Business Review, May 2015, Vol. 14, No. 5, 261-268 doi: 10.17265/1537-1514/2015.05.004  Virtual Sign—Using a Bidirectional Translator in  Serious Games* Marcelo Norberto, Jorge Lopes, Paula Escudeiro, Nuno Escudeiro, Rosa Reis  Instituto Superior de Engenharia do Porto, Porto, Portugal Maciel Barbosa  Universidade do Porto, Porto, Portugal José Bidarra  Universidade Aberta, Porto, Portugal Ana Bela Baltazar Center for Translators and Interpreters of Sign Language, Porto, Portugal  The work presented in this paper is one of the outcomes of the virtual sign project that aims to assist the communication with deaf students in the classroom. The project is being developed by Portuguese researchers. The main goal of the virtual sign project is the creation of a bidirectional Portuguese sign language translator. This translator supports the development of a serious game that was developed to facilitate the sign language learning process. In the game, it is possible to gather different gestures and reproduce them. The game includes three different scenarios. These scenarios are linked to increasing levels of proficiency in sign language. The first one addresses the alphabet, the second one goes a step forward to teach words, and the third one introduces full sentences. The game experience can be enhanced by using the Kinect to perform the gestures. This game intends to be of great assistance in the process of learning the Portuguese sign language. Keywords: serious games, sign language, education, Kinect, accessibility, deaf community                                                           Acknowledgments: This work was supported by Engineering Institute of Oporto and GILT (Graphics, Interaction and Learning Technologies) under Grant Nos. 60973089. FCT (Fundação para a ciência e Tecnologia) project. Marcelo Norberto, licentiate in informatics engineering, Instituto Superior de Engenharia do Porto, Games Interaction and Learning Technologies, Porto, Portugal. Jorge Lopes, licentiate in informatics engineering, Instituto Superior de Engenharia do Porto, Games Interaction and Learning Technologies, Porto, Portugal. Paula Escudeiro, Ph.D., adjunct professor and director of the GILT Research Group, Instituto Superior de Engenharia do Porto, Porto, Portugal. Nuno Escudeiro, Ph.D., professor of informatics, Instituto Superior de Engenharia do Porto, Porto, Portugal. Rosa Reis, associate professor of informatics, Instituto Superior de Engenharia do Porto, Porto, Portugal. Maciel Barbosa, Ph.D., professor of electrical engineering, Faculdade de Engenharia do Porto, Universidade do Porto, Porto, Portugal. José Bidarra, Ph.D., assistant professor of science and technology, Universidade Aberta, Porto, Portugal. Ana Baltazar, master in translation and interpretation of Portuguese sign language, President of the Center for Translators and Interpreters of Sign Language, Porto, Portugal. Correspondence concerning this article should be addressed to Marcelo Norberto, Rua Doutor António, Bernardino de Almeida 431, 4200-072 Porto, Portugal. E-mail: matno@isep.ipp.pt; marcelo_torresn@hotmail.com. DAVID  PUBLISHING D VIRTUAL SIGN—USING A BIDIRECTIONAL TRANSLATOR IN SERIOUS GAMES  262 Introduction This project is being developed within the field of sign language. The Portuguese sign language is processed through systematized gestures which involve the adoption of hand configurations and movement, including the body and facial expression, therefore the information is captured visually. Besides those      that really need to use this language, only a few know how to apply or understand it (Kyle & Woll, 1988). However, the surrounding community, such as teachers, technicians, relatives, and others, also need it.    Thus, the virtual sign project scope is to act directly in this part of the society, providing sign language learning tools. Despite being a field vaguely explored, projects have recently started appearing related to this topic. The translation of sign language to text requires an analysis of hand configurations, movement, body inclination, and facial expression. Sensor gloves (5DT) with 14 sensors each along with Microsoft Kinect are used to estimate the finger positioning providing information about the movement and orientation of the hands and facial expression. In the opposing way (with the written Portuguese language translation to sign language), the text is processed and the gestures are shown by a 3D avatar. As a demonstration of virtual sign project potentialities, there is a didactic game in development which goal is to facilitate the sign language learning process. The game induces the user to learn sign language through an innovating and fun model whilst interacting with a virtual world, using gesture execution of sign language in order to concretize the intermediate goals and accomplish the missions along the game. The various objectives of this project are being developed with the support of technologies, such as Blender (Brito, 2007), Unity 3D, Visual Studio, Ogre, C#, and C++.  Background Literature One of the components of this project consists of a didactic game about sign language, where the player can enjoy the game while learning gestures simultaneously. Kinect has also been integrated into the game in order to make it more interactive and appealing.  The main objective of this project is to facilitate the learning of sign language and improve the dexterity of those who already know, making learning a pleasurable experience. The deaf community in Portugal is around 100,000 individuals and yet the digital content available for this community is still rather low (McNeill, 2000; Sandler, 2006). With this project, not only something more is being created for this community, it is also encouraging other people to learn and become able to better understand this community. This research showed that there are some projects related to this theme/area, but none of them implies an automatic bidirectional translation process in a game as this one does, therefore, making this project very innovative. There are a rising number of serious games projects. Some of the most relevant related works within the sign language scope are described below. The game Copycat is the most similar project in comparison to this research. It consists of a game where sign language gestures need to be executed properly in order to proceed. The movement analysis is done through gloves with sensors. However, the researchers from the Copycat project have published a video where they show their intention to use Kinect for movement detection. Their current research platform is a custom system that uses computer vision, colored gloves, and wrist-mounted three-axis accelerometers to collect data as users sign and machine learning to recognize the signs for game play. That system was built on top of VIRTUAL SIGN—USING A BIDIRECTIONAL TRANSLATOR IN SERIOUS GAMES  263Ubuntu Linux and uses the Kinect system as input for the computer vision, which replaces the gloves and sensors.  ProDeaf is an application that does the translation of Portuguese text or voice to Brazilian gesture language. This project is not a serious game, but it is very similar to one of the main components used on the virtual sign game, which is the text to gesture translation. The objective of the ProDeaf is to make the communication between mute and deaf people easier by making digital content accessible in Brazilian gesture language. The translation is done using a 3D avatar that performs the gestures. This software is already used by over 130,000 users. Kinect Sign Language Translator is another project that is similar to the virtual sign translator. The project was a result of collaboration, facilitated by Microsoft research, among the Chinese Academy of Sciences, Beijing Union University, and Microsoft Research Asia, each of which made crucial contributions. Dedicated researchers in China have created the Kinect Sign Language Translator, a prototype system that understands the gestures of sign language and converts them to spoken and written language—and vice versa. The system captures a conversation from both sides: It displays the signer and renders a written and spoken translation of the sign language in real-time, and it also takes the non-signer’s spoken words and turns them into accurate and understandable sign language. An avatar on the screen represents the non-signer and makes the appropriate sign languages gestures. Game Based Learning Game-based learning is a type of game played with defined learning outcomes. Nowadays, the conventional learning process does not always give the right motivation to the one who is learning, therefore, if the learner finds it boring, their motivation levels will be very low making the process of retaining information harder. The $30 billion worldwide computer and video games industry alone show how important games can be. Games are known to be fun and enjoyable to play. It is common for people to play games for hours. The use of games as learning tools can make the learning process more enjoyable, and if the learner has fun with it, he will keep playing more. In a good educational game, the player will learn what he needs to know, whilst having fun playing the game. Game Play The game is played in first person view in which the player controls a character on the map. Each map represents a level and each level has several scattered objects through the map for the player to interact with. All objects obtained by the player will be stored in his inventory and can be accessed at any point during the game.  Along the game, the player will be able to play mini-games in order to obtain some gestures. Most of these gestures and objects can be used through the inventory. The character will then perform the gesture, so that the user can visualize how it is done. To progress in the game, you must collect all the gestures scattered around the level. The faster the player manages to collect all the gestures, the higher his score will be. Should the player choose to play using Kinect, he must have first obtained all the gestures in the level and perform them themselves. VIRTUAL SIGN—USING A BIDIRECTIONAL TRANSLATOR IN SERIOUS GAMES  264 The game has three main scenarios: the first one for letters and numbers, the second with words, and the third with sentences. After each scenario, there is also an evaluation test to check the player improvement. The first functionality developed was the inventory. In the inventory, the items acquired by the player would be saved and could be accessed at any time. Then scripts to interact with the objects were created. These scripts were optimized to be reused for multiple objects without having to resort to changing the code. Several objects were distributed over the game map, each one with a script allowing the player to collect the object and add it to the inventory. Finally, the graphic interface was created (Blackman, 2011).  The inventory consists of 42 spaces that are empty upon initialization. With the inventory set up and ready to receive the objects that the user can acquire, the handling of collisions with these objects was created in order to detect when the user is within a reasonable distance to perform the interaction (Creighton, 2010).  The player’s score is incremented during the game, as they acquire new gestures. The shorter the time it takes between the acquisitions of two objects, the greater the score will be. If the delay is less than one minute, 100 points are acquired; if it is between one and two minutes, 50 points are acquired; and if the delay exceeds two minutes, 25 points will be obtained regardless of time spent. Some of the gestures can only be acquired through mini-games where the player must score a certain amount of points in order to get the gesture. Some of the mini-games are on the test level. The test level is where the player performance is evaluated at the end of each scenario to check what knowledge he has acquired. On these levels, the player will be asked to perform the gestures or beat games in order to get more points. After finishing the structure of the game, a connection with the virtual sign application (Vsapp) was established using sockets. The Vsapp detects and translates the gestures that the user makes, to text. With the connection established, authors are able to ask the user to perform a gesture and with the data provided by the Vsapp confirm if the gesture is made properly. If not, the user is notified of what gesture he did and which gesture was requested. It’s only possible to proceed in the game when the user manages to perform the requested gesture correctly. Figure 1 bellow shows the first scene of the game in which the score can be seen on top of the screen, the timer on the top right, and the mini-map on the bottom right.   Figure 1. Virtual sign game.  VIRTUAL SIGN—USING A BIDIRECTIONAL TRANSLATOR IN SERIOUS GAMES  265Requirements The functional requirements identified early in the project were:  Start menu, where the player can choose the type of game he wants (with or without Kinect), consults the options or exit;  Menu options, where you can change the graphics quality and volume, save or load the game and see the table of high scores; The game requirements within the levels are:  handling and controlling of the character;  interacting with non-player characters (NPCs);  consulting the inventory and use the items in it;  interaction with map objects;  access to the above options menu;  access to the mini-games. This project aims to be fairly intuitive, allowing easy adaptation and learning. The interfaces were developed with the care to enable a pleasant interaction. The character controls were also structured to present a simple usage. Along the game, there are several short explanations of how the player should act to fulfill the objectives and surpass the levels. The gaming performance is a factor of the utmost importance. Any perceptive delay can affect the game play, making the game annoying rather than fun. To maintain the performance, this game was tested to never run less than 60 frames per second on an optimal computer. The ideal frame rate for a game must be around 40 frames per second. The essential functions are constantly executed and the code must be optimized to avoid the waste of resources.  Besides the code, all the factors that constrain the performance of the game are taken into account, such as textures, bumps, and number of vertices of the 3D models among others. Based on all these factors, the performance of the game is assured. Game Design The project is divided into layers, given its high degree of complexity. At the top level, there is the interface. All the functionalities of the project can be accessed through this layer by the user. This layer is responsible for the transmission of the actions of the user for the following layers. On the lower level, there are three layers. The sockets layer is responsible for linking unity to the virtual sign translator (Catue, 2012). The game engine layer is responsible for the execution of the game itself, representing the functions of unity. Finally, the business layer is where the game functions are available to the player, hence why is it directly below interface as it can be seen in Table 1 bellow?  Table 1 Application Architecture User Interface Sockets Game engine Business layer Hardware  VIRTUAL SIGN—USING A BIDIRECTIONAL TRANSLATOR IN SERIOUS GAMES  266 In the development of this project, a draft was first developed implementing the basic functionalities (such as avatar animations). After having a basic scenario created, the development phase of the scripts started. The character motion scripts used were provided by unity (Goldstone, 2011).  Scenarios There are three main scenarios in this game (Mariais, Michau, & Pernin, 2010). The scenarios were built in accordance to the game plot. The first one can be seen in Figure 2. The first one is the one where the player can obtain letters and numbers and it takes place in a desert. There are three main checkpoints where the player must perform the gestures and only after passing those checkpoints, the player can access the next area of the scenario. Each area is shown in the following image as well as the full scenario.  The second scenario can be seen in Figure 3 and has to use them to acquire new gestures. It also has checkpoints, however, unlike the first, where there are three checkpoints where the player has to perform six gestures in each; on this second level, the player only has to perform one word per checkpoint, but there are as many checkpoints as words.   Figure 2. First scenario.   Figure 3. Second scenario.  VIRTUAL SIGN—USING A BIDIRECTIONAL TRANSLATOR IN SERIOUS GAMES  267As for the third and last scenario, it can be seen in Figure 4, the scenario provided by one of unity assets was used and complemented with what NPCs and objects necessary for the task, the logic of this scene is just as the second level except that instead using words, the player uses full sentences.   Figure 4. Third scenario.  Other than these three main scenes, there is the starting menu, the evaluation levels, and mini-games. Conclusions With this project, a larger support to the deaf and mute community was created and the sign language learning process has now an extra tool to support it. Since there are not many completed projects in this field, this project becomes even a greater asset for the development of Portuguese sign language. Throughout the tests that were accomplished, a slight variation was detected during the execution of the gestures depending on the person, however, those small variations were ignored, therefore obtaining higher success rate when classifying the signs. With this, this paper can assure that the application can be used by any users even if the person does not have any knowledge in sign language.  This project optimizes the learning and communication through Portuguese sign language, increasing the possibilities of those who do not have knowledge on the matter. The project is directed to all those that want to improve or learn their sign language skills. References Blackman, S. (2011). Beginning 3D game development with unity: All-in-one, multi-platform game development. New York: Apress. Brito, A. (2007). Blender 3D-Guia do Usuário. São Paulo: Novatec Publishing Ltd.  Catue, D. (2012). Programming with the Kinect for windows: Software development kit. Retrieved from https://www.microsoft.com/learning/en-us/book.aspx?id=16076&locale=en-us Creighton, R. H. (2010). Unity 3D game development by example: A seat-of-your-pants manual for building fun, groovy little games quickly. Olton: Packt Publishing Ltd. VIRTUAL SIGN—USING A BIDIRECTIONAL TRANSLATOR IN SERIOUS GAMES  268 Goldstone, W. (2011). Unity 3x game development essentials. Olton: Packt Publishing Ltd. Kyle, J. G., & Woll, B. (1988). Sign language: The study of deaf people and their language. Cambridge: Cambridge University Press. Mariais, C., Michau, F., & Pernin, J. P. (2010). The use of game principles in the design of learning role-playing game scenarios. Proceedings from European Conference on Games Based Learning. McNeill, D. (2000). Language and gesture. Cambridge: Cambridge University Press.  Sandler, W. (2006). Sign language and linguistic universals. Cambridge: Cambridge University Press. 

Virtual sign: using a bidirectional translator in serious games

https://repositorioaberto.uab.pt/bitstream/10400.2/14513/1/Virtual%20Sign%e2%80%94Using%20a%20Bidirectional%20Translator.pdf

Virtual sign: using a bidirectional translator in serious games

Abstract

Similar works

Full text

Available Versions

Repositório Aberto da Universidade Aberta