Comunicação apresentada na 7ª Conferência Internacional de Arte Digital realizada em Óbidos de 19-20 de março de 2015This paper presents the development of a game aimed at making the process of learning sign language enjoyable and interactive, using the VirtualSign Translator. In this game the player controls a character that interacts with various objects and non-player characters with the aim of collecting several gestures from the Portuguese Sign Language. Through the connection with VirtualSign Translator the data gloves and Kinect support this interaction and the character can then represent the gestures. This allows for the user to visualize and learn or train the various existing configurations of gestures.  To improve the interactivity and to make the game more interesting and motivating, several checkpoints were placed along game levels. This provides the players with a chance to test the knowledge they have acquired so far on the checkpoints, after performing the signs using Kinect.  A High Scores system was also created, as well as a History option, to ensure that the game is a continuous and motivating learning process

Baltazar, Ana Bela

Barbosa, Maciel

Bidarra, José

Escudeiro, Nuno

Escudeiro, Paula

Lopes, Jorge

Norberto, Marcelo

Reis, Rosa

Rodrigues, Pedro

The Repositório Aberto

A Kinect Game in the VirtualSign Project: Training and 
Learning with Gestures 
Paula Escudeiro*, Nuno Escudeiro*, Rosa Reis*, Ana Bela Baltazar*, Pedro Rodrigues*, Jorge 
Lopes*, Marcelo Norberto*, Maciel Barbosa**, and José Bidarra*** 
*Engineering Institute of Oporto, Portugal; **Faculty of Engineering, University ofOporto, Portugal; ***CIAC, Universidade Aberta (Open University), Lisbon, Portugal 
Abstract  — This paper presents the development 
of a game aimed at making the process of learning sign 
language enjoyable and interactive, using the VirtualSign 
Translator. In this game the player controls a character 
that interacts with various objects and non-player 
characters with the aim of collecting several gestures from 
the Portuguese Sign Language. Through the connection 
with VirtualSign Translator the data gloves and Kinect 
support this interaction and the character can then 
represent the gestures. This allows for the user to visualize 
and learn or train the various existing configurations of 
gestures.  To improve the interactivity and to make the 
game more interesting and motivating, several checkpoints 
were placed along game levels. This provides the players 
with a chance to test the knowledge they have acquired so 
far on the checkpoints, after performing the signs using 
Kinect.  A High Scores system was also created, as well as a 
History option, to ensure that the game is a continuous and 
motivating learning process. 
Index Terms — Kinect game, game development, 
educational content, educational game, Portuguese Sign 
Language. 
I. INTRODUCTION 
In this day and age, games and simulations create 
remarkable opportunities to overcome the scarcity of 
educational digital content available for the hearing 
impaired community. Furthermore, freedom of choice, 
challenge, participation, transparency, integrity, 
collaboration, fun, speed, and innovation should become 
a part of students’ learning experiences. In this context, 
playing games may be an important aspect of learning 
as this generation’s game-playing experiences are more 
widespread than the game-playing experiences of 
previous generations. The interest in gaming for 
educational purposes has increased over the last decade, 
with researchers identifying key pedagogical features 
that make good games inherently strong learning tools. 
What underlies the allure of games? Educational game 
researcher James Gee [1] shows how good game 
designers manage to get new players to learn their long, 
complex, and difficult games. A well-designed game 
entices players into the “reality” of the game world and 
keeps them there until the goals of the game have been 
met [2]. Gee points out that incorporating appropriate 
challenges that are “doable,” and other widely accepted 
effective learning principles that are supported by 
research in cognitive science, are in fact a large part of 
what makes good games motivating and entertaining 
[3]. 
Making these opportunities available to those who 
endure handicap and disabilities is a core concern in 
today’s society and a must to promote equity and 
inclusion. In this work we propose a new approach by 
using a game to make the process of learning sign 
language enjoyable and interactive. In this game the 
player controls a character that interacts with various 
objects and non-player characters with the aim of 
collecting several gestures from the Portuguese Sign 
Language. The character can then perform these 
gestures himself. This allows the user to visualize and 
learn or train the various existing gestures. The user 
interacts with the game through data gloves and a 
Kinect device. To raise the interactivity and to make the 
game more interesting and motivating, several 
checkpoints were placed along game levels. This 
provides the players with a chance to test the knowledge 
they have acquired so far on the checkpoints, after 
performing the signs using Kinect. A High Scores 
system was also created, as well as a History option, to 
ensure that the game is a continuous and motivating 
learning process. 
II. THE GAME
The project consists of a didactic game about 
Portuguese Sign Language, where the player can enjoy 
the game while learning gestures simultaneously [4]. 
The VirtualSign Translator has been connected to the 
game [5] in order to make it more interactive, more 
appealing, and more effective using the translator Kinect 
and gloves. 
The main objective of this project is to facilitate the 
learning of the Portuguese Sign Language and to 
improve the dexterity of those who already know it, 
making learning a pleasant experience. 
The deaf community in Portugal has around 100 000 
individuals and yet the digital content available for this 
community is still rather scarce. With this project, not 
only we are promoting knowledge for this restricted 
community, but we are also encouraging other people to 
learn and become able to better understand this 
community [6]. 
The game is played in first person view, so the player 
controls a character in a specific scenario.  
Each map represents a level and each level has 
several scattered objects, through the map for the player 
to interact with, most of those objects are gestures. All 
objects collected by the player will be stored in his 
inventory and can be accessed at any point during the 
game. Most of these gestures and objects can be used 
through the inventory; the character will then perform 
the gesture so that the user can visualize how it is done, 
either automatically at checkpoints or by clicking them 
in the inventory. 
To progress in the game the player must collect all the 
gestures scattered around the level [7]. The faster the 
player manages to collect all the gestures the higher his 
score will be. 
The player may choose to play using Kinect but he 
must have first obtained all the gestures in the level and 
perform them. 
Our research showed that there are some projects 
related to this theme/area but none of them implies an 
automatic bidirectional translation process in a game as 
this does, therefor making this project very innovative.  
Some of the related work is described below. 
 
A. CopyCat  
The game CopyCat is the most similar project in 
comparison to ours. It consists of a game where sign 
language gestures need to be executed properly in order 
to proceed. The movement analysis is done through 
gloves with sensors. However, the researchers from the 
CopyCat project have published a video where they 
show their intention to use Kinect for movement 
detection. 
 
B. ProDeaf 
ProDeaf is an application that does the translation of 
Portuguese text or voice to Brazilian gesture language.  
The objective of the ProDeaf is to make the 
communication between mute and deaf people easier, 
making digital content accessible in Brazilian gesture 
language. 
 
C. Kinect Sign Language Translator in Beijing 
University 
Researchers in China have created the Kinect Sign 
Language Translator, a prototype system that 
understands the gestures of sign language and converts 
them to spoken and written language—and vice 
versa. This project was a result of collaboration, 
facilitated by Microsoft Research, between the Chinese 
Academy of Sciences, Beijing Union University, and 
Microsoft Research Asia, each of which made crucial 
contributions. 
 
 
 
D. Faceshift  
This programme analyses facial expressions, namely, 
the orientation of the user’s head and eyes. The 
information analysed by the application is used to 
animate virtual characters that can be used in videos or 
games. 
 
E. MoVER 
MoVER (Movement in Virtual Environment for 
Rehabilitation), a Serious Game that simulates 
physiotherapical movements through challenges to 
perform virtual tasks using the human body. This 
project was developed in the University of São Paulo. 
III. APPLICATION REQUIREMENTS 
In this section the functional and non-functional 
requirements of the application are described. 
Functional requirements represent the features available 
to the user. 
 
A. Functional Requirements 
The functional requirements were identified in the 
early stages of development of the project, and include 
the following: 
1. Start Menu, where the player can choose the type 
of game he wants (with or without Kinect), consult 
the options or exit; 
2. Options Menu, where you can change the graphics 
quality, volume, save or load the game and see the 
table of high scores. 
 
The game requirements within the levels are: 
- Handling and controlling of the character; 
- Interacting with NPCs (Non-Player Characters); 
- Consulting the inventory and use the items in it; 
- Interacting with map objects; 
- Accessing to the above options menu. 
 
B. Non-Functional Requirements 
Regarding non-functional requirements, our work 
focused on the following: 
 
1. Usability 
Usability is the ease of use of the application and its 
accessibility to the user [8]. 
With this in mind during the project design it is 
expected the interface to be fairly intuitive, allowing for 
easy adaptation and learning. The user interfaces must 
be simple so anyone can use them easily. 
The character controls are simple, and throughout the 
game there are explanations of what needs to be done. 
The accessibility is guaranteed because it is only 
necessary to run an executable file in order to play while 
Kinect is not in use.  
While playing using the Kinect, two applications are 
required as well as the necessary drivers, and also 
assuring that the Kinect is properly plugged in. 
 
2. Performance 
The gaming performance is always a factor of the 
utmost importance, because the response time from the 
game to the user is always immediate, any delay or 
decrease of the number of frames per second can affect 
the gameplay making the game frustrating rather than 
fun.  
To maintain the performance this game was tested to 
never run less than 60 frames per second on a computer. 
The essential functions must be constantly executed and 
the code must be optimized to avoid waste of resources.  
Besides the code, all the factors that constrain the 
performance of the game should be taken into account, 
such as textures, bumps, and number of vertices of the 
3D models among others.  
The connection to the VirtualSign translator is 
instantaneous and there is no delay from the moment of 
the translation of the preformed gesture to the moment 
its receive in the game. Therefor the player wont notice 
any delays while preforming the gestures. 
IV. GAME ARCHITECTURE 
For this project two applications were developed, the 
game application in Unity 3D and the interface that 
connects the virtual sign translator to Unity. The 
interface was developed in Microsoft Visual Studio. 
The project was divided into layers, given its high 
degree of complexity. 
At the top level there is the interface.  The user can 
access all the functionalities of the project through this 
layer. This layer is responsible for forwarding the 
actions of the user to the next layers. 
On the lower level there are three layers. The sockets 
layer is responsible for linking the Unity game 
application to the Kinect that is why this layer is below 
the interface in order to provide the layer above with the 
player input. Another layer of the lower level is is the 
game engine; this layer is responsible for the execution 
of the game itself, representing the functions of Unity. 
Finally, there is the business layer, which is where the 
game functions are available to the player. Figure 1 
shows the layers of game. 
 
 
Fig. 1. Layers of game) 
For the development of this project it was decided to 
start with the implementation of the basic functionality 
and then proceed to the animation of the character, 
which was later replaced by an animated avatar now 
available at the GILT research group (Graphic 
Interaction and Learning Technologies). 
In the development of this project a draft was first 
developed implementing the basic functionalities, as 
shown in Figure 2.  Following this the player’s avatar 
was animated, however it was eventually replaced by 
the one provided by GILT. 
 
 
Fig. 2. Scenario of the basic functionalities of the game 
After having a basic scenario created the development 
phase of the scripts started. 
All the translations are handled by the VirtualSign 
Translator therefore it’s not referenced in the game 
architecture as it is an different application. The sockets 
as mentioned before are responsible for the connection 
between the two applications and the information is 
transferred from one to another instantaneously.  
For this application the official Unity Sockets 
example was adapted, and a window was created where 
the user could see the connection status. This window is 
set to run on the thread so it does not  
directly affect the performance of the recognition 
application.  
To make the connection with Kinect the IP address 
and port are needed. This information is sent to the API 
and this will make the connection, which will return a 
message acknowledging the success or failure of the 
operation. This API is entitled SharpConnect.dll. This 
DLL file was slightly modified in order to work 
properly based on the functionalities that were 
developed. After the connection is established, the 
information from Kinect is received and analysed. Then 
we get the confirmation that it is (or not) the information 
to be expected. If it is not, then the user will be notified 
of what gesture he did and which gesture was expected. 
It will be possible to proceed only when the user 
manages to perform the requested gesture correctly. He 
or she has then learned and trained the requested 
gesture, according to the sign language system. 
 
 
V. GAME PLAY 
The first functionality developed was the inventory. 
The inventory stores the items acquired by the player 
and provides access to them at any time. 
Following that, the scripts to interact with the objects 
were created. These scripts were optimized so they 
could be later reused for multiple objects without having 
to change the code [2]. 
Having been established some objects on the map 
with the script to be added the inventory proceeded to 
the creation of this graphical interface.  
The map has objects and those objects contain scripts, 
which allows them to be added to the inventory of the 
player. Creation of the graphical interface was then 
started. 
The inventory consists of forty-two spaces that are 
empty upon initialization. With the inventory set up and 
ready to receive the objects that the user can acquire, the 
handling of collisions with these objects was created in 
order to detect when the user is within a reasonable 
distance to perform the interaction. 
There are also various non-player characters and 
interactive objects that will give information to the 
player about quests he must accomplish in order to 
obtain new gestures. This quests will ask the player to 
retrieve a certain gesture and use it in a specific situation 
such as using water to put off a fire, where the player 
will have to perform the water gesture near the fire 
object in the scene. There is always feedback provided 
to the user so that he knows what and when to do 
perform a gesture. 
Players’ scores are incremented during the game as 
they acquire new gestures. The shorter the time it takes 
between the acquisitions of two objects, the greater the 
score. If the delay is less than one minute, one hundred 
points are acquired, if it is between one and two 
minutes, fifty points are acquired, if the delay exceeds 
two minutes, twenty-five points will be obtained 
regardless of time spent. Figure 3 shows the score of 
game. 
 
 
Fig. 3. Score of the game 
After finishing the structure of the game, the 
development of the application to use with the Kinect 
was started. Since this application would be an 
adaptation of the VirtualSign project, it was necessary to 
be developed in C# on the Microsoft Visual Studio 
environment for compatibility reasons. 
The VirtualSign application detects and translates the 
gestures that the user makes, saving the one with higher 
probability of success. With this in mind we needed 
access to this part application code in order to be able to 
send the necessary content to Unity. The connection 
method used was the Socket as Unity supports .NET 
Framework. For this connection the use of an API that 
makes the connection in Unity and Development was 
needed. An application where relevant text would be 
sent was also developed.  
These gestures are then sent to the game and the 
player visualises them on the checkpoints so that he 
knows whether he is performing correctly or not. When 
all the gestures of the checkpoint are performed 
correctly the user can then move on to the next area of 
the level. 
VI. CONCLUSION 
The implications for understanding the relationship 
between games and learning are that games need not be 
defined as an essential instrument or a type of content 
but as contemporary human creations whose forms and 
meanings are strategic for education, more specifically 
concerning the hearing impaired community. 
The selection of this target population is due to the 
growing number of students with special needs who 
complete the elementary and high school and come to 
higher education. This situation demands for new means 
that allow these individuals to have easy access to 
educational digital content. In order to motivate them 
towards the learning process we have created a game 
that combines the sign language learning process with 
the pleasant feeling of playing a digital game.  
We believe this is a great time to take on the 
challenge of adopting new digital media, serious games 
and interactive simulations. However, the development 
of a game is always a complex task and many 
adversities were faced along the way. A lot of effort and 
time were needed to face challenges and solve 
problems, but a fair amount of knowledge was acquired 
during this process. 
ACKNOWLEDGEMENT 
This work was supported by the Engineering Institute 
of Oporto and GILT (Graphics, Interaction and Learning 
Technologies) under grant No. 60973089, a project FCT 
(Fundação para a Ciência e Tecnologia). 
REFERENCES 
[1] J. P. Gee, What video games have to teach us about learning and 
literacy, New York: Palgrave Macmillan, 2003. 
[2] K. Salen & E. Zimmerman, Rules of play: Game design 
fundamentals. Cambridge, MA: MIT Press, 2004 
[3] J. P. Gee, Situated language and learning: A critique of 
traditional schooling, New York: Routledge, 2004. 
[4] I Reyes, “Comunicar a través del silencio:las posibilidades de la 
lengua de signos”, Universidad de Sevilla, 2005. 
[5] D. Catue, Programming with the Kinect for Windows: Software 
Development Kit, Microsoft, 2012. 
[6] A. Herrero-Blanco and V. Salazar-Garcia, “Non-verbal 
predicability and copula support rule in Spanish Sign Language”. 
In: de Groot, Casper / Hengeveld, Kees (eds): Morphosyntactic, 
2005 
[7] M. Prensky, Digital Game-Based Learning, New York: 
McGraw-Hill, 2001. 
[8] I. Holm, Ideas and Beliefs in Architecture and Industrial design: 
How attitudes, orientations, and underlying assumptions shape 
the built environment, Oslo School of Architecture and Design., 
2006


English

A kinect game in the VirtualSign project: training and learning with gestures

Repositório Aberto da Universidade Aberta

A Kinect Game in the VirtualSign Project: Training and Learning with Gestures Paula Escudeiro*, Nuno Escudeiro*, Rosa Reis*, Ana Bela Baltazar*, Pedro Rodrigues*, Jorge Lopes*, Marcelo Norberto*, Maciel Barbosa**, and José Bidarra*** *Engineering Institute of Oporto, Portugal; **Faculty of Engineering, University ofOporto, Portugal; ***CIAC, Universidade Aberta (Open University), Lisbon, Portugal Abstract  — This paper presents the development of a game aimed at making the process of learning sign language enjoyable and interactive, using the VirtualSign Translator. In this game the player controls a character that interacts with various objects and non-player characters with the aim of collecting several gestures from the Portuguese Sign Language. Through the connection with VirtualSign Translator the data gloves and Kinect support this interaction and the character can then represent the gestures. This allows for the user to visualize and learn or train the various existing configurations of gestures.  To improve the interactivity and to make the game more interesting and motivating, several checkpoints were placed along game levels. This provides the players with a chance to test the knowledge they have acquired so far on the checkpoints, after performing the signs using Kinect.  A High Scores system was also created, as well as a History option, to ensure that the game is a continuous and motivating learning process. Index Terms — Kinect game, game development, educational content, educational game, Portuguese Sign Language. I. INTRODUCTION In this day and age, games and simulations create remarkable opportunities to overcome the scarcity of educational digital content available for the hearing impaired community. Furthermore, freedom of choice, challenge, participation, transparency, integrity, collaboration, fun, speed, and innovation should become a part of students’ learning experiences. In this context, playing games may be an important aspect of learning as this generation’s game-playing experiences are more widespread than the game-playing experiences of previous generations. The interest in gaming for educational purposes has increased over the last decade, with researchers identifying key pedagogical features that make good games inherently strong learning tools. What underlies the allure of games? Educational game researcher James Gee [1] shows how good game designers manage to get new players to learn their long, complex, and difficult games. A well-designed game entices players into the “reality” of the game world and keeps them there until the goals of the game have been met [2]. Gee points out that incorporating appropriate challenges that are “doable,” and other widely accepted effective learning principles that are supported by research in cognitive science, are in fact a large part of what makes good games motivating and entertaining [3]. Making these opportunities available to those who endure handicap and disabilities is a core concern in today’s society and a must to promote equity and inclusion. In this work we propose a new approach by using a game to make the process of learning sign language enjoyable and interactive. In this game the player controls a character that interacts with various objects and non-player characters with the aim of collecting several gestures from the Portuguese Sign Language. The character can then perform these gestures himself. This allows the user to visualize and learn or train the various existing gestures. The user interacts with the game through data gloves and a Kinect device. To raise the interactivity and to make the game more interesting and motivating, several checkpoints were placed along game levels. This provides the players with a chance to test the knowledge they have acquired so far on the checkpoints, after performing the signs using Kinect. A High Scores system was also created, as well as a History option, to ensure that the game is a continuous and motivating learning process. II. THE GAMEThe project consists of a didactic game about Portuguese Sign Language, where the player can enjoy the game while learning gestures simultaneously [4]. The VirtualSign Translator has been connected to the game [5] in order to make it more interactive, more appealing, and more effective using the translator Kinect and gloves. The main objective of this project is to facilitate the learning of the Portuguese Sign Language and to improve the dexterity of those who already know it, making learning a pleasant experience. The deaf community in Portugal has around 100 000 individuals and yet the digital content available for this community is still rather scarce. With this project, not only we are promoting knowledge for this restricted community, but we are also encouraging other people to learn and become able to better understand this community [6]. The game is played in first person view, so the player controls a character in a specific scenario.  Each map represents a level and each level has several scattered objects, through the map for the player to interact with, most of those objects are gestures. All objects collected by the player will be stored in his inventory and can be accessed at any point during the game. Most of these gestures and objects can be used through the inventory; the character will then perform the gesture so that the user can visualize how it is done, either automatically at checkpoints or by clicking them in the inventory. To progress in the game the player must collect all the gestures scattered around the level [7]. The faster the player manages to collect all the gestures the higher his score will be. The player may choose to play using Kinect but he must have first obtained all the gestures in the level and perform them. Our research showed that there are some projects related to this theme/area but none of them implies an automatic bidirectional translation process in a game as this does, therefor making this project very innovative.  Some of the related work is described below.  A. CopyCat  The game CopyCat is the most similar project in comparison to ours. It consists of a game where sign language gestures need to be executed properly in order to proceed. The movement analysis is done through gloves with sensors. However, the researchers from the CopyCat project have published a video where they show their intention to use Kinect for movement detection.  B. ProDeaf ProDeaf is an application that does the translation of Portuguese text or voice to Brazilian gesture language.  The objective of the ProDeaf is to make the communication between mute and deaf people easier, making digital content accessible in Brazilian gesture language.  C. Kinect Sign Language Translator in Beijing University Researchers in China have created the Kinect Sign Language Translator, a prototype system that understands the gestures of sign language and converts them to spoken and written language—and vice versa. This project was a result of collaboration, facilitated by Microsoft Research, between the Chinese Academy of Sciences, Beijing Union University, and Microsoft Research Asia, each of which made crucial contributions.    D. Faceshift  This programme analyses facial expressions, namely, the orientation of the user’s head and eyes. The information analysed by the application is used to animate virtual characters that can be used in videos or games.  E. MoVER MoVER (Movement in Virtual Environment for Rehabilitation), a Serious Game that simulates physiotherapical movements through challenges to perform virtual tasks using the human body. This project was developed in the University of São Paulo. III. APPLICATION REQUIREMENTS In this section the functional and non-functional requirements of the application are described. Functional requirements represent the features available to the user.  A. Functional Requirements The functional requirements were identified in the early stages of development of the project, and include the following: 1. Start Menu, where the player can choose the type of game he wants (with or without Kinect), consult the options or exit; 2. Options Menu, where you can change the graphics quality, volume, save or load the game and see the table of high scores.  The game requirements within the levels are: - Handling and controlling of the character; - Interacting with NPCs (Non-Player Characters); - Consulting the inventory and use the items in it; - Interacting with map objects; - Accessing to the above options menu.  B. Non-Functional Requirements Regarding non-functional requirements, our work focused on the following:  1. Usability Usability is the ease of use of the application and its accessibility to the user [8]. With this in mind during the project design it is expected the interface to be fairly intuitive, allowing for easy adaptation and learning. The user interfaces must be simple so anyone can use them easily. The character controls are simple, and throughout the game there are explanations of what needs to be done. The accessibility is guaranteed because it is only necessary to run an executable file in order to play while Kinect is not in use.  While playing using the Kinect, two applications are required as well as the necessary drivers, and also assuring that the Kinect is properly plugged in.  2. Performance The gaming performance is always a factor of the utmost importance, because the response time from the game to the user is always immediate, any delay or decrease of the number of frames per second can affect the gameplay making the game frustrating rather than fun.  To maintain the performance this game was tested to never run less than 60 frames per second on a computer. The essential functions must be constantly executed and the code must be optimized to avoid waste of resources.  Besides the code, all the factors that constrain the performance of the game should be taken into account, such as textures, bumps, and number of vertices of the 3D models among others.  The connection to the VirtualSign translator is instantaneous and there is no delay from the moment of the translation of the preformed gesture to the moment its receive in the game. Therefor the player wont notice any delays while preforming the gestures. IV. GAME ARCHITECTURE For this project two applications were developed, the game application in Unity 3D and the interface that connects the virtual sign translator to Unity. The interface was developed in Microsoft Visual Studio. The project was divided into layers, given its high degree of complexity. At the top level there is the interface.  The user can access all the functionalities of the project through this layer. This layer is responsible for forwarding the actions of the user to the next layers. On the lower level there are three layers. The sockets layer is responsible for linking the Unity game application to the Kinect that is why this layer is below the interface in order to provide the layer above with the player input. Another layer of the lower level is is the game engine; this layer is responsible for the execution of the game itself, representing the functions of Unity. Finally, there is the business layer, which is where the game functions are available to the player. Figure 1 shows the layers of game.   Fig. 1. Layers of game) For the development of this project it was decided to start with the implementation of the basic functionality and then proceed to the animation of the character, which was later replaced by an animated avatar now available at the GILT research group (Graphic Interaction and Learning Technologies). In the development of this project a draft was first developed implementing the basic functionalities, as shown in Figure 2.  Following this the player’s avatar was animated, however it was eventually replaced by the one provided by GILT.   Fig. 2. Scenario of the basic functionalities of the game After having a basic scenario created the development phase of the scripts started. All the translations are handled by the VirtualSign Translator therefore it’s not referenced in the game architecture as it is an different application. The sockets as mentioned before are responsible for the connection between the two applications and the information is transferred from one to another instantaneously.  For this application the official Unity Sockets example was adapted, and a window was created where the user could see the connection status. This window is set to run on the thread so it does not  directly affect the performance of the recognition application.  To make the connection with Kinect the IP address and port are needed. This information is sent to the API and this will make the connection, which will return a message acknowledging the success or failure of the operation. This API is entitled SharpConnect.dll. This DLL file was slightly modified in order to work properly based on the functionalities that were developed. After the connection is established, the information from Kinect is received and analysed. Then we get the confirmation that it is (or not) the information to be expected. If it is not, then the user will be notified of what gesture he did and which gesture was expected. It will be possible to proceed only when the user manages to perform the requested gesture correctly. He or she has then learned and trained the requested gesture, according to the sign language system.   V. GAME PLAY The first functionality developed was the inventory. The inventory stores the items acquired by the player and provides access to them at any time. Following that, the scripts to interact with the objects were created. These scripts were optimized so they could be later reused for multiple objects without having to change the code [2]. Having been established some objects on the map with the script to be added the inventory proceeded to the creation of this graphical interface.  The map has objects and those objects contain scripts, which allows them to be added to the inventory of the player. Creation of the graphical interface was then started. The inventory consists of forty-two spaces that are empty upon initialization. With the inventory set up and ready to receive the objects that the user can acquire, the handling of collisions with these objects was created in order to detect when the user is within a reasonable distance to perform the interaction. There are also various non-player characters and interactive objects that will give information to the player about quests he must accomplish in order to obtain new gestures. This quests will ask the player to retrieve a certain gesture and use it in a specific situation such as using water to put off a fire, where the player will have to perform the water gesture near the fire object in the scene. There is always feedback provided to the user so that he knows what and when to do perform a gesture. Players’ scores are incremented during the game as they acquire new gestures. The shorter the time it takes between the acquisitions of two objects, the greater the score. If the delay is less than one minute, one hundred points are acquired, if it is between one and two minutes, fifty points are acquired, if the delay exceeds two minutes, twenty-five points will be obtained regardless of time spent. Figure 3 shows the score of game.   Fig. 3. Score of the game After finishing the structure of the game, the development of the application to use with the Kinect was started. Since this application would be an adaptation of the VirtualSign project, it was necessary to be developed in C# on the Microsoft Visual Studio environment for compatibility reasons. The VirtualSign application detects and translates the gestures that the user makes, saving the one with higher probability of success. With this in mind we needed access to this part application code in order to be able to send the necessary content to Unity. The connection method used was the Socket as Unity supports .NET Framework. For this connection the use of an API that makes the connection in Unity and Development was needed. An application where relevant text would be sent was also developed.  These gestures are then sent to the game and the player visualises them on the checkpoints so that he knows whether he is performing correctly or not. When all the gestures of the checkpoint are performed correctly the user can then move on to the next area of the level. VI. CONCLUSION The implications for understanding the relationship between games and learning are that games need not be defined as an essential instrument or a type of content but as contemporary human creations whose forms and meanings are strategic for education, more specifically concerning the hearing impaired community. The selection of this target population is due to the growing number of students with special needs who complete the elementary and high school and come to higher education. This situation demands for new means that allow these individuals to have easy access to educational digital content. In order to motivate them towards the learning process we have created a game that combines the sign language learning process with the pleasant feeling of playing a digital game.  We believe this is a great time to take on the challenge of adopting new digital media, serious games and interactive simulations. However, the development of a game is always a complex task and many adversities were faced along the way. A lot of effort and time were needed to face challenges and solve problems, but a fair amount of knowledge was acquired during this process. ACKNOWLEDGEMENT This work was supported by the Engineering Institute of Oporto and GILT (Graphics, Interaction and Learning Technologies) under grant No. 60973089, a project FCT (Fundação para a Ciência e Tecnologia). REFERENCES [1] J. P. Gee, What video games have to teach us about learning and literacy, New York: Palgrave Macmillan, 2003. [2] K. Salen & E. Zimmerman, Rules of play: Game design fundamentals. Cambridge, MA: MIT Press, 2004 [3] J. P. Gee, Situated language and learning: A critique of traditional schooling, New York: Routledge, 2004. [4] I Reyes, “Comunicar a través del silencio:las posibilidades de la lengua de signos”, Universidad de Sevilla, 2005. [5] D. Catue, Programming with the Kinect for Windows: Software Development Kit, Microsoft, 2012. [6] A. Herrero-Blanco and V. Salazar-Garcia, “Non-verbal predicability and copula support rule in Spanish Sign Language”. In: de Groot, Casper / Hengeveld, Kees (eds): Morphosyntactic, 2005 [7] M. Prensky, Digital Game-Based Learning, New York: McGraw-Hill, 2001. [8] I. Holm, Ideas and Beliefs in Architecture and Industrial design: How attitudes, orientations, and underlying assumptions shape the built environment, Oslo School of Architecture and Design., 2006

This paper presents the developmentof a game aimed at making the process of learning signlanguage enjoyable and interactive, using the VirtualSignTranslator. In this game the player controls a characterthat interacts with various objects and non-playercharacters with the aim of collecting several gestures fromthe Portuguese Sign Language. Through the connectionwith VirtualSign Translator the data gloves and Kinectsupport this interaction and the character can thenrepresent the gestures. This allows for the user to visualizeand learn or train the various existing configurations ofgestures. To improve the interactivity and to make thegame more interesting and motivating, several checkpointswere placed along game levels. This provides the playerswith a chance to test the knowledge they have acquired sofar on the checkpoints, after performing the signs usingKinect. A High Scores system was also created, as well as aHistory option, to ensure that the game is a continuous andmotivating learning process

Paula Escudeiro

Nuno Escudeiro

Rosa Reis

Ana Bela Baltazar

Jorge Lopes

Pedro Rodrigues

Marcelo Norberto

Fernando Maciel Barbosa

José Bidarra

Open Repository of the University of Porto

A kinect game in the VirtualSign Project: training and learning with gestures

This paper presents the development
of a game aimed at making the process of learning sign
language enjoyable and interactive, using the VirtualSign
Translator. In this game the player controls a character
that interacts with various objects and non-player
characters with the aim of collecting several gestures from
the Portuguese Sign Language. Through the connection
with VirtualSign Translator the data gloves and Kinect
support this interaction and the character can then
represent the gestures. This allows for the user to visualize
and learn or train the various existing configurations of
gestures. To improve the interactivity and to make the
game more interesting and motivating, several checkpoints
were placed along game levels. This provides the players
with a chance to test the knowledge they have acquired so
far on the checkpoints, after performing the signs using
Kinect. A High Scores system was also created, as well as a
History option, to ensure that the game is a continuous and
motivating learning process

Repositório Aberto da Universidade do Porto

https://repositorioaberto.uab.pt/bitstream/10400.2/4446/1/artech2015_submission_61.pdf

A kinect game in the VirtualSign project: training and learning with gestures

Abstract

Similar works

Full text

Available Versions

The Repositório Aberto

Repositório Aberto da Universidade Aberta

Open Repository of the University of Porto

Repositório Aberto da Universidade do Porto