Modelo de um jogo de cooperação orientado para a comunidade surda

The rise of digital content and appearance of new devices capable of a more precise motion recognition creates an opportunity for developers to spread their work to a wider audience. A cooperation game model for the deaf is presented in this master thesis. It’s integrated within the VirtualSign project that consist on a bidirectional translator of sign language. The game consists of a first person puzzle game and requires two players to cooperate in order to get through the game. The game has a chat where users can type but it also makes use of the VirtualSign translator to allow deaf users to chat within the game using sign language. The cooperation is not only necessary but also rewarded as the players performance is represented through a score. The players can compare their performances as team with others through the high score list. The interface is fairly intuitive and allows the player to access all the core features needed for the game and the chat. The game story is also translated through the avatar. The game was designed for the deaf community and players aged above 12 years old. It was tested by 10 individuals with ages ranging from 23 to 65 and the feedback was very positive allowing the improvement of a few features. The results of the users testing is represented on the QEF where the game scored a 95% quality rate. This game aims to improve the social inclusion of the deaf community and show that it’s possible to communicate with them in games using their own language.Com o aumento do conteúdo digital e o surgimento de novas tecnologias capazes de obter uma maior precisão na captura e reconhecimento de movimentos, surge a oportunidade para os criadores de software de expandirem os seus conteúdos para mais pessoas. Nesta dissertação do mestrado é apresentado um modelo de um jogo de cooperação orientado para a comunidade surda. Este jogo está integrado no projeto VirtualSign que consiste num tradutor bidirecional de língua gestual. O jogo consiste na resolução de puzzles e é jogado em primeira pessoa. Dentro do jogo existe um chat de texto, no entanto permite aos surdos que usem o tradutor VirtualSign para falar em língua gestual. A conversa aparece sempre em texto e traduzida para língua gestual por um Avatar. A cooperação dentro do jogo não só é necessária como é recompensada através da pontuação dos jogadores. Os jogadores podem comparar as suas performances como equipa através da lista de melhores pontuações. O jogo apresenta vários desafios que só podem ser ultrapassados com a cooperação dos jogadores. A interface é intuitiva e permite aos utilizadores o acesso a todas as principais funcionalidades do jogo e também ao chat. A história do jogo é também traduzida pelo avatar. Isto permite aos utilizadores surdos uma fácil interpretação do jogo, o que não acontece quando só é apresentado texto escrito. O jogo foi criado com principal foco na comunidade surda, para jogadores com idades superiores a 12 anos. Os testes foram feitos por 10 jogadores com idades contidas entre 23 e 65 anos e o feedback foi bastante positivo, o que permitiu que fossem melhorados alguns aspetos funcionais do jogo. Os testes foram feitos em duas fases, alfa e beta. O resultado dos testes encontram-se representados no QEF. O jogo teve uma pontuação final de 95% utilizando o QEF. Este jogo ambiciona melhorar a inclusão social da comunidade surda e mostrar que é possível comunicar com surdos dentro de um jogo utilizando Língua gestual

Game design and the gamification of content : assessing a project for learning sign language

Comunicação apresentada na EDULEARN 2015, realizada em Barcelona de 6-8 de julho de 2015This paper discusses the concepts of game design and gamification of content, based on the development of a serious game aimed at making the process of learning sign language enjoyable and interactive. In this game the player controls a character that interacts with various objects and non- player characters, with the aim of collecting several gestures from the Portuguese Sign Language corpus. The learning model used pushes forward the concept of gamification as a learning process valued by students and teachers alike, and illustrates how it may be used as a personalized device for amplifying learning. Our goal is to provide a new methodology to involve students and general public in learning specific subjects using a ludic, participatory and interactive approach supported by ICT- based tools. Thus, in this paper we argue that perhaps some education processes could be improved by adding the gaming factor through technologies that are able to involve students in a way that is more physical (e.g. using Kinect and sensor gloves), so learning becomes more intense and memorable

A kinect game in the VirtualSign project: training and learning with gestures

Comunicação apresentada na 7ª Conferência Internacional de Arte Digital realizada em Óbidos de 19-20 de março de 2015This paper presents the development of a game aimed at making the process of learning sign language enjoyable and interactive, using the VirtualSign Translator. In this game the player controls a character that interacts with various objects and non-player characters with the aim of collecting several gestures from the Portuguese Sign Language. Through the connection with VirtualSign Translator the data gloves and Kinect support this interaction and the character can then represent the gestures. This allows for the user to visualize and learn or train the various existing configurations of gestures. To improve the interactivity and to make the game more interesting and motivating, several checkpoints were placed along game levels. This provides the players with a chance to test the knowledge they have acquired so far on the checkpoints, after performing the signs using Kinect. A High Scores system was also created, as well as a History option, to ensure that the game is a continuous and motivating learning process

Serious game for sign language

The work presented in this paper consists in the development of a game to make the process of learning sign language enjoyable and interactive. In order to do this, a game was created in which the player controls a character and interacts with various objects and non-player characters with the aim of collecting several gestures from the Portuguese Sign Language. These gestures can then be represented by the character. This allows the user to visualize and learn or train the various existing gestures. To raise the interactivity and to make the game more interesting and motivating, several checkpoints were placed along the level. This will provide the players a chance to test the knowledge they have acquired so far on the checkpoints by using Kinect. A High Scores system was also created as well as a history to ensure that the game is a continuous motivating process as well as a learning process.info:eu-repo/semantics/publishedVersio

Tradutor bidirecional de língua gestual portuguesa

Uma das maiores preocupações da Sociedade é promover a igualdade de oportunidades e a inclusão social. Este é um tópico chave na agenda Europeia, em particular ao nível do Ensino. Apesar do esforço que tem vindo a ser realizado por inúmeros investigadores e instituições, as dificuldades inerentes à comunicação entre ouvintes e surdos continua a ser uma barreira significativa para a inclusão. O elevado ritmo de desenvolvimento da tecnologia assistiva possibilitou o aparecimento recente de várias soluções para promover a inclusão da comunidade surda um pouco por todo o mundo. No entanto, no que respeita à língua gestual portuguesa, as soluções atualmente disponíveis são ineficientes e não cobrem as necessidades da comunidade. A criação de um tradutor bidirecional entre português escrito e língua gestual portuguesa permitirá uma comunicação mais fluída e eficiente entre ouvintes e surdos, gerando igualdade de oportunidades e promovendo a inclusão da comunidade surda na sociedade. O objetivo desta tese consiste na promoção da inclusão social da comunidade surda, tornando a comunicação entre surdos e ouvintes mais eficaz. O tradutor automático desenvolvido neste trabalho, denominado VirtualSign, tem capacidade para traduzir gestos em texto e texto em gestos usando um modelo de tradução desenvolvido especificamente para o efeito. O tradutor de gestos para texto utiliza um par de luvas de dados com 14 sensores para reconhecimento da configuração manual e o Kinect para reconhecer os movimentos corporais efetuados pelo utilizador. Os sinais recebidos destes dispositivos são convertidos num modelo de tradução que alimenta um componente de aprendizagem automática para identificar os gestos que estão a ser produzidos. A tradução de texto para gestos utiliza um avatar 3D criado para reproduzir as animações que representam o texto a traduzir. Estas animações representam palavras e expressões da língua portuguesa de sinais que são armazenadas numa base de dados. Um dos aspetos inovadores deste trabalho consiste no facto destas animações serem modeladas com recurso a uma linguagem de scripting o que reduz significativamente o espaço. Esta característica torna possível a utilização do sistema de tradução em dispositivos móveis e em outros contextos em que o espaço de armazenamento seja um aspeto crítico. Para apoiar os surdos e intérpretes de língua gestual a carregar a base de dados de gestos com animações for desenvolvido o VirtualSign Studio; uma aplicação de apoio à configuração de gestos em quaisquer línguas ou dialeto e a geração automática dos scripts correspondentes através da manipulação direta do avatar, ou seja, sem necessidade de conhecimentos técnicos da área informática. Este sistema de configuração está a ser utilizado por uma equipa de surdos e intérpretes de vários países incluindo: Portugal, Brasil, Alemanha, Reino Unido, Eslovénia, Grécia e Chipre. O sistema de tradução VirtualSign está em fase de pré-produção estando a ser desenvolvidas várias aplicações para o dia-a-dia que a breve trecho estarão disponíveis para o público.One of the modern society’s main concerns is promoting equal opportunities to everyone andsocial inclusion is a key topic in the agenda of the European Higher Education. Despite the efforts being made there is still not enough support for the deaf community. The technology has been growing very fast, with this some new applications start to appear, but the Portuguese sign language area is still lacking. This thesis main objective is the creation of bidirectional translator of Portuguese sign language. To be capable of making this translator it was necessary to create two different applications with the capacity of translating gesture into text and text into gesture. The gesture to text translator needs to have two different devices besides the computer, this two devices are a pair of gloves called 5DT gloves that contains 14 different sensors to be capable of recognizing the configurations made by the hand of the user. The second device is the Kinect that is a camera with sensors that has the capacity of recognizing movements made by the user. The text to gesture translator uses one avatar 3D that has been created to reproduce gestores that represents the transcribed text. To perform the gestures the avatar 3D uses an animation parameterization system to make the correct gestures. Translating sign language requires a vast set of parameters that need to be taken into consideration such as hand configuration, arms movement and facial expression. Those characteristics are the components of the translation moments. So, to store all the information required for the translation of words to signs, considering the large number of existing words, the creation of the VirtualSign Studio (VSS) was necessary. The VSS application allows the users to create gestures that represent words. All this development has been possible after an extensive study about sign language and after a study made about the state of art of applications that use sign language. This dissertation was realized on the Instituto Superior de Engenharia do Porto in the TMDEI discipline. This project goal is to obtain a major social inclusion of the deaf community in society, making the communication between deaf people and non-deaf much easier

Adult Learning Sign Language by combining video, interactivity and play in a 3D game platform

One in every six persons in the UK suffers a hearing loss, either as a condition they have been born with or a disorder they acquired during their life. 900,000 people in the UK are severely or profoundly deaf and based on a study by Action On Hearing Loss UK in 2013 only 17 percent of this population, can use the British Sign Language (BSL). That leaves a massive proportion of people with a hearing impediment who do not use sign language struggling in social interaction and suffering from emotional distress, and an even larger proportion of Hearing people who cannot communicate with those of the deaf community. This paper presents a theoretical framework for the design of interactive games to support learning BSL supporting the entire learning cycle, instruction, practice and assessment. It then describes the proposed design of a game based on this framework aiming to close the communication gap between able hearing people and people with a hearing impediment, by providing a tool that facilitates BSL learning targeting adult population. The paper concludes with the planning of a large scale study and directions for further development of this educational resource

Using Serious Games for Learning British Sign Language Combining Video, Enhanced Interactivity, and VR Technology

One in every six persons in the UK suffers a hearing loss, either as a condition they have been born with or they disordered they acquired during their life. 900,000 people in the UK are severely or profoundly deaf and based on a study by Action On Hearing Loss UK in 2013 only 17 percent of this population, can use the British Sign Language (BSL). That leaves a massive proportion of people with a hearing impediment who do not use sign language struggling in social interaction and suffering from emotional distress, and an even larger proportion of Hearing people who cannot communicate with those of the deaf community. This paper presents a serious game (SG) that aims to close the communication gap between able hearing people and people with a hearing impediment by providing a tool that facilitates BSL learning targeting adult population. The paper presents the theoretical framework supporting adult learning based on which a SG game using Virtual Reality (VR) technology has been developed. It explains the experimental framework of the study and presents the creation of the research instruments to facilitate the study comprising of a SG that integrates video and conventional video based educational material. It reports and analyses the study results that demonstrate the advantage of the SG in effectively supporting users learning a set of BSL signs and it presents qualitative outcomes that inform the further development of the game to serve learning needs. The paper closes with conclusions, directions for further development of this educational resource and future studies

Using serious games for learning sign language combining video, enhanced interactivity and VR technology

One in every six persons in the UK suffers a hearing loss, either as a condition they have been born with or they disordered they acquired during their life. 900,000 people in the UK are severely or profoundly deaf and based on a study by Action On Hearing Loss UK in 2013 only 17 percent of this population, can use the British Sign Language (BSL). That leaves a massive proportion of people with a hearing impediment who do not use sign language struggling in social interaction and suffering from emotional distress, and an even larger proportion of Hearing people who cannot communicate with those of the deaf community. This paper presents a serious game (SG) that aims to close the communication gap between able hearing people and people with a hearing impediment by providing a tool that facilitates BSL learning targeting adult population. The paper presents the theoretical framework supporting adult learning based on which a SG game using Virtual Reality (VR) technology has been developed. It explains the experimental framework of the study and presents the creation of the research instruments to facilitate the study comprising of a SG that integrates video and conventional video based educational material. It reports and analyses the study results that demonstrate the advantage of the SG in effectively supporting users learning a set of BSL signs and it presents qualitative outcomes that inform the further development of the game to serve learning needs. The paper closes with conclusions, directions for further development of this educational resource and future studies

Virtual sign : a real time bidirectional translator of portuguese sign language

Promoting equity, equal opportunities to all and social inclusion of people with disabilities is a concern of modern societies at large and a key topic in the agenda of European Higher Education. Despite all the progress, we cannot ignore the fact that the conditions provided by the society for the deaf are still far from being perfect. The communication with deaf by means of written text is not as efficient as it might seem at first. In fact, there is a very deep gap between sign language and spoken/written language. The vocabulary, the sentence construction and the grammatical rules are quite different among these two worlds. These facts bring significant difficulties in reading and understanding the meaning of text for deaf people and, on the other hand, make it quite difficult for people with no hearing disabilities to understand sign language. The deployment of tools to assist the daily communication, in schools, in public services, in museums and other, between deaf people and the rest may be a significant contribution to the social inclusion of the deaf community. The work described in this paper addresses the development of a bidirectional translator between Portuguese Sign Language and Portuguese text. The translator from sign language to text resorts to two devices, namely the Microsoft Kinect and 5DT Sensor Gloves in order to gather data about the motion and shape of the hands. The hands configurations are classified using Support Vector Machines. The classification of the movement and orientation of the hands are achieved through the use of Dynamic Time Warping algorithm. The translator exhibits a precision higher than 90%. In the other direction, the translation of Portuguese text to Portuguese Sign Language is supported by a 3D avatar which interprets the entered text and performs the corresponding animations

Meta-modelo de mecânicas de jogos sérios para surdos e cegos

A promoção da igualdade de oportunidades e inclusão social das pessoas com incapacidades é uma preocupação das sociedades modernas e do Espaço Europeu da Educação. No entanto, apesar da evolução da ciência e da disponibilidade de novas tecnologias para encontrar soluções para combater este problema, ainda há uma grande dificuldade na comunicação com essas comunidades nas suas vidas diárias e nos diferentes ambientes em que estão inseridas. O ambiente educacional não está preparado atualmente para estas comunidades, o que é um problema, já que a educação tem um grande impacto na vida de todos os cidadãos e a existência de barreiras limita oportunidades futuras. O uso de jogos sérios em um ambiente educacional estimula a aprendizagem e o pensamento crítico, mas apesar da relevância dos jogos sérios na comunicação e inclusão social, a partir da recolha de informações concluímos que não há grande variedade de jogos e ferramentas que facilitem a sua criação e sejam acessíveis a pessoas com incapacidades. Nesta tese será discutida e apresentada a arquitetura e o desenvolvimento da plataforma que visa melhorar a inclusão social e o acesso à educação para os surdos e cegos. Para isso, serão disponibilizadas ferramentas que possibilitem que estas comunidades participem na criação de jogos educativos e também que joguem com todos os restantes utilizadores da plataforma, de forma a superar as dificuldades de comunicação e barreiras enfrentadas nas suas vidas diárias, especialmente no ambiente educacional. A plataforma desenvolvida permitirá a criação de jogos educativos, do género Quiz, com a possibilidade de integrar a tecnologia iBeacon, de forma rápida e simples, por qualquer e para qualquer tipo de usuário. Tanto a plataforma quanto os jogos resultantes terão mecanismos de input e output que permitirão a inclusão de utilizadores com incapacidades, como a síntese de voz, o VirtualSign (tradutor bidirecional de linguagem gestual) e o controlo através de comandos de voz.Promoting equal opportunities and social inclusion for people with disabilities is a concern of modern societies and the European Education Area. However, despite the evolution of science and the availability of new technologies to find solutions to combat this problem, there is still a great difficulty in communication with these communities in their daily life and in different environments in which they are inserted. The educational environment is not currently prepared for these communities, which is a problem, since education has a great impact on the lives of all citizens and the existence of barriers limits future opportunities. The use of serious games in an educational environment stimulates learning and critical thinking, but despite the relevance of serious games regarding communication and social inclusion, from the information gathering we concluded that there are no large array of games and tools that facilitate its creation and are accessible to people with disabilities. In this thesis we will discuss and present the architecture and the development of the platform that aims to improve social inclusion and access to education for deaf and blind people. For this purpose, we will provide tools which enable these communities to participate in the creation of educational games and also playing along with all other users of the platform, in order to overcome the communication difficulties and barrier faced by these communities in their daily life, especially in the educational environment. The platform developed will allow the creation of educational games, of the genre Quiz with the possibility of integrating iBeacon technology, quickly and simply, by any and for any type of users. Both the platform and the resulting games will have input and output mechanisms that will allow the inclusion of users with disabilities, such as voice synthesis, VirtualSign (twoway sign language translator) and voice command control