Hand Sign to Bangla Speech: A Deep Learning in Vision based system for Recognizing Hand Sign Digits and Generating Bangla Speech

Recent advancements in the field of computer vision with the help of deep neural networks have led us to explore and develop many existing challenges that were once unattended due to the lack of necessary technologies. Hand Sign/Gesture Recognition is one of the significant areas where the deep neural network is making a substantial impact. In the last few years, a large number of researches has been conducted to recognize hand signs and hand gestures, which we aim to extend to our mother-tongue, Bangla (also known as Bengali). The primary goal of our work is to make an automated tool to aid the people who are unable to speak. We developed a system that automatically detects hand sign based digits and speaks out the result in Bangla language. According to the report of the World Health Organization (WHO), 15% of people in the world live with some kind of disabilities. Among them, individuals with communication impairment such as speech disabilities experience substantial barrier in social interaction. The proposed system can be invaluable to mitigate such a barrier. The core of the system is built with a deep learning model which is based on convolutional neural networks (CNN). The model classifies hand sign based digits with 92% accuracy over validation data which ensures it a highly trustworthy system. Upon classification of the digits, the resulting output is fed to the text to speech engine and the translator unit eventually which generates audio output in Bangla language. A web application to demonstrate our tool is available at http://bit.ly/signdigits2banglaspeech

Supervisor

In this thesis we propose a Dynamic Deformable Template (DDT)  using a Stable Mass Spring Model (SMSM). In our work,  we combine the idea of deformable templates and physically based  models  in  the  proposed  DDTs. A  DDT  is  a  template  whose deformation is governed by its physical properties. We use two different strategies for generating a  DDT. Handcrafted  DDTs  are  those  whose  structural  descriptions  are specific to the shape of the object class and vary among the classes. Rectangular grid DDTs are  structurally  equal  for  all  the  shape  classes  and  capable  of  sampling  space around the  object  as  well. Object  specific  shape  information  is  introduced  in  to  the templates as model forces. We also experimented with two different matching quality measures:  global  and  local  measures. The  first  one  measures  the  quality  of  global fitting of  the  templates  and  the  second  one  measures  the  quality  of  fitting  of  the templates at local level. The first measure can be applied to both types of templates. The second measure can only be applied to the rectangular grid DDTs as it requires comparison between  corresponding  parts  of  templates. We  used  handwritten  digit